[LON-CAPA-admin] loncron count

[Stuart Raeburn]

James,

> The next set checked is "Delayed Messages".  The N/W/E is then 26/1119/0

In general I would say if the count is from notices and warnings then  
I would not be concerned.  The E count (Error count) is the number  
which should be a cause for concern, but in this case it is zero, so  
no worries there.

Anyway the large increase in the "Warnings" count comes from entries  
in the log file:

/home/httpd/perl/logs/lonnet.perm.log

That file is never rotated, and contains a permanent record of certain  
types of transaction conditions for the lifetime of your LON-CAPA  
installation.

When loncron parses that file the warnings count is incremented by 1  
for each line in lonnet.perm.log which contains a record starting:

timestamp:F:server:command details

e.g.,
1351688191:F:scl1:log

These are transactions which were delayed (because connection to the  
remote server had been lost), and for which successful writing to a  
file containing the incomplete transaction, where the file is named:

timestamp.count.pid.command.server

in the directory:
/home/httpd/sockets/delayed

could not be verified -- i.e., F is for Fail.
(This could occur, for example, if the filesystem on your server was  
out of space, or had dropped to read-only mode for some reason).

Note: in the file name: timestamp is UNIX timestamp, count is number,  
pid is process ID, command is the command to be executed, server is  
the remote server.

It is very unusual for verification to fail. However, in your case  
there are 38 such unverified recorded delayed transactions between  
10/23/2012 and 10/31/2012.
This is surprising. (Those items will have added 38 to the warnings count).

It is much more common is for the entry in  
/home/httpd/perl/logs/lonnet.perm.log for the delayed transaction to  
be of the form:

timestamp:D:server:command details

(where D is for Delayed).

These delayed commands will be retried when connection to the remote  
server has been re-established.  If the re-trial is successful, there  
will be a corresponding entry in  
/home/httpd/perl/logs/lonnet.perm.log, of the form:

timestamp:S:server:command details

where the "S" indicates that completion of the delayed transaction was  
successful.

In loncron the warnings count in incremented by:
5*$unsend

where $unsend is incremented by 1 for each "D" transaction, and  
decremented for each "S" transaction in  
/home/httpd/perl/logs/lonnet.perm.log.

The information in the excerpt from your lon_status file written by  
loncron indicated:

Total unsend messages: 211
(which would add: 5*211 (i.e., 1055) to the warnings count.

The next section of the lon_status report ("Outgoing buffer") displays  
the contents of files in the /home/httpd/sockets/delayed directory

These will be items which were delayed, and have yet to be successfully sent.
Apparently these are all update notifications (presumably for  
resources belonging to authors in the sc domain) which, in the past  
were copied to remote servers (which left behind a subscription on the  
resource's home server -- i.e., your home server).

Looking at the identity of the remote servers, these all look to be  
servers which are no longer in use (so connection is never going to be  
re-established, and the delayed notifications are never going to be  
sent).

Stuart Raeburn
LON-CAPA Academic Consortium


Quoting James Clawson <clawsonj at mailbox.sc.edu>:

> Thanks...
>
> When I look at the connection status report, I see a lot of stuff I
> don't understand (that will come as I learn), but I do notice this:
> Towards the end of the page, the "lonr" log is checked (it is next to
> last) and the Notices/Warnings/Errors are 26/26/0
> The next set checked is "Delayed Messages".  The N/W/E is then 26/1119/0
>
> This is the source of such a high loncron count, but I don't know what
> the warnings are.
>
> Attached is a text file for just this section.
>
> James
>
>
>
> On 11/7/12 1:27 PM, Stuart Raeburn wrote:
>> Hi,
>>
>> High loncron count will be reported by:
>> Main Menu -> Status of domain servers -> Domain Status
>>
>> for a server in your domain where:
>> Notices+4*Warnings+100*Errors > 2500
>>
>> as determined on the last run of /home/httpd/perl/loncron by www
>> (typically the most recent nightly run at 5.10 am local time)
>>
>> As Domain Coordinator you can display the output from the last run   
>> of loncron via:
>>
>> Main Menu -> Status of domain servers -> Connection Status
>>
>> on the particular server in the domain for which the high loncron   
>> count is reported.
>>
>> That web page will show how the totals for Notices, Warning and   
>> Errors were accumulated, as the different logs were checked by   
>> loncron.
>>
>>
>> Stuart Raeburn
>> LON-CAPA Academic Consortium
>>
>>
>> Quoting James Clawson <clawsonj at mailbox.sc.edu>:
>>
>>> What is meant by a "High loncron count."?
>>>
>>>
>>> I see this when I log in as domain coordinator, and go to server
>>> utilities/domain status.
>>>
>>> Thanks
>>>
>>> -- 
>>> James Clawson, Jr.
>>> Laboratory Manager
>>> University of South Carolina
>>> Department of Physics and Astronomy
>>> Jones PSC Room 402D
>>> 803-777-8994