[LON-CAPA-admin] {Disarmed} {Disarmed} Re: Lots of errors...

[Lucas, Mark]

James,

Stuart can suggest longer term solutions, but basically find a few log files that might be cleaned out
and stop/start your web server and loncontrol on your library server. That should fix things.

/home/httpd/perl/logs sometimes has older log files. As well, if you have /var/logs on your home
drive, you can probably find some space there.

Is the library server on a VM where you can grovel more space? (we just had to do that)
/home/httpd/lonUsers will probably be the largest culprit, and there is currently no graceful
way to archive records.

Later,
Mark


> On Sep 6, 2021, at 2:39 PM, Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org> wrote:
> 
> /home is full.
> 
> 
> 
> James Clawson
> Laboratory Manger
> University of South Carolina
> Department of Physics and Astronomy
> clawsonj at mailbox.sc.edu
> 
> From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Raeburn, Stuart via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
> Sent: Monday, September 6, 2021 2:08 PM
> To: lon-capa-admin at mail.lon-capa.org <lon-capa-admin at mail.lon-capa.org>
> Subject: Re: [LON-CAPA-admin] Lots of errors...
>  
> James,
> 
> >
> > We are getting a lot of error reports similar to what is attached.
> >
> 
> You might check to see whether the disk partition for /home is full (i.e, 100% used) on your LON-CAPA library server.  What does the command:
> 
> df -h
> 
> report?
> 
> Stuart Raeburn
> LON-CAPA Academic Consortium
> ________________________________________
> From: Raeburn, Stuart <raeburn at msu.edu>
> Sent: Sunday, September 5, 2021 10:32 PM
> To: lon-capa-admin at mail.lon-capa.org
> Subject: Re: Lots of errors...
> 
> James,
> 
> I am able to transfer my user session on another LON-CAPA node in the network to the sc access server, and the sc access server is able to retrieve information about my courses etc.
> 
> However, I am not able to then transfer my user session from the sc access server to the sc library server.  I am able to display the LON-CAPA course catalog for the sc domain on both the sc access server and sc library server.
> 
> Based on this I would conclude that the sc access server is able to connect to other nodes in the LON-CAPA network (including the sc library server), but it seems that the sc library server is not able to make a "LON-CAPA" connection back to my home server.
> 
> I don't know whether your campus firewall rules limit outbound connections for your LON-CAPA library server to IP addresses outside the sc network, so I don't know whether what I see currently is normal for your library server.
> 
> From the command line, these commands (run on each server -- library and access)
> 
> ps -ef |grep lond |grep sc
> ps -ef |grep lonc |grep sc
> 
> should show whether there are connections between lonc (on the library node) and lond on the access node), and vice versa.
> 
> For the first one you should see something like
> lond: Listening to <sc hostID>
> 
> For the second you should see something like:
> Connection count: 1 Retries remaining: 5
> 
> In the error report from the errorhandler web form submitted by a student, the error is:
> ... write error at /home/httpd/lib/perl//Apache/lonuserstate.pm line 541
> 
> which I expect would be called when a course session was being loaded (after role selection).
> 
> There are log files in /home/httpd/perl/logs.
> 
> >
> > Students normally log into the access server and have been reporting an issue
> > that homework submissions are not being taken or rejected.
> >
> 
> Storage of a submission is considered operation "critical", so if a connection cannot be made from the access server to the library server, any "delayed" transactions will be queued, and completed when connectivity is restored.  You will find information about delayed transactions (destined for the library server) in /home/httpd/perl/logs/lonnet.perm.log on the access server.
> 
> If you have additional information to share, while troubleshooting, feel free to contact me off-list.
> 
> Stuart Raeburn
> LON-CAPA Academic Consortium
> ________________________________________
> From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
> Sent: Sunday, September 5, 2021 8:48 PM
> To: lon-capa-admin at mail.lon-capa.org
> Subject: [LON-CAPA-admin] Lots of errors...
> 
> Hello,
> 
> We are getting a lot of error reports similar to what is attached.
> 
> On another (or possibly related) note:
> 
> We are having login issues with our library server but not always the access one (I've been able to get in most of the time).  Students normally log into the access server and have been reporting an issue that homework submissions are not being taken or rejected.  The screen returns to the same state as before answering (no increase in submission attempts).   I have verified this behavior.   One thought I had was that it could be that the access server is not communicating with the library server (I don't have the knowledge yet to know how to check if this is true or not), and something is wrong on the library server.
> 
> As far as I can tell, this is behavior that started a few days ago (Thursday).
> 
> Any help on where to start diagnosis is appreciated.
> 
> James Clawson
> Laboratory Manger
> University of South Carolina
> Department of Physics and Astronomy
> clawsonj at mailbox.sc.edu<mailto:clawsonj at mailbox.sc.edu>
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:https://protect2.fireeye.com/v1/url?k=482cf89a-17b7c19f-482cb65b-86564cfb0d85-05a5963fe217332c&q=1&e=520bac02-9ae8-47fa-8eed-452b8fdd9144&u=http%3A%2F%2Fmail.lon-capa.org%2Fmailman%2Flistinfo%2Flon-capa-admin
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin

-- 
Mark Lucas
Professor of Instruction
Department of Physics and Astronomy (College of Arts and Sciences)/Ohio Honors Program (Honors College)
252D Clippinger Lab, Ohio University, Athens, OH 45701
email: lucasm at ohio.edu
phone: (740)597-2984
Pronouns: He, Him, His