[LON-CAPA-admin] Students not seeing homework on access servers

Fri Oct 2 14:51:50 EDT 2015

Mike,

> Please help us understand and solve these errors.

An "Invalid symb" error message will be logged in the Apache error log  
file if the user's course session files have expired on the server  
currently hosting the user session.

If you look in /home/httpd/perl/tmp you will find files named:

<username>_<domain>_<coursenum>.db
<username>_<domain>_<coursenum>_parms.db
<username>_<domain>_<coursenum>.state
<username>_<domain>_<coursenum>_symb.db

where <username> is the username of the user (e.g., mikeb), <domain>  
is the course domain (i.e., purdue), and <coursenum> is the  
coursenumber (e.g., 74323912a211e547fpurduel1

The script: /home/httpd/perl/loncron which is run as user www at 5.10  
am (see entry in /etc/cron.d/loncapa) will, amongst other things,  
clean-up files in /home/httpd/perl/tmp with last modification times  
that are more than 24 hours ago (as determined by the value of  
lonExpire, in seconds, found in /etc/httpd/conf/loncapa.conf).

If the <username>_<domain>_<coursenum>.db file is missing, and a  
student has left a browser tab/window open which is logged into  
LON-CAPA, certain actions (e.g., reloading a LON-CAPA page, make a  
submission to a displayed homework problem) will result in a 406  
return code "HTTP_NOT_ACCEPTABLE", and a logged "Invalid symb"  
message.  An "Access denied" screen will be displayed in the student's  
browser.  Subsequent actions will typically result in display of the  
roles screen with the message: "As your session file for the course or  
community has expired, you will need to re-select it.".

Anyway, you might look in the Apache access logs to see who the user  
was in each of these cases found in the Apache error logs, e.g., by  
matching the IP address, and then take a look at log-in history and  
access history, to determine if the "Invalid symb" was logged in the  
case of a request made after the last 5:10 am run of  
/home/httpd/perl/loncron had cleaned up stale session files for a user  
with more than 24 hours of inactivity.

The user's activity.log file can also be useful -- you'll find that on  
the LON-CAPA library server in the user's directory:

/home/httpd/lonUsers/purdue/<1>/<2>/<3>/<username>/

where <1> is the first character, <2> second character, <3> third  
character in the user's username.

The following perl one-liner run in that directory displays the  
dates/times in a more readable format:

cat activity.log | perl -p -e '($date,$host,$command)=split /:/,  
$_;$_=localtime($date);$_.=":".$host.":".$command;'

Stuart Raeburn
LON-CAPA Academic Consortium

Quoting Mike Budzik <mikeb at purdue.edu>:

> Stuart,
> I'm not sure you addressed the Invalid symb issue in Richard's email and we
> are still seeing Invalid symb errors in our error logs.  At first we
> thought it wasn't affecting users because problem reports from users
> stopped, but the messages in the log didn't.  I no longer believe that this
> isn't impacting users.
>
> [Thu Oct 01 20:19:17 2015] [error] access to
> /res/purdue/purdue_math/math16010/Differentiation/Implicit
> Differentiation/Problem/11_d.problem failed for IP-ADDRESS, reason: Invalid
> symb for /res/purdue/purdue_math/math16010/Differentiation/Implicit
> Differentiation/Problem/11_d.problem:
> uploaded/purdue/7p186918837ca553dpurduel1/default_1432132025.sequence___17___purdue/purdue_math/math16010/Differentiation/Implicit
> Differentiation/Problem/11_d.problem
>
> [Thu Oct 01 21:05:30 2015] [error] access to
> /res/purdue/purdue_math/math16010/Differentiation/Implicit
> Differentiation/Problem/11_d.problem failed for IP-ADDRESS, reason: Invalid
> symb for /res/purdue/purdue_math/math16010/Differentiation/Implicit
> Differentiation/Problem/11_d.problem:
> uploaded/purdue/7p186918837ca553dpurduel1/default_1432132025.sequence___17___purdue/purdue_math/math16010/Differentiation/Implicit
> Differentiation/Problem/11_d.problem
>
> [Fri Oct 02 02:46:43 2015] [error] access to
> /public/purdue/7p186918837ca553dpurduel1/syllabus failed for IP-ADDRESS,
> reason: Invalid symb for
> /public/purdue/7p186918837ca553dpurduel1/syllabus:
> uploaded/purdue/7p186918837ca553dpurduel1/default.sequence___1___public/purdue/7p186918837ca553dpurduel1/syllabus
>
> It looks like in the case of the last one, the user received a 406 error:
> from the access log:
> IP-ADDRESS - USERNAME [02/Oct/2015:02:46:42 -0400] "GET
> /public/purdue/7p186918837ca553dpurduel1/syllabus HTTP/1.1" 406 386 "-"
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.78.2
> (KHTML, like Gecko) Version/7.0.6 Safari/537.78.2"
>
>
> Please help us understand and solve these errors.
> Thank you,
> Mike Budzik
>
>
> On Sun, Sep 20, 2015 at 6:41 PM, Stuart Raeburn <raeburn at msu.edu> wrote:
>
>> Richard,
>>
>> Apache2::RequestIO::print: (103) Software caused connection abort at....
>>>
>>> And then there are different files they are getting that at
>>>
>>> at /home/httpd/lib/perl//Apache/lonerrorhandler.pm line 53
>>> at /home/httpd/lib/perl//Apache/lonnavmaps.pm line 1831
>>> at /home/httpd/lib/perl//Apache/lonnavmaps.pm line 1864
>>> at /home/httpd/lib/perl//Apache/lonnavdisplay.pm line 118
>>> at /home/httpd/lib/perl//Apache/lonhomework.pm line 1108
>>> and others.
>>>
>>
>> This usually means a student got tired of waiting for the page to load,
>> and clicked away.  As a result the server's response could be longer be
>> returned to the client, so the message "Software caused connection abort"
>> is logged and Apache/mod_perl aborts at the current $r->print() line, which
>> is what you'll find at those lines in each of the reported perl modules.
>>
>> The first of these -- the one involving lonerrorhandler.pm is potentially
>> the most interesting, as it suggests an internal server error occurred
>> first, so you might look in the Apache error_log to find out what the error
>> was.
>>
>> Of course, students clicking away, is a symptom.  The larger question is
>> why are pages loading slowly on your access servers.  The first thing I
>> would check is that the memcached daemon is running:
>>
>> ps -ef |grep memcached |grep -v grep
>>
>> (The output should be something like:
>> www  <process> 1  0 <date> <time> /usr/bin/memcached -m 400 -v
>>
>> and is caching; see my post to the admin list from earlier this year for a
>> script you can use to dump the contents of memcached:
>>
>> http://mail.lon-capa.org/pipermail/lon-capa-admin/2015-February/003005.html
>>
>> Also see:
>> http://mail.lon-capa.org/pipermail/lon-capa-admin/2015-February/003013.html
>>
>>
>> Stuart Raeburn
>> LON-CAPA Academic Consortium
>>
>>
>> Quoting "Lucas, Richard" <clucas at purdue.edu>:
>>
>> Ok apparently the student issue was a false alarm raised by the
>>> instructor so no issues there. I am still wondering if there is  anything I
>>> can do about the errors though.
>>>
>>> Thanks,
>>> Richard
>>>
>>> From: lon-capa-admin-bounces at mail.lon-capa.org  [mailto:
>>> lon-capa-admin-bounces at mail.lon-capa.org] On Behalf Of  Lucas, Richard
>>> Sent: Sunday, September 20, 2015 4:04 PM
>>> To: lon-capa-admin at mail.lon-capa.org
>>> Subject: [LON-CAPA-admin] Students not seeing homework on access servers
>>>
>>> We just added two access servers on Friday and today there are many
>>> students reporting they are unable to see their homework. I am not  sure
>>> what to look at here, but I do see a bunch of errors in the  httpd
>>> error_log. Most of them start with this:
>>>
>>> Apache2::RequestIO::print: (103) Software caused connection abort at....
>>>
>>> And then there are different files they are getting that at
>>>
>>> at /home/httpd/lib/perl//Apache/lonerrorhandler.pm line 53
>>> at /home/httpd/lib/perl//Apache/lonnavmaps.pm line 1831
>>> at /home/httpd/lib/perl//Apache/lonnavmaps.pm line 1864
>>> at /home/httpd/lib/perl//Apache/lonnavdisplay.pm line 118
>>> at /home/httpd/lib/perl//Apache/lonhomework.pm line 1108
>>> and others.
>>>
>>> I also see some other messages like this, but they are a lot fewer:
>>>
>>> [Sun Sep 20 15:21:09 2015] [error] access to
>>> /res/purdue/purdue_math/math15800/Polynomial and Rational
>>> Functions/Rational Functions/Rational
>>> Functions_problems/asyp1_03_d.problem faile
>>> d for 10.192.57.242, reason: Invalid symb for
>>> /res/purdue/purdue_math/math15800/Polynomial and Rational
>>> Functions/Rational Functions/Rational
>>> Functions_problems/asyp1_03_d.problem: upl
>>> oaded/purdue/4o12229eff3c955c9purduel1/default_1442167687.sequence___4___purdue/purdue_math/math15800/Polynomial
>>> and Rational Functions/Rational Functions/Rational  Functions_problems/a
>>> syp1_03_d.problem
>>> [Sun Sep 20 15:21:09 2015] [error] access to
>>> /res/purdue/purdue_math/math15800/Polynomial and Rational
>>> Functions/Rational Functions/Rational
>>> Functions_problems/asyp1_03_d.problem faile
>>> d for 10.192.57.242, reason: Invalid Access for peng119 domain  purdue
>>> access bre
>>>
>>> Any ideas here?
>>>
>>>
>>> Thanks,
>>> Richard
>>>
>>
>> _______________________________________________
>> LON-CAPA-admin mailing list
>> LON-CAPA-admin at mail.lon-capa.org
>> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin