[LON-CAPA-admin] Sudden Not open to be viewed

[Stuart Raeburn]

Lee,

> Has anyone encountered this before?

I have not seen it.

> The issue appears to resolve after around ten minutes, although it  
> is difficult to tell as some students didn't keep trying to access  
> the resource.  This leads me to think something may be amiss with  
> memcached.

The message: "Not open to be viewed  The problem will be open on an  
unknown date" will be displayed for a LON-CAPA problem if the open  
date which lonhomework::check_access() retrieves via a call to &EXT()  
in lonnet.pm is an empty string.

If you have a known time/date when this issue has occurred for a  
student on a particular server then you could look in the Apache  
access logs for requests for this same resource from around the same  
time by other students in the same course with sessions on the same  
server, and compare the number of bytes returned for each of the  
requests.

I would expect small differences because of different student names  
and usernames included in the HTML output, but they should all be  
similar, if all students were receiving "Not open to be viewed" until  
the unexpired cache of  parameter values had reached its 10 minute  
lifetime, and a new call to lonnet::dump() was made to retrieve the  
resourcedata for the course, and then to cache that in memcache with a  
key: courseres%3aCID%3aDOM(where CID is the courseID (e.g.,  
19727ade67f42eauiucl1) and DOM is domain (uiuc).

There is a script available from:

http://source.loncapa.org/cgi-bin/cvsweb.cgi/~checkout~/loncom/debugging_tools/memcached_dump.pl?rev=1.1;content-type=text/plain

which can be downloaded and run as user www to display what is  
currently cached in memcache on a particular server.  That script was  
posted to this list -- see the thread:  
http://mail.lon-capa.org/pipermail/lon-capa-admin/2015-February/003005.html --  
for related discussion about memcache.

You might add that script to /home/httpd/perl/debug/, set it to be  
owned by www:www and change permissions to make to executable.

If analysis of the Apache logs supports the idea that cached data was  
the cause of the problem, then you might modify memcached_dump.pl to  
only report cached opendates for courses where this issue is seen, and  
run the script (as www) from cron every five minutes or so to document  
what opendate values are being cached.

The script could also be run as-is and the output piped to grep to  
extract opendate parameter values in s apecific course, e.g.,

/home/httpd/perl/debug/memcached_dump.pl -u  |grep  
'msu_32813041047878598msul1' |grep "opendate'"

would find all cached opendate values in the  
msu_32813041047878598msul1 course (num: 32813041047878598msul1;  
domain: msu).


Stuart Raeburn
LON-CAPA Academic Consortium

Quoting "Bynum, Lee Hamilton" <leebynum at illinois.edu>:

> Hello Everyone,
>
> We've recently experienced a number of students encountering "Not  
> open to be viewed  The problem will be open on an known date" in  
> assignments and quizzes that they are currently working on.  This  
> appears to be a student by student issue as entire courses are not  
> reporting the problem, although it is across multiple courses and  
> access servers.
>
> The issue appears to resolve after around ten minutes, although it  
> is difficult to tell as some students didn't keep trying to access  
> the resource.  This leads me to think something may be amiss with  
> memcached.  Unfortunately I have not gotten to an instance of this  
> problem within the ten minute window.
>
> The behavior was first observed on Sunday, the first day of daylight  
> savings, but our admins report nothing irregular regarding the  
> system clocks.  A restart of the servers did not resolve this issue  
> either.  We'd hoped the restart would have cleared any bad cache in  
> memcached.
>
> I haven't noticed anything out of the ordinary in the logs, but I  
> could be missing something.  Has anyone encountered this before?   
> Any and all suggestions as to troubleshooting and where to check are  
> most welcome.
>
> Thanks,
>
> Lee