[LON-CAPA-admin] Not Open To Be Viewed Troubles

Tue Oct 16 15:13:47 EDT 2018

I have another batch of these unavailable resources.  While I dig into the new data points, I figure I should give an update.

It is our suspicion that removing the local files from the access server helps prevent this from occurring.  We had an incident on Friday on a single access server.  We removed the local files on that server as a precaution.  Over the weekend, more incidents occurred, but not on the access server we cleared the files from.

During the same period we had two additional behaviors that seem related to access server files not syncing with the library properly.  One student reported being given additional time on an exam midway through.  This behavior is less likely to be reported, so it is unclear if there are more incidents.  There is also a behavior in which a student repeatedly needed to log in to answer the same questions over again.  This is reflected in the log as an answer submitted but not evaluated as the access was removed.  The odd part of that is that it results in multiple markaccess entries.

With this newest batch we were able to demonstrate this behavior isolated to individual students despite other students using the same resources at the same time on the same server.

It is also worth noting that this is the first batch of incidents this semester.  It is unclear what might have changed on our servers to prompt it, but that is definitely something we are looking into.

In short, suddenly unavailable resources continue to pop up and we are still investigating.

Lee

> -----Original Message-----
> From: lon-capa-admin-bounces at mail.lon-capa.org <lon-capa-admin-
> bounces at mail.lon-capa.org> On Behalf Of Bynum, Lee Hamilton
> Sent: Wednesday, February 21, 2018 9:50 AM
> To: list about administration and system updating <lon-capa-admin at mail.lon-
> capa.org>
> Subject: Re: [LON-CAPA-admin] Not Open To Be Viewed Troubles
> 
> Stuart,
> 
> This has been very helpful for digging into what's going on.  In case I'm barking
> up the wrong tree, though, I'd like to sanity check some potential state values.
> I'm seeing values that are either single digit integers or a series of integers,
> which I suspect come from the sequence based classes.  For example, "2" or
> "2002202".  Are those the sort of values to be expected in user.state?
> 
> Lee
> ________________________________________
> From: lon-capa-admin-bounces at mail.lon-capa.org [lon-capa-admin-
> bounces at mail.lon-capa.org] on behalf of Stuart Raeburn [raeburn at msu.edu]
> Sent: Wednesday, October 04, 2017 10:28 PM
> To: lon-capa-admin at mail.lon-capa.org
> Subject: Re: [LON-CAPA-admin] Not Open To Be Viewed Troubles
> 
> Lee,
> 
> Reviewing my previous answer, I find that I provided an answer to a different
> question than the one you had actually asked.
> 
> Accordingly, I am going to revise my answer from: "it is in memory on the access
> server", to it is in memory (in %env) and also in the user's session file, on the
> filesystem on the access server.
> 
> As you noted the username_courseid.state file in /home/httpd/perl/tmp
> contains the conditional statement:
> 
> &EXT('user.resource.resource.200.1.awarddetail','BuffersLab') eq 'APPROX_ANS'
> 
> The user's session file (a GDBM file):
> 
> /home/httpd/lonIDs/username_sessionnum_domain_authhost.id
> 
> will contain a key of: user.state.uiuc_3g9196182397259fcuiuclibrary1,
> with the corresponding value containing the results of evaluated conditions.
> 
> The routine: &Apache::lonuserstate::evalstate() is used to determine (and save)
> the value for:
> user.state.uiuc_3g9196182397259fcuiuclibrary1, which is appended to %env,
> and also to the session file.
> 
> The evalstate() routine will evaluate the conditions in perl Safe space, which will
> include calling lonnet::EXT(), which in turn calls lonnet::restore to retrieve the
> user's submission history for the resource to which the alias: 'BuffersLab' points.
> 
> The routine: &Apache::lonnet::devalidate() will remove
> user.state.uiuc_3g9196182397259fcuiuclibrary1 from %env (and from the user's
> session file).
> 
> The devalidate() routine is called whenever lonnet::store() or lonnet::cstore are
> called in course context to store the user's submissions, points etc.
> 
> Note: the question I answered with my previous response to this thread was
> where can you find values for items such as:
> &EXT('user.resource.resource.200.1.awarddetail','BuffersLab') when a problem
> is being rendered, and in that case the answer is in memory.
> 
> Stuart Raeburn
> LON-CAPA Academic Consortium
> 
> >>
> >> Does this live entirely in memory or am I missing something?
> >>
> >
> > It is in memory on the access server.
> >
> > When the problem is rendered, the &initialize_storage() routine is
> > called in structuretags.pm within start_problem() which will
> populate
> > the (global) %history hash in Apache::lonhomework with submission
> and
> > award history etc. for the current resource (in the current course
> > context and instance for the current user) by calling
> > lonnet::restore().
> >
> > See lines 1024-1025 in the version of structuretags.pm included in
> > loncapa-2.11.2.
> >
> > Later during rendering, the global %history hash is cleared with undef
> > (within the end_problem routine).
> >
> >
> > Stuart Raeburn
> > LON-CAPA Academic Consortium
> >
> >> I'm tracing the path of where a condition is stored against the
> > files
> >> on our servers.
> >>
> >> Condition to be met:
> >>
> > &EXT('user.resource.resource.200.1.awarddetail','BuffersLab') eq
> >> 'APPROX_ANS'
> >>
> >>
> >
> Library1:/home/httpd/lonUsers/uiuc/u/s/e/username/uiuc_3g9196182397259fc
> uiuclibrary1.db
> >
> >> contains:
> >>
> >
> 2:uiuc/dmills/CHEM105/Buffers/Buffers.sequence___5___uiuc/dmills/CHEM105
> /Buffers/Lab.problem:resource.200.1.awarddetail
> > =
> >> APPROX_ANS
> >>
> >> I'm having a difficult time locating that condition on the access
> servers, though.  I've found the conditional statement in the .state
> >
> >> file, but I'm not seeing it in any of the user files.  Does this
> > live
> >> entirely in memory or am I missing something?
> >>
> >> Thanks,
> >>
> >> Lee
> >>
> >>
> >> -----Original Message-----
> >> From: lon-capa-admin-bounces at mail.lon-capa.org
> >> [mailto:lon-capa-admin-bounces at mail.lon-capa.org] On Behalf Of
> > Stuart
> >> Raeburn
> >> Sent: Tuesday, October 3, 2017 7:14 PM
> >> To: lon-capa-admin at mail.lon-capa.org
> >> Subject: Re: [LON-CAPA-admin] Not Open To Be Viewed Troubles
> >>
> >> Hello Lee,
> >>
> >> There is no log file of access server parameter file changes, but
> > you
> >> could make your own (see below).
> >>
> >> Looking back through the archives, it seems you ran into this issue
> > in March.
> >>
> >> I posted a response to the list to that earlier post. See:
> >>
> >> mail.lon-capa.org/pipermail/lon-capa-admin/2017-March/003273.html
> >>
> >> which discussed how to view cached parameters in memcache.
> >>
> >> Assuming the open date parameter in effect is for all students, or
> >> for a specific section, and not for individual student(s), then
> >> parameters will be retrieved using lonnet::get_courseresdata().
> That
> >
> >> will, in turn, call &lonnet::dump(), when the cache on the access
> >> server has expired (it's valid for 10 minutes).
> >>
> >> You could modify lonnet::get_courseresdata() to write the hash
> >> returned by dump() for a particular course to a file (e.g.,
> >> /home/httpd/perl/tmp/debug/$coursenum on the access server).
> >>
> >> Your code could check if the file already exists, and if it did
> move
> >
> >> the existing file to /home/httpd/perl/tmp/debug/$coursenum.old,
> >> before writing the latest data to a new file.
> >>
> >> You could then create a perl script, which would be run by cron
> > every
> >> 5 minutes to look in /home/httpd/perl/tmp/debug/.  The script would
> compare the contents of $coursenum.old and $coursenum (if both
> > exist)
> >> and record what had changed for course: $coursenum for the current
> timestamp, in a log file.  The script would then unlink $coursenum.old.
> >>
> >>
> >> Stuart Raeburn
> >> LON-CAPA Academic Consortium
> >>
> >>
> >>> Hello Admins,
> >>>
> >>> I am diving back into an issue we've had in which resources are
> >>> temporarily "Not open to be viewed."  The resources should be
> >>> available now and do become available after 10-20 minutes or a
> >> change
> >>> of access server.  The leading theory is that the parameter file
> is
> >>> being read incorrectly, resulting in the access server thinking
> the
> >>> resource is unavailable until the bad parameter is updated.
> >>>
> >>> Is there a log anywhere of access server parameter file changes?
> >>> Because things seem to go back to normal after a while I have yet
> > to
> >>
> >>> have the problem reported to me in time to look at the potentially
> >>> problematic files in real time.
> >>>
> >>> Sorry for rehashing old troubles, but I'm hoping this time will to
> >> the trick.
> >>>
> >>> Lee
> >>
> >> _______________________________________________
> >> LON-CAPA-admin mailing list
> >> LON-CAPA-admin at mail.lon-capa.org
> >
> > _______________________________________________
> > LON-CAPA-admin mailing list
> > LON-CAPA-admin at mail.lon-capa.org
> 
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin