[LON-CAPA-admin] lonsql errors

Robert F Gonzales rgonzal at binghamton.edu
Tue May 8 13:11:37 EDT 2012 

Sorry in advance for the long post.  I tried to include some log file
entries.

I've got a problem when trying to 'View recent activity'.    The result is
a lonsql_errors file that fills up the hard drive.

When this happens, parse_activity_log.pl is consuming about 98% of the cpu
on the library server.

I can't see recent activity after April 24.  Here is my lonsql.log file
since that time:

Tue Apr 24 09:15:34 2012 (28593): preparing activity log tables for
65304592703204f4fbinghamtona3
Tue Apr 24 09:15:40 2012 (5505): Child 28599 died
Tue Apr 24 09:15:41 2012 (5505): Child 28641 died
Tue Apr 24 09:16:13 2012 (28593): Child -1 died
Tue Apr 24 09:16:13 2012 (28593): /home/httpd/perl/parse_activity_log.pl
-course=65304592703204f4fbinghamtona3 -domain=binghamton
Tue Apr 24 09:16:13 2012 (5505): Child 28593 died
Wed Apr 25 04:33:15 2012 (5505): Child 402 died
Wed Apr 25 04:33:56 2012 (5505): Child 431 died
Wed Apr 25 04:35:00 2012 (5505): Child 433 died
Wed Apr 25 04:35:30 2012 (5505): Child 439 died
Thu Apr 26 12:30:16 2012 (5505): Child 4908 died
Thu Apr 26 12:35:33 2012 (5505): Child 4914 died
Thu Apr 26 12:42:49 2012 (5505): Child 4918 died
Thu Apr 26 15:08:36 2012 (5505): Child 4922 died
Sun Apr 29 11:43:28 2012 (5505): Child 11841 died
Sun Apr 29 11:43:28 2012 (5505): Child 11824 died
Mon Apr 30 06:42:55 2012 (5505): Child 11860 died
Mon Apr 30 09:45:44 2012 (5505): Child 12330 died
Mon May  7 09:34:11 2012 (5505): Child 17553 died
Mon May  7 09:46:56 2012 (5505): Child 17555 died
Mon May  7 09:55:11 2012 (5505): Child 21832 died
Mon May  7 10:21:32 2012 (5505): Child 22408 died
Tue May  8 10:10:01 2012 (16977): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:10:47 2012 (16955): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:10:47 2012 (16955): Child -1 died
Tue May  8 10:10:47 2012 (16955): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:11:14 2012 (16955): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:11:14 2012 (16955): Child -1 died
Tue May  8 10:11:14 2012 (16955): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:13:00 2012 (16955): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:13:00 2012 (16955): Child -1 died
Tue May  8 10:13:00 2012 (16955): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:13:06 2012 (5505): Child 16923 died
Tue May  8 10:13:12 2012 (16955): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:13:12 2012 (16955): Child -1 died
Tue May  8 10:13:12 2012 (16955): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:13:12 2012 (5505): Child 16955 died
Tue May  8 10:13:17 2012 (5505): Child 17073 died
Tue May  8 10:18:04 2012 (22133): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:18:09 2012 (22133): Child -1 died
Tue May  8 10:18:09 2012 (22133): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:18:40 2012 (22133): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:18:40 2012 (22133): Child -1 died
Tue May  8 10:18:40 2012 (22133): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:18:47 2012 (22133): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:18:47 2012 (22133): Child -1 died
Tue May  8 10:18:47 2012 (22133): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:19:07 2012 (22133): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:19:07 2012 (22133): Child -1 died
Tue May  8 10:19:07 2012 (22133): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:19:13 2012 (5505): Child 22130 died
Tue May  8 10:19:31 2012 (22133): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:19:31 2012 (22133): Child -1 died
Tue May  8 10:19:31 2012 (22133): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:19:31 2012 (5505): Child 22133 died
Tue May  8 10:19:36 2012 (5505): Child 22135 died
Tue May  8 10:22:00 2012 (22205): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:22:01 2012 (22205): Child -1 died
Tue May  8 10:22:01 2012 (22205): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=Binghamton
Tue May  8 10:26:58 2012 (22205): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:27:02 2012 (22205): Child -1 died
Tue May  8 10:27:02 2012 (22205): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:45:18 2012 (22207): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:45:19 2012 (22207): Child -1 died
Tue May  8 10:45:19 2012 (22207): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:46:27 2012 (22207): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:46:28 2012 (22207): Child -1 died
Tue May  8 10:46:28 2012 (22207): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:46:33 2012 (5505): Child 22196 died
Tue May  8 10:46:43 2012 (22207): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:46:43 2012 (22207): Child -1 died
Tue May  8 10:46:43 2012 (22207): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 10:46:43 2012 (5505): Child 22207 died
Tue May  8 10:46:48 2012 (5505): Child 22205 died
Tue May  8 10:48:21 2012 (22724): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 10:48:22 2012 (22724): Child -1 died
Tue May  8 10:48:22 2012 (22724): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 11:02:55 2012 (22724): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 11:03:02 2012 (22724): Child -1 died
Tue May  8 11:03:02 2012 (22724): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 11:03:55 2012 (22726): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 11:03:57 2012 (22726): Child -1 died
Tue May  8 11:03:57 2012 (22726): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 11:17:20 2012 (22726): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 11:17:21 2012 (22726): Child -1 died
Tue May  8 11:17:21 2012 (22726): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 11:17:26 2012 (22718): <font color="blue">WARNING: Could not
retrieve from database:Got error 28 from storage engine</font>
Tue May  8 11:17:26 2012 (5505): Child 22718 died
Tue May  8 11:22:23 2012 (22726): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 11:22:24 2012 (22726): Child -1 died
Tue May  8 11:22:24 2012 (22726): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 11:22:25 2012 (5505): Child 22726 died
Tue May  8 11:22:28 2012 (5505): Child 22724 died
Tue May  8 11:22:29 2012 (23193): <font color="blue">WARNING: Could not
retrieve from database:Got error 28 from storage engine</font>
Tue May  8 11:27:21 2012 (5505): <font color='red'>CRITICAL: Shutting
down</font>
Tue May  8 11:27:21 2012 (5505): Child 23193 died
Tue May  8 11:43:56 2012 (3543): <font color='red'>CRITICAL: ----------
Starting ----------</font>
Tue May  8 12:00:48 2012 (3544): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 12:01:23 2012 (3549): preparing activity log tables for
6M25568f4e8c34e61binghamtona5
Tue May  8 12:01:23 2012 (3549): /home/httpd/perl/parse_activity_log.pl
-course=6M25568f4e8c34e61binghamtona5 -domain=binghamton
Tue May  8 12:13:01 2012 (3543): <font color='red'>CRITICAL: Shutting
down</font>
Tue May  8 12:15:32 2012 (3478): <font color='red'>CRITICAL: ----------
Starting ----------</font>
-------------------------------------------------

Here are a couple of entries from the lonsql_errors file.  The %2e
continue for pages and then there is another entry and then more %2e's.
This continues until the drive is full.

Unable to lock
/home/httpd/lonUsers/binghamton/6/M/2/6M25568f4e8c34e61binghamtona5/activi
ty.log.lock.  Aborting
Error occured during insert.Attempted
INSERT IGNORE INTO 6M25568f4e8c34e61binghamtona5_binghamton_activity
VALUES (723,'2012-4-27
3:44:50',185,'CSTORE','',1,'resource%2eexperimental%2emFe%2eanswername=INT
ERNAL&resource%2eexperimental%2emFe%2eawarddetail=TOO_LONG&ip=128%2e226%2e
62%2e40&resource%2eexperimental%2esolved=incorrect_attempted&resource%2eex
perimental%2eprevious=0&host=binghamtona6&resource%2eexperimental%2emc%2ea
warddetail=SIG_FAIL&resource%2eexperimental%2emc%2esubmission=4%2e084&reso
urce%2eexperimental%2emFe%2esubmission=%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2
e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e
--cut-
%2e&resource%2eexperimental%2eaward=MISSING_ANSWER&resourc
e%2eexperimental%2emc%2eawardmsg=4%3a1%3a3&resource%2eexperimental%2emc%2e
answer
name=INTERNAL&resource%2eexperimental%2emFe%2eawardmsg='),(242,'2012-4-27
3:44:5
0',185,'POST','',1,'HWVAL_Poxa%3d&HWVAL_mFe%3d%2e%2e%2e%2e%2e%2e%2e%2e%2e%
2e%2e%
2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e
%2e%2e
%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%
--cut-
%2e&submitted%3dpart_experimental&HWVAL_mc%3d4%2e084&HWVAL_PFe%3d&submit_e
xperimental%3dSubmit%20Answer&counter%3d505&symb%3duploaded%2fbinghamton%2
f6M25568f4e8c34e61binghamtona5%2fdefault_1292337827%2esequence___3___bingh
amton%2fgonzales%2fPostlab%2fPart%20D%20%2d%20IronOxalato%2eproblem&HWVAL_
PK%3d'),(242,'2012-4-27
3:44:51',185,'POST','',1,'HWVAL_Poxa%3d&HWVAL_mFe%3d%2e%2e%2e%2e%2e%2e%2e%
2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e
%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2e%2
e%2e%2e%2e%2
------------------

So, do I have some corrupt file(s)?  Are there any tools for "fixing"
corrupt files?  Any help
would be appreciated.

Would I be creating any additional problems by making
parse_activity_log.pl non-executable till I figure out what's going on?

Thanks,
Bob Gonzales
Binghamton University
Chemistry Dept

More information about the LON-CAPA-admin mailing list