[LON-CAPA-admin] problems with library server
Michael Dugdale
michael.dugdale at johnabbott.qc.ca
Mon Sep 12 11:28:49 EDT 2011
I had a similar issue on Sunday on our library server, johnabbottl1 running CentOS 5.6 on x86-64.
I was, however, able to ssh in and restart the LON-CAPA services. Everything *seems* to be running okay since then.
Here's the relevant log entry from our logs.
Sun Sep 11 05:10:22 2011 (3555): <font color='red'>CRITICAL: ABNORMAL EXIT. Child 3555 for server johnabbottl1 died through a crash with this error msg->[Corrupted storable string (binary v2.7) at ../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al) line 366.
]</font>
Sun Sep 11 05:10:22 2011 (3555): Famous last words: Sun Sep 11 05:10:22 2011: Catching exception - Sun Sep 11 05:10:22 2011: <font color='red'>CRITICAL: ABNORMAL EXIT. Child 3555 for server johnabbottl1 died through a crash with this error msg->[Corrupted storable string (binary v2.7) at ../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al) line 366.
]</font>
Sun Sep 11 05:10:23 2011 (9824): <font color="green">Successfull ssl authentication with johnabbotta2 </font>
Cheers,
Michael Dugdale
On 2011-09-12, at 11:20 , Robert F Gonzales wrote:
> In the last 20 hours I've had my library server, binghamtonl1, stopped
> responding. Here are is some info:
>
> - it won't respond to a ping
> - it won't bring up the loncapa log in page
> - I can log in via ssh and stop and start loncontrol and httpd.
> Everything seems normal after that.
>
> I browsed through the log files and the only odd things I see are the
> following:
>
> -----------------------------------------------------------------
> lond log
>
> Sun Sep 11 17:33:47 2011 (11965): <font color='red'>CRITICAL: ABNORMAL
> EXIT. Child 11965 for server binghamtonl1 died through a crash with this
> error ms
> g->[Corrupted storable string (binary v2.7) at ../../lib/Storable.pm
> (autosplit into ../../lib/auto/Storable/thaw.al) line 366, <MANAGERS> line
> 9.
> ]</font>
> Sun Sep 11 17:33:47 2011 (11965): Famous last words: Sun Sep 11 17:33:47
> 2011: Catching exception - Sun Sep 11 17:33:47 2011: <font
> color='red'>CRITICAL
> : ABNORMAL EXIT. Child 11965 for server binghamtonl1 died through a crash
> with this error msg->[Corrupted storable string (binary v2.7) at
> ../../lib/Sto
> rable.pm (autosplit into ../../lib/auto/Storable/thaw.al) line 366,
> <MANAGERS> line 9.
> ]</font>
>
> Mon Sep 12 05:10:22 2011 (15439): <font color='red'>CRITICAL: ABNORMAL
> EXIT. Child 15439 for server binghamtonl1 died through a crash with this
> error msg->[Corrupted storable string (binary v2.7) at
> ../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al)
> line 366, <GEN1312> line 1.
> ]</font>
> Mon Sep 12 05:10:22 2011 (15439): Famous last words: Mon Sep 12 05:10:22
> 2011: Catching exception - Mon Sep 12 05:10:22 2011: <font
> color='red'>CRITICAL: ABNORMAL EXIT. Child 15439 for server binghamtonl1
> died through a crash with this error msg->[Corrupted storable string
> (binary v2.7) at ../../lib/Storable.pm (autosplit into
> ../../lib/auto/Storable/thaw.al) line 366, <GEN1312> line 1.
> ]</font>
>
> --------------------------------------------------------------------
> londerrors.log
> /etc/init.d/httpd: line 81: echo: write error: Input/output error
> /etc/rc.d/init.d/functions: line 399: echo: write error: Input/output
> error
> /etc/rc.d/init.d/functions: line 400: echo: write error: Input/output
> error
> /etc/rc.d/init.d/functions: line 401: echo: write error: Input/output
> error
> /etc/rc.d/init.d/functions: line 402: echo: write error: Input/output
> error
> /etc/rc.d/init.d/functions: line 403: echo: write error: Input/output
> error
> /etc/rc.d/init.d/functions: line 404: echo: write error: Input/output
> error
> /etc/rc.d/init.d/functions: line 405: echo: write error: Input/output
> error
> /etc/init.d/httpd: line 90: echo: write error: Input/output error
>
> --------------------------------------------------------------------
>
> Lonnet.log - I edited this and removed some of the similar lines from the
> reboot
>
> Mon Sep 12 05:10:17 2011 (13758): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13005): %homecache is 1272
> Mon Sep 12 05:10:17 2011 (13005): %remembered is 7
> Mon Sep 12 05:10:17 2011 (13005): kicks is 0
> Mon Sep 12 05:10:17 2011 (13005): hits is 82903
> Mon Sep 12 05:10:17 2011 (13005): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13755): hits is 16835
> Mon Sep 12 05:10:17 2011 (13755): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13736): hits is 24
> Mon Sep 12 05:10:17 2011 (13736): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13628): hits is 0
> Mon Sep 12 05:10:17 2011 (13215): %badServerCache is 7
> Mon Sep 12 05:10:17 2011 (13392): kicks is 0
> Mon Sep 12 05:10:17 2011 (13628): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13392): hits is 47119
> Mon Sep 12 05:10:17 2011 (13628): Shutting down
> Mon Sep 12 05:10:17 2011 (13392): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13613): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13621): Starting Shut down
> Mon Sep 12 05:10:17 2011 (13621): %badServerCache is 7
> Mon Sep 12 05:10:17 2011 (13238): %homecache is 139
> Mon Sep 12 05:10:17 2011 (13621): %homecache is 65
> Mon Sep 12 05:10:17 2011 (13621): %remembered is 7
> Mon Sep 12 05:10:17 2011 (13238): %remembered is 7
> Mon Sep 12 05:10:17 2011 (13621): kicks is 0
> Mon Sep 12 05:10:17 2011 (13238): kicks is 0
> Mon Sep 12 05:10:17 2011 (13621): hits is 0
> Mon Sep 12 05:10:17 2011 (13238): hits is 1325
> Mon Sep 12 05:10:17 2011 (13621): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13238): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13621): Shutting down
> Mon Sep 12 05:10:17 2011 (13239): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13624): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13624): Shutting down
> Mon Sep 12 05:10:17 2011 (13215): %homecache is 562
> Mon Sep 12 05:10:17 2011 (13215): %remembered is 7
> Mon Sep 12 05:10:17 2011 (13215): kicks is 0
> Mon Sep 12 05:10:17 2011 (13215): hits is 10868
> Mon Sep 12 05:10:17 2011 (13215): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13215): Shutting down
> Mon Sep 12 05:10:17 2011 (13305): %remembered is 7
> Mon Sep 12 05:10:17 2011 (13305): kicks is 0
> Mon Sep 12 05:10:17 2011 (13305): hits is 1056
> Mon Sep 12 05:10:17 2011 (13305): Flushing log buffers
> Mon Sep 12 05:10:17 2011 (13305): Shutting down
> Mon Sep 12 05:10:22 2011 (12991): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 05:10:22 2011 (12991): Detected 64bit platform (1)
> Mon Sep 12 05:10:22 2011 (12991): <span style='color:yellow;'>INFO: Read
> file types</span>
> Mon Sep 12 05:11:00 2011 (15517): Trying to reconnect lonc for pittphyast1
> (nplq1.phyast.pitt.edu)
> Mon Sep 12 05:11:41 2011 (15533): Trying to reconnect lonc for authorl1
> (educog.com)
> Mon Sep 12 05:16:44 2011 (15559): Trying to reconnect lonc for
> mariettahsl1 (capa4.phy.ohiou.edu)
> Mon Sep 12 05:17:08 2011 (15567): Trying to reconnect lonc for authorl1
> (educog.com)
> Mon Sep 12 05:18:03 2011 (15601): Trying to reconnect lonc for fsua4
> (loncapa4.fsu.edu)
> Mon Sep 12 07:09:36 2011 (16078): User anord1 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 08:11:00 2011 (16281): Trying to reconnect lonc for asul1
> (andes.eas.asu.edu)
> Mon Sep 12 08:15:57 2011 (16301): User afranco1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 08:29:45 2011 (16351): Trying to reconnect lonc for jmul1
> (lc.cit.jmu.edu)
> Mon Sep 12 08:32:45 2011 (16350): Trying to reconnect lonc
> Mon Sep 12 08:32:45 2011 (16350): lonc at pid 12858 responding, sending
> USR1
> Mon Sep 12 08:32:46 2011 (16350): User ppatel at binghamton is unknown in
> authenticate
> Mon Sep 12 08:32:56 2011 (16345): User ppatel22 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 08:35:26 2011 (16367): User ppatel22 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 08:46:34 2011 (16411): Trying to reconnect lonc for jmul1
> (lc.cit.jmu.edu)
> Mon Sep 12 08:58:36 2011 (16390): User rkelly6 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 09:00:04 2011 (16389): Flushing log buffers
> Mon Sep 12 09:00:04 2011 (16399): Flushing log buffers
> Mon Sep 12 09:00:06 2011 (16394): Flushing log buffers
> Mon Sep 12 09:00:06 2011 (16385): Flushing log buffers
> Mon Sep 12 09:00:08 2011 (16439): Flushing log buffers
> Mon Sep 12 09:07:12 2011 (16501): User gkim21 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 09:09:03 2011 (16452): Flushing log buffers
> Mon Sep 12 09:13:46 2011 (16506): User ajimene7 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 09:14:01 2011 (16450): Trying to reconnect lonc
> Mon Sep 12 09:14:01 2011 (16450): lonc at pid 12858 responding, sending
> USR1
> Mon Sep 12 09:14:02 2011 (16450): User ajimene7 at binghamton.edu at
> binghamton is unknown in authenticate
> Mon Sep 12 09:14:34 2011 (16449): User ajimene7 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 09:32:36 2011 (16389): User sgallo3 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 09:32:44 2011 (16452): User agabrie4 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 09:34:31 2011 (16389): User agabrie4 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 09:34:39 2011 (16451): User sroccar1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 09:38:05 2011 (16639): User cgerard1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 09:39:21 2011 (16640): Flushing log buffers
> Mon Sep 12 09:40:10 2011 (16639): User rcordov1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 09:40:39 2011 (16646): User jgagnon2 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 09:42:55 2011 (16657): User jlee231 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 09:47:09 2011 (16653): User arogoze1 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 10:11:51 2011 (16700): User ctate1 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 10:15:26 2011 (16792): User gberra1 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 10:16:38 2011 (16726): Flushing log buffers
> Mon Sep 12 10:17:12 2011 (16805): User mvitovs1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:18:38 2011 (16731): Flushing log buffers
> Mon Sep 12 10:22:53 2011 (16731): User dwilli21 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:23:02 2011 (16831): User wcarter1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:23:36 2011 (16817): User jkim179 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 10:26:21 2011 (16827): User jgelman1 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 10:27:07 2011 (16839): User cminass2 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:29:50 2011 (16839): User sbertra1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:30:01 2011 (16830): Flushing log buffers
> Mon Sep 12 10:52:55 2011 (16839): Starting Shut down
> Mon Sep 12 10:52:55 2011 (16839): %badServerCache is 7
> Mon Sep 12 10:52:55 2011 (16839): %homecache is 307
> Mon Sep 12 10:52:55 2011 (16297): Starting Shut down
> Mon Sep 12 10:52:55 2011 (16839): %remembered is 7
> Mon Sep 12 10:52:55 2011 (16839): kicks is 0
> Mon Sep 12 10:52:55 2011 (16839): hits is 12800
> Mon Sep 12 10:52:55 2011 (16698): Starting Shut down
> Mon Sep 12 10:52:55 2011 (16839): Flushing log buffers
> Mon Sep 12 10:52:55 2011 (16698): %badServerCache is 7
> Mon Sep 12 10:52:55 2011 (16698): %homecache is 159
> Mon Sep 12 10:52:55 2011 (16698): %remembered is 7
> Mon Sep 12 10:52:55 2011 (16698): kicks is 0
> Mon Sep 12 10:52:55 2011 (16698): hits is 2371
>
> Edited out similar lines here at the reboot
>
> Mon Sep 12 10:52:56 2011 (16658): Shutting down
> Mon Sep 12 10:52:56 2011 (16653): Shutting down
> Mon Sep 12 10:52:56 2011 (16646): Shutting down
> Mon Sep 12 10:52:57 2011 (16910): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
> Mon Sep 12 10:53:13 2011 (16922): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:53:13 2011 (16922): Detected 64bit platform (1)
> Mon Sep 12 10:53:26 2011 (17061): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:53:26 2011 (17061): Detected 64bit platform (1)
> Mon Sep 12 10:53:27 2011 (17061): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
> Mon Sep 12 10:53:40 2011 (17061): Name shs-dunk-lin.shs.sarasota.k12.fl.us
> no IP found
> Mon Sep 12 10:53:40 2011 (17061): Name theoryx2.uwinnipeg.ca no IP found
> Mon Sep 12 10:53:53 2011 (17061): Name lyra.mc21.fee.unicamp.br no IP
> found
> Mon Sep 12 10:53:53 2011 (17061): Name lon-capa.colorado.edu no IP found
> Mon Sep 12 10:53:53 2011 (17061): Name nyul1.physics.nyu.edu no IP found
> Mon Sep 12 10:53:53 2011 (17061): Name loncapa.physics.mcgill.ca no IP
> found
> Mon Sep 12 10:53:53 2011 (17061): Name u140.n100.queensu.ca no IP found
> Mon Sep 12 10:53:58 2011 (17061): caching of id -> iphost%3aiphost failed
> Mon Sep 12 10:53:58 2011 (17199): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:53:58 2011 (17199): Detected 64bit platform (1)
> Mon Sep 12 10:53:59 2011 (17199): <span style='color:yellow;'>INFO: Read
> file types</span>
> Mon Sep 12 10:53:59 2011 (17199): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
> Mon Sep 12 10:54:00 2011 (17199): caching of id ->
> dns%3a%2fadm%2fdns%2fdomain failed
> Mon Sep 12 10:54:00 2011 (17199): Name shs-dunk-lin.shs.sarasota.k12.fl.us
> no IP found
> Mon Sep 12 10:54:00 2011 (17199): Name theoryx2.uwinnipeg.ca no IP found
> Mon Sep 12 10:54:05 2011 (17199): Name lyra.mc21.fee.unicamp.br no IP
> found
> Mon Sep 12 10:54:05 2011 (17199): Name lon-capa.colorado.edu no IP found
> Mon Sep 12 10:54:05 2011 (17199): Name nyul1.physics.nyu.edu no IP found
> Mon Sep 12 10:54:05 2011 (17199): Name loncapa.physics.mcgill.ca no IP
> found
> Mon Sep 12 10:54:05 2011 (17199): Name u140.n100.queensu.ca no IP found
> Mon Sep 12 10:54:05 2011 (17199): caching of id -> iphost%3aiphost failed
> Mon Sep 12 10:54:08 2011 (17214): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:54:08 2011 (17214): Detected 64bit platform (1)
> Mon Sep 12 10:54:11 2011 (17230): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:54:11 2011 (17230): Detected 64bit platform (1)
> Mon Sep 12 10:54:11 2011 (17231): Name shs-dunk-lin.shs.sarasota.k12.fl.us
> no IP found
> Mon Sep 12 10:54:11 2011 (17231): Name theoryx2.uwinnipeg.ca no IP found
> Mon Sep 12 10:54:16 2011 (17231): Name lyra.mc21.fee.unicamp.br no IP
> found
> Mon Sep 12 10:54:16 2011 (17231): Name lon-capa.colorado.edu no IP found
> Mon Sep 12 10:54:16 2011 (17231): Name nyul1.physics.nyu.edu no IP found
> Mon Sep 12 10:54:16 2011 (17231): Name loncapa.physics.mcgill.ca no IP
> found
> Mon Sep 12 10:54:16 2011 (17231): Name u140.n100.queensu.ca no IP found
> Mon Sep 12 10:54:27 2011 (17370): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:54:27 2011 (17370): Detected 64bit platform (1)
> Mon Sep 12 10:54:27 2011 (17370): <span style='color:yellow;'>INFO: Read
> file types</span>
> Mon Sep 12 10:54:30 2011 (17373): <font color="yellow">INFO: Read
> configuration</font>
> Mon Sep 12 10:54:30 2011 (17373): Detected 64bit platform (1)
> Mon Sep 12 10:54:30 2011 (17373): <span style='color:yellow;'>INFO: Read
> file types</span>
> Mon Sep 12 10:54:57 2011 (17379): User gonzales at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:55:11 2011 (17385): User slazar1 at binghamton rejected by
> binghamtonl1
> Mon Sep 12 10:55:24 2011 (17384): Flushing log buffers
> Mon Sep 12 10:55:36 2011 (17385): User wcarter1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:55:41 2011 (17378): Trying to reconnect lonc
> Mon Sep 12 10:55:41 2011 (17378): lonc at pid 17238 responding, sending
> USR1
> Mon Sep 12 10:55:43 2011 (17378): User vhall1 at elps is unknown in
> authenticate
> Mon Sep 12 10:55:52 2011 (17385): User slazar1 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 10:56:00 2011 (17381): Trying to reconnect lonc
> Mon Sep 12 10:56:00 2011 (17381): lonc at pid 17238 responding, sending
> USR1
> Mon Sep 12 10:56:01 2011 (17381): User vhall1 at elps is unknown in
> authenticate
> Mon Sep 12 10:56:18 2011 (17381): User jgelman1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 10:56:18 2011 (17385): User vhall1 at binghamton authorized by
> binghamtonl1
> Mon Sep 12 10:58:47 2011 (17383): Flushing log buffers
> Mon Sep 12 11:00:56 2011 (17385): Flushing log buffers
> Mon Sep 12 11:01:06 2011 (17386): Flushing log buffers
> Mon Sep 12 11:02:39 2011 (17379): Flushing log buffers
> Mon Sep 12 11:02:43 2011 (17378): Flushing log buffers
> Mon Sep 12 11:02:44 2011 (17381): Flushing log buffers
> Mon Sep 12 11:02:49 2011 (17384): Flushing log buffers
> Mon Sep 12 11:03:11 2011 (17383): User bmather1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 11:03:21 2011 (17383): Flushing log buffers
> Mon Sep 12 11:03:21 2011 (17377): Flushing log buffers
> Mon Sep 12 11:04:44 2011 (17386): Flushing log buffers
> Mon Sep 12 11:05:27 2011 (17378): Trying to reconnect lonc
> Mon Sep 12 11:05:27 2011 (17378): lonc at pid 17238 responding, sending
> USR1
> Mon Sep 12 11:05:28 2011 (17378): User jkrok1 at elps is unknown in
> authenticate
> Mon Sep 12 11:06:00 2011 (17377): User mvitovs1 at binghamton authorized
> by binghamtonl1
> Mon Sep 12 11:06:32 2011 (17383): Trying to reconnect lonc
> Mon Sep 12 11:06:32 2011 (17383): lonc at pid 17238 responding, sending
> USR1
> Mon Sep 12 11:06:33 2011 (17383): User jkrok1 at elps is unknown in
> authenticate
> Mon Sep 12 11:07:37 2011 (17385): Flushing log buffers
> Mon Sep 12 11:11:24 2011 (17386): Flushing log buffers
> Mon Sep 12 11:12:40 2011 (17384): Flushing log buffers
> Mon Sep 12 11:12:59 2011 (17379): Flushing log buffers
> Mon Sep 12 11:13:41 2011 (17377): Flushing log buffers
> Mon Sep 12 11:14:25 2011 (17383): Flushing log buffers
> Mon Sep 12 11:14:27 2011 (17381): Flushing log buffers
> Mon Sep 12 11:14:47 2011 (17378): Flushing log buffers
> Mon Sep 12 11:15:45 2011 (17384): Flushing log buffers
> --------------------------------------------------------------------------
> ----
>
>
> Any ideas on what to look for would be appreciated!
>
> Bob Gonzales
> Binghamton University
> Chemistry Dept
> rgonzal at binghamton.edu
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin
>
More information about the LON-CAPA-admin
mailing list