[LON-CAPA-admin] problems with library server
Robert F Gonzales
rgonzal at binghamton.edu
Mon Sep 12 11:20:39 EDT 2011
In the last 20 hours I've had my library server, binghamtonl1, stopped
responding. Here are is some info:
- it won't respond to a ping
- it won't bring up the loncapa log in page
- I can log in via ssh and stop and start loncontrol and httpd.
Everything seems normal after that.
I browsed through the log files and the only odd things I see are the
following:
-----------------------------------------------------------------
lond log
Sun Sep 11 17:33:47 2011 (11965): <font color='red'>CRITICAL: ABNORMAL
EXIT. Child 11965 for server binghamtonl1 died through a crash with this
error ms
g->[Corrupted storable string (binary v2.7) at ../../lib/Storable.pm
(autosplit into ../../lib/auto/Storable/thaw.al) line 366, <MANAGERS> line
9.
]</font>
Sun Sep 11 17:33:47 2011 (11965): Famous last words: Sun Sep 11 17:33:47
2011: Catching exception - Sun Sep 11 17:33:47 2011: <font
color='red'>CRITICAL
: ABNORMAL EXIT. Child 11965 for server binghamtonl1 died through a crash
with this error msg->[Corrupted storable string (binary v2.7) at
../../lib/Sto
rable.pm (autosplit into ../../lib/auto/Storable/thaw.al) line 366,
<MANAGERS> line 9.
]</font>
Mon Sep 12 05:10:22 2011 (15439): <font color='red'>CRITICAL: ABNORMAL
EXIT. Child 15439 for server binghamtonl1 died through a crash with this
error msg->[Corrupted storable string (binary v2.7) at
../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al)
line 366, <GEN1312> line 1.
]</font>
Mon Sep 12 05:10:22 2011 (15439): Famous last words: Mon Sep 12 05:10:22
2011: Catching exception - Mon Sep 12 05:10:22 2011: <font
color='red'>CRITICAL: ABNORMAL EXIT. Child 15439 for server binghamtonl1
died through a crash with this error msg->[Corrupted storable string
(binary v2.7) at ../../lib/Storable.pm (autosplit into
../../lib/auto/Storable/thaw.al) line 366, <GEN1312> line 1.
]</font>
--------------------------------------------------------------------
londerrors.log
/etc/init.d/httpd: line 81: echo: write error: Input/output error
/etc/rc.d/init.d/functions: line 399: echo: write error: Input/output
error
/etc/rc.d/init.d/functions: line 400: echo: write error: Input/output
error
/etc/rc.d/init.d/functions: line 401: echo: write error: Input/output
error
/etc/rc.d/init.d/functions: line 402: echo: write error: Input/output
error
/etc/rc.d/init.d/functions: line 403: echo: write error: Input/output
error
/etc/rc.d/init.d/functions: line 404: echo: write error: Input/output
error
/etc/rc.d/init.d/functions: line 405: echo: write error: Input/output
error
/etc/init.d/httpd: line 90: echo: write error: Input/output error
--------------------------------------------------------------------
Lonnet.log - I edited this and removed some of the similar lines from the
reboot
Mon Sep 12 05:10:17 2011 (13758): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13005): %homecache is 1272
Mon Sep 12 05:10:17 2011 (13005): %remembered is 7
Mon Sep 12 05:10:17 2011 (13005): kicks is 0
Mon Sep 12 05:10:17 2011 (13005): hits is 82903
Mon Sep 12 05:10:17 2011 (13005): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13755): hits is 16835
Mon Sep 12 05:10:17 2011 (13755): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13736): hits is 24
Mon Sep 12 05:10:17 2011 (13736): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13628): hits is 0
Mon Sep 12 05:10:17 2011 (13215): %badServerCache is 7
Mon Sep 12 05:10:17 2011 (13392): kicks is 0
Mon Sep 12 05:10:17 2011 (13628): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13392): hits is 47119
Mon Sep 12 05:10:17 2011 (13628): Shutting down
Mon Sep 12 05:10:17 2011 (13392): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13613): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13621): Starting Shut down
Mon Sep 12 05:10:17 2011 (13621): %badServerCache is 7
Mon Sep 12 05:10:17 2011 (13238): %homecache is 139
Mon Sep 12 05:10:17 2011 (13621): %homecache is 65
Mon Sep 12 05:10:17 2011 (13621): %remembered is 7
Mon Sep 12 05:10:17 2011 (13238): %remembered is 7
Mon Sep 12 05:10:17 2011 (13621): kicks is 0
Mon Sep 12 05:10:17 2011 (13238): kicks is 0
Mon Sep 12 05:10:17 2011 (13621): hits is 0
Mon Sep 12 05:10:17 2011 (13238): hits is 1325
Mon Sep 12 05:10:17 2011 (13621): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13238): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13621): Shutting down
Mon Sep 12 05:10:17 2011 (13239): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13624): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13624): Shutting down
Mon Sep 12 05:10:17 2011 (13215): %homecache is 562
Mon Sep 12 05:10:17 2011 (13215): %remembered is 7
Mon Sep 12 05:10:17 2011 (13215): kicks is 0
Mon Sep 12 05:10:17 2011 (13215): hits is 10868
Mon Sep 12 05:10:17 2011 (13215): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13215): Shutting down
Mon Sep 12 05:10:17 2011 (13305): %remembered is 7
Mon Sep 12 05:10:17 2011 (13305): kicks is 0
Mon Sep 12 05:10:17 2011 (13305): hits is 1056
Mon Sep 12 05:10:17 2011 (13305): Flushing log buffers
Mon Sep 12 05:10:17 2011 (13305): Shutting down
Mon Sep 12 05:10:22 2011 (12991): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 05:10:22 2011 (12991): Detected 64bit platform (1)
Mon Sep 12 05:10:22 2011 (12991): <span style='color:yellow;'>INFO: Read
file types</span>
Mon Sep 12 05:11:00 2011 (15517): Trying to reconnect lonc for pittphyast1
(nplq1.phyast.pitt.edu)
Mon Sep 12 05:11:41 2011 (15533): Trying to reconnect lonc for authorl1
(educog.com)
Mon Sep 12 05:16:44 2011 (15559): Trying to reconnect lonc for
mariettahsl1 (capa4.phy.ohiou.edu)
Mon Sep 12 05:17:08 2011 (15567): Trying to reconnect lonc for authorl1
(educog.com)
Mon Sep 12 05:18:03 2011 (15601): Trying to reconnect lonc for fsua4
(loncapa4.fsu.edu)
Mon Sep 12 07:09:36 2011 (16078): User anord1 at binghamton authorized by
binghamtonl1
Mon Sep 12 08:11:00 2011 (16281): Trying to reconnect lonc for asul1
(andes.eas.asu.edu)
Mon Sep 12 08:15:57 2011 (16301): User afranco1 at binghamton authorized
by binghamtonl1
Mon Sep 12 08:29:45 2011 (16351): Trying to reconnect lonc for jmul1
(lc.cit.jmu.edu)
Mon Sep 12 08:32:45 2011 (16350): Trying to reconnect lonc
Mon Sep 12 08:32:45 2011 (16350): lonc at pid 12858 responding, sending
USR1
Mon Sep 12 08:32:46 2011 (16350): User ppatel at binghamton is unknown in
authenticate
Mon Sep 12 08:32:56 2011 (16345): User ppatel22 at binghamton rejected by
binghamtonl1
Mon Sep 12 08:35:26 2011 (16367): User ppatel22 at binghamton rejected by
binghamtonl1
Mon Sep 12 08:46:34 2011 (16411): Trying to reconnect lonc for jmul1
(lc.cit.jmu.edu)
Mon Sep 12 08:58:36 2011 (16390): User rkelly6 at binghamton authorized by
binghamtonl1
Mon Sep 12 09:00:04 2011 (16389): Flushing log buffers
Mon Sep 12 09:00:04 2011 (16399): Flushing log buffers
Mon Sep 12 09:00:06 2011 (16394): Flushing log buffers
Mon Sep 12 09:00:06 2011 (16385): Flushing log buffers
Mon Sep 12 09:00:08 2011 (16439): Flushing log buffers
Mon Sep 12 09:07:12 2011 (16501): User gkim21 at binghamton authorized by
binghamtonl1
Mon Sep 12 09:09:03 2011 (16452): Flushing log buffers
Mon Sep 12 09:13:46 2011 (16506): User ajimene7 at binghamton rejected by
binghamtonl1
Mon Sep 12 09:14:01 2011 (16450): Trying to reconnect lonc
Mon Sep 12 09:14:01 2011 (16450): lonc at pid 12858 responding, sending
USR1
Mon Sep 12 09:14:02 2011 (16450): User ajimene7 at binghamton.edu at
binghamton is unknown in authenticate
Mon Sep 12 09:14:34 2011 (16449): User ajimene7 at binghamton authorized
by binghamtonl1
Mon Sep 12 09:32:36 2011 (16389): User sgallo3 at binghamton authorized by
binghamtonl1
Mon Sep 12 09:32:44 2011 (16452): User agabrie4 at binghamton rejected by
binghamtonl1
Mon Sep 12 09:34:31 2011 (16389): User agabrie4 at binghamton authorized
by binghamtonl1
Mon Sep 12 09:34:39 2011 (16451): User sroccar1 at binghamton authorized
by binghamtonl1
Mon Sep 12 09:38:05 2011 (16639): User cgerard1 at binghamton authorized
by binghamtonl1
Mon Sep 12 09:39:21 2011 (16640): Flushing log buffers
Mon Sep 12 09:40:10 2011 (16639): User rcordov1 at binghamton authorized
by binghamtonl1
Mon Sep 12 09:40:39 2011 (16646): User jgagnon2 at binghamton rejected by
binghamtonl1
Mon Sep 12 09:42:55 2011 (16657): User jlee231 at binghamton authorized by
binghamtonl1
Mon Sep 12 09:47:09 2011 (16653): User arogoze1 at binghamton rejected by
binghamtonl1
Mon Sep 12 10:11:51 2011 (16700): User ctate1 at binghamton authorized by
binghamtonl1
Mon Sep 12 10:15:26 2011 (16792): User gberra1 at binghamton authorized by
binghamtonl1
Mon Sep 12 10:16:38 2011 (16726): Flushing log buffers
Mon Sep 12 10:17:12 2011 (16805): User mvitovs1 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:18:38 2011 (16731): Flushing log buffers
Mon Sep 12 10:22:53 2011 (16731): User dwilli21 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:23:02 2011 (16831): User wcarter1 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:23:36 2011 (16817): User jkim179 at binghamton authorized by
binghamtonl1
Mon Sep 12 10:26:21 2011 (16827): User jgelman1 at binghamton rejected by
binghamtonl1
Mon Sep 12 10:27:07 2011 (16839): User cminass2 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:29:50 2011 (16839): User sbertra1 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:30:01 2011 (16830): Flushing log buffers
Mon Sep 12 10:52:55 2011 (16839): Starting Shut down
Mon Sep 12 10:52:55 2011 (16839): %badServerCache is 7
Mon Sep 12 10:52:55 2011 (16839): %homecache is 307
Mon Sep 12 10:52:55 2011 (16297): Starting Shut down
Mon Sep 12 10:52:55 2011 (16839): %remembered is 7
Mon Sep 12 10:52:55 2011 (16839): kicks is 0
Mon Sep 12 10:52:55 2011 (16839): hits is 12800
Mon Sep 12 10:52:55 2011 (16698): Starting Shut down
Mon Sep 12 10:52:55 2011 (16839): Flushing log buffers
Mon Sep 12 10:52:55 2011 (16698): %badServerCache is 7
Mon Sep 12 10:52:55 2011 (16698): %homecache is 159
Mon Sep 12 10:52:55 2011 (16698): %remembered is 7
Mon Sep 12 10:52:55 2011 (16698): kicks is 0
Mon Sep 12 10:52:55 2011 (16698): hits is 2371
Edited out similar lines here at the reboot
Mon Sep 12 10:52:56 2011 (16658): Shutting down
Mon Sep 12 10:52:56 2011 (16653): Shutting down
Mon Sep 12 10:52:56 2011 (16646): Shutting down
Mon Sep 12 10:52:57 2011 (16910): caching of id ->
dns%3a%2fadm%2fdns%2fhosts failed
Mon Sep 12 10:53:13 2011 (16922): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:53:13 2011 (16922): Detected 64bit platform (1)
Mon Sep 12 10:53:26 2011 (17061): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:53:26 2011 (17061): Detected 64bit platform (1)
Mon Sep 12 10:53:27 2011 (17061): caching of id ->
dns%3a%2fadm%2fdns%2fhosts failed
Mon Sep 12 10:53:40 2011 (17061): Name shs-dunk-lin.shs.sarasota.k12.fl.us
no IP found
Mon Sep 12 10:53:40 2011 (17061): Name theoryx2.uwinnipeg.ca no IP found
Mon Sep 12 10:53:53 2011 (17061): Name lyra.mc21.fee.unicamp.br no IP
found
Mon Sep 12 10:53:53 2011 (17061): Name lon-capa.colorado.edu no IP found
Mon Sep 12 10:53:53 2011 (17061): Name nyul1.physics.nyu.edu no IP found
Mon Sep 12 10:53:53 2011 (17061): Name loncapa.physics.mcgill.ca no IP
found
Mon Sep 12 10:53:53 2011 (17061): Name u140.n100.queensu.ca no IP found
Mon Sep 12 10:53:58 2011 (17061): caching of id -> iphost%3aiphost failed
Mon Sep 12 10:53:58 2011 (17199): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:53:58 2011 (17199): Detected 64bit platform (1)
Mon Sep 12 10:53:59 2011 (17199): <span style='color:yellow;'>INFO: Read
file types</span>
Mon Sep 12 10:53:59 2011 (17199): caching of id ->
dns%3a%2fadm%2fdns%2fhosts failed
Mon Sep 12 10:54:00 2011 (17199): caching of id ->
dns%3a%2fadm%2fdns%2fdomain failed
Mon Sep 12 10:54:00 2011 (17199): Name shs-dunk-lin.shs.sarasota.k12.fl.us
no IP found
Mon Sep 12 10:54:00 2011 (17199): Name theoryx2.uwinnipeg.ca no IP found
Mon Sep 12 10:54:05 2011 (17199): Name lyra.mc21.fee.unicamp.br no IP
found
Mon Sep 12 10:54:05 2011 (17199): Name lon-capa.colorado.edu no IP found
Mon Sep 12 10:54:05 2011 (17199): Name nyul1.physics.nyu.edu no IP found
Mon Sep 12 10:54:05 2011 (17199): Name loncapa.physics.mcgill.ca no IP
found
Mon Sep 12 10:54:05 2011 (17199): Name u140.n100.queensu.ca no IP found
Mon Sep 12 10:54:05 2011 (17199): caching of id -> iphost%3aiphost failed
Mon Sep 12 10:54:08 2011 (17214): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:54:08 2011 (17214): Detected 64bit platform (1)
Mon Sep 12 10:54:11 2011 (17230): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:54:11 2011 (17230): Detected 64bit platform (1)
Mon Sep 12 10:54:11 2011 (17231): Name shs-dunk-lin.shs.sarasota.k12.fl.us
no IP found
Mon Sep 12 10:54:11 2011 (17231): Name theoryx2.uwinnipeg.ca no IP found
Mon Sep 12 10:54:16 2011 (17231): Name lyra.mc21.fee.unicamp.br no IP
found
Mon Sep 12 10:54:16 2011 (17231): Name lon-capa.colorado.edu no IP found
Mon Sep 12 10:54:16 2011 (17231): Name nyul1.physics.nyu.edu no IP found
Mon Sep 12 10:54:16 2011 (17231): Name loncapa.physics.mcgill.ca no IP
found
Mon Sep 12 10:54:16 2011 (17231): Name u140.n100.queensu.ca no IP found
Mon Sep 12 10:54:27 2011 (17370): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:54:27 2011 (17370): Detected 64bit platform (1)
Mon Sep 12 10:54:27 2011 (17370): <span style='color:yellow;'>INFO: Read
file types</span>
Mon Sep 12 10:54:30 2011 (17373): <font color="yellow">INFO: Read
configuration</font>
Mon Sep 12 10:54:30 2011 (17373): Detected 64bit platform (1)
Mon Sep 12 10:54:30 2011 (17373): <span style='color:yellow;'>INFO: Read
file types</span>
Mon Sep 12 10:54:57 2011 (17379): User gonzales at binghamton authorized
by binghamtonl1
Mon Sep 12 10:55:11 2011 (17385): User slazar1 at binghamton rejected by
binghamtonl1
Mon Sep 12 10:55:24 2011 (17384): Flushing log buffers
Mon Sep 12 10:55:36 2011 (17385): User wcarter1 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:55:41 2011 (17378): Trying to reconnect lonc
Mon Sep 12 10:55:41 2011 (17378): lonc at pid 17238 responding, sending
USR1
Mon Sep 12 10:55:43 2011 (17378): User vhall1 at elps is unknown in
authenticate
Mon Sep 12 10:55:52 2011 (17385): User slazar1 at binghamton authorized by
binghamtonl1
Mon Sep 12 10:56:00 2011 (17381): Trying to reconnect lonc
Mon Sep 12 10:56:00 2011 (17381): lonc at pid 17238 responding, sending
USR1
Mon Sep 12 10:56:01 2011 (17381): User vhall1 at elps is unknown in
authenticate
Mon Sep 12 10:56:18 2011 (17381): User jgelman1 at binghamton authorized
by binghamtonl1
Mon Sep 12 10:56:18 2011 (17385): User vhall1 at binghamton authorized by
binghamtonl1
Mon Sep 12 10:58:47 2011 (17383): Flushing log buffers
Mon Sep 12 11:00:56 2011 (17385): Flushing log buffers
Mon Sep 12 11:01:06 2011 (17386): Flushing log buffers
Mon Sep 12 11:02:39 2011 (17379): Flushing log buffers
Mon Sep 12 11:02:43 2011 (17378): Flushing log buffers
Mon Sep 12 11:02:44 2011 (17381): Flushing log buffers
Mon Sep 12 11:02:49 2011 (17384): Flushing log buffers
Mon Sep 12 11:03:11 2011 (17383): User bmather1 at binghamton authorized
by binghamtonl1
Mon Sep 12 11:03:21 2011 (17383): Flushing log buffers
Mon Sep 12 11:03:21 2011 (17377): Flushing log buffers
Mon Sep 12 11:04:44 2011 (17386): Flushing log buffers
Mon Sep 12 11:05:27 2011 (17378): Trying to reconnect lonc
Mon Sep 12 11:05:27 2011 (17378): lonc at pid 17238 responding, sending
USR1
Mon Sep 12 11:05:28 2011 (17378): User jkrok1 at elps is unknown in
authenticate
Mon Sep 12 11:06:00 2011 (17377): User mvitovs1 at binghamton authorized
by binghamtonl1
Mon Sep 12 11:06:32 2011 (17383): Trying to reconnect lonc
Mon Sep 12 11:06:32 2011 (17383): lonc at pid 17238 responding, sending
USR1
Mon Sep 12 11:06:33 2011 (17383): User jkrok1 at elps is unknown in
authenticate
Mon Sep 12 11:07:37 2011 (17385): Flushing log buffers
Mon Sep 12 11:11:24 2011 (17386): Flushing log buffers
Mon Sep 12 11:12:40 2011 (17384): Flushing log buffers
Mon Sep 12 11:12:59 2011 (17379): Flushing log buffers
Mon Sep 12 11:13:41 2011 (17377): Flushing log buffers
Mon Sep 12 11:14:25 2011 (17383): Flushing log buffers
Mon Sep 12 11:14:27 2011 (17381): Flushing log buffers
Mon Sep 12 11:14:47 2011 (17378): Flushing log buffers
Mon Sep 12 11:15:45 2011 (17384): Flushing log buffers
--------------------------------------------------------------------------
----
Any ideas on what to look for would be appreciated!
Bob Gonzales
Binghamton University
Chemistry Dept
rgonzal at binghamton.edu
More information about the LON-CAPA-admin
mailing list