[LON-CAPA-admin] lonc connection dropping

[lucasm at ohiou.edu]

Hi,

I'm running 1.3.3 and have a recurring problem with connections dropping 
out every once in a while. Last night around 7PM oucapa10 dropped it's 
connection to oucapa2 (ping capa2 from capa10 comes up con lost) and it 
did not pick back up. It did not lose it's connection to capa7.

(1) Are people having trouble with the connections?

(2) I'm including stuff from lonc_errors and lonc.log below. Can anyone 
help with this?

This problem is approaching the level that I can't trust the system (we're 
talking a dropout like this at least once every two or three weeks - this 
one I didn't catch for 12 hours since I didn't check e-mail last night).

I've created a small script that returns the result of a LC ping using 
wget and creates output suitable for MRTG. The problem here is that the 
wget seems to fail sometimes (so the script fails on a con lost, but also 
on a failure of wget). I'll have to fix that.

Later,
Mark



In lonc_errors I find the following at the time of the crash:

Event: trapped error in `Connection to lonc client 9921': Can't locate 
object me
thod "Shutdown" via package "LondConnection=HASH(0x9536eb4)" (perhaps you 
forgot
 to load "LondConnection=HASH(0x9536eb4)"?) at /home/httpd/perl/loncnew 
line 746
.
Event: trapped error in `Connection to lonc client 9922': Event 
'Connection to l
onc client 0': GLOB(0x952efec) isn't a valid IO at 
/home/httpd/perl/loncnew line
 641
Event: trapped error in `Connection to lonc client 9923': Event 
'Connection to l
onc client 0': GLOB(0x952efec) isn't a valid IO at 
/home/httpd/perl/loncnew line
 641
Event: trapped error in `Connection to lonc client 9924': Event 
'Connection to l
onc client 0': GLOB(0x952efec) isn't a valid IO at 
/home/httpd/perl/loncnew line
 641
Event: trapped error in `Connection to lonc client 9925': Event 
'Connection to l
onc client 0': GLOB(0x952efec) isn't a valid IO at 
/home/httpd/perl/loncnew line
 641
Event: trapped error in ?: Event 'Connection to lonc client 0': 
GLOB(0x952efec)
isn't a valid IO at /home/httpd/perl/loncnew line 641
Attempt to free unreferenced scalar: SV 0x953febc, Perl interpreter: 
0x90dd008 a
t /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Event.pm line 
162.
Attempt to free unreferenced scalar: SV 0x953febc, Perl interpreter: 
0x90dd008 a
t /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/Event.pm line 
162.
[ Sun May 15 19:10:57 2005 ] Dumping LondConnectionObject:
TransactionReply ->
LoncapaHim -> oucapa2
State -> Connected
LondVersion -> Unknown
Timeoutable -> 1
CipherKey ->
Cipher ->
InformWritable -> 1
Socket -> IO::Socket::INET=GLOB(0x95b57c0)
TimeoutValue -> 30
AuthenticationMode -> ssl
TimeoutCallback ->
Port -> 5663
TransactionRequest -> init:ssl

TimeoutRemaining -> 30
LocalKeyFile ->
TransitionCallback ->
InformReadable -> 0
Host -> capa2.phy.ohiou.edu


--------------------------------------------------
In lonc.log I get something like this for capa2 connections around this 
time:

Sun May 15 18:58:20 2005 (20512) [oucapa2] [Sun May 15 18:58:20 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)]  Created 
connection 2 to host oucapa2
Sun May 15 18:58:20 2005 (20512) [oucapa2] [Sun May 15 18:58:20 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)]  Created 
connection 3 to host oucapa2
Sun May 15 18:58:20 2005 (20512) [oucapa2] [Sun May 15 18:58:20 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)] <font 
color='green'>SUCCESS: Connection 3 to oucapa2 now ready for action</font>
Sun May 15 18:58:20 2005 (20512) [oucapa2] [Sun May 15 18:58:20 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)] <font 
color='green'>SUCCESS: Connection 3 to oucapa2 now ready for action</font>
Sun May 15 19:00:40 2005 (20512) [oucapa2] [Sun May 15 19:00:40 2005: 
oucapa2 Connection count: 3 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: A socket timeout was detected</font>
Sun May 15 19:00:40 2005 (20512) [oucapa2] [Sun May 15 19:00:40 2005: 
oucapa2 Connection count: 3 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: Failing transaction 
encrypt:auth:ohiou:bm299302:bm2002
Sun May 15 19:00:40 2005 (20512) [oucapa2] [Sun May 15 19:00:40 2005: 
oucapa2 Connection count: 3 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>
Sun May 15 19:00:40 2005 (20512) [oucapa2] [Sun May 15 19:00:40 2005: 
oucapa2 Connection count: 2 Retries remaining: 1 (ssl)] <font 
color='blue'>WARNING: Lond connection lost.</font>
Sun May 15 19:00:40 2005 (20512) [oucapa2] [Sun May 15 19:00:40 2005: 
oucapa2 Connection count: 2 Retries remaining: 1 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>
Sun May 15 19:10:09 2005 (20512) [oucapa2] [Sun May 15 19:10:09 2005: 
oucapa2 Connection count: 2 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: A socket timeout was detected</font>
Sun May 15 19:10:09 2005 (20512) [oucapa2] [Sun May 15 19:10:09 2005: 
oucapa2 Connection count: 2 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: Failing transaction 
encrypt:auth:ohiou:ep207904:aalekej04
Sun May 15 19:10:09 2005 (20512) [oucapa2] [Sun May 15 19:10:09 2005: 
oucapa2 Connection count: 2 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>
Sun May 15 19:10:09 2005 (20512) [oucapa2] [Sun May 15 19:10:09 2005: 
oucapa2 Connection count: 1 Retries remaining: 1 (ssl)] <font 
color='blue'>WARNING: Lond connection lost.</font>
Sun May 15 19:10:09 2005 (20512) [oucapa2] [Sun May 15 19:10:09 2005: 
oucapa2 Connection count: 1 Retries remaining: 1 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>
Sun May 15 19:10:45 2005 (20512) [oucapa2] [Sun May 15 19:10:25 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)] <font 
color='red'>CRITICAL: Failed to make a connection with lond.</font>
Sun May 15 19:10:45 2005 (20512) [oucapa2] [Sun May 15 19:10:25 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>
Sun May 15 19:10:57 2005 (20512) [oucapa2] [Sun May 15 19:10:56 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)]  Created 
connection 2 to host oucapa2
Sun May 15 19:10:57 2005 (20512) [oucapa2] [Sun May 15 19:10:56 2005: 
oucapa2 Connection count: 1 Retries remaining: 2 (ssl)] <font 
color='green'>SUCCESS: Connection 2 to oucapa2 now ready for action</font>
Sun May 15 19:19:37 2005 (20512) [oucapa2] [Sun May 15 19:19:37 2005: 
oucapa2 Connection count: 2 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: A socket timeout was detected</font>
Sun May 15 19:19:37 2005 (20512) [oucapa2] [Sun May 15 19:19:37 2005: 
oucapa2 Connection count: 2 Retries remaining: 2 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>
Sun May 15 19:19:37 2005 (20512) [oucapa2] [Sun May 15 19:19:37 2005: 
oucapa2 Connection count: 1 Retries remaining: 1 (ssl)] <font 
color='blue'>WARNING: Lond connection lost.</font>
Sun May 15 19:19:37 2005 (20512) [oucapa2] [Sun May 15 19:19:37 2005: 
oucapa2 Connection count: 1 Retries remaining: 1 (ssl)] <font 
color='blue'>WARNING: Shutting down a socket</font>




----------------------------------------------------------------------------
Mark Lucas					email: lucasm at ohiou.edu
252D Clippinger Lab  				phone: (740)597-2984
Department of Physics and Astronomy             fax:   (740)593-0433
Ohio University
Athens, OH 45701