[LON-CAPA-admin] access servers lost connection?

Wed Jun 7 17:33:00 EDT 2006

On Wednesday 07 June 2006 11:50 am, Guy Albertelli II wrote:
> Hi Todd,
>
> > I need some help on this one.  Any ideas?
>
> Hmm, both can talk to msu throguh ssl:
> www      26428  0.4  0.2  22344 17120 ?        S    13:47   0:00 lond:
> Listening to csma1 (ssl) Wed Jun  7 13:48:36 2006
> www      26482  1.3  0.2  20968 15656 ?        S    13:48   0:00 lond:
> Listening to csml1 (ssl) Wed Jun  7 13:48:44 2006
>
>
> So something is right.

I agree that something is right.  I see similar entries, myself, from both 
servers.  They  just don't seem to like each other.
>
> 1) Can you take a look at lonc.log and lonc_errors on the access server,
> and lond.log and lond_error on the library server?

OK. Here you go.  The certs are supposedly installed on csml1, and look 
reasonable, in terms of talking to other library servers (msu and ohiou)

I did a stop/start of both loncontrol and httpd on the access server (csma1).

When I tried to log in, via the access server, this is what happened in the 
logs.

lonc.log on access server:
Wed Jun  7 13:47:23 2006 (21565) [csma1] [Wed Jun  7 13:45:48 2006: Parent 
keeping the flock] <font color='red'>CRITICAL: Forking server for 
csma1</font>
Wed Jun  7 13:47:23 2006 (21618) [csma1] [Wed Jun  7 13:47:23 2006: Connected 
to csma1]  Created connection 1 to host csma1
Wed Jun  7 13:47:23 2006 (21618) [csma1] [Wed Jun  7 13:47:23 2006: Connected 
to csma1] <font color='yellow'>INFO: Connected to lond version: version:
$Revision: 1.305.2.5 $</font>
Wed Jun  7 13:47:23 2006 (21618) [csma1] [Wed Jun  7 13:47:23 2006: Connected 
to csma1] <font color='green'>SUCCESS: Connection 1 to csma1 now ready for 
action</font>
Wed Jun  7 13:47:23 2006 (21565) [csml1] [Wed Jun  7 13:45:48 2006: Parent 
keeping the flock] <font color='red'>CRITICAL: Forking server for 
csml1</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1]  Created connection 2 to host csml1
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1]  Created connection 2 to host csml1
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Lond connection lost.</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Shutting down a socket</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Lond connection lost.</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Shutting down a socket</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Failing transaction home</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Failing transaction log</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Failing transaction log</font>
Wed Jun  7 13:47:23 2006 (21620) [csml1] [Wed Jun  7 13:47:23 2006: Connected 
to csml1] <font color='blue'>WARNING: Failing transaction log</font>

library server lond.log:
Wed Jun  7 13:47:23 2006 (27926): <font color="green"> Attempting to start 
child (IO::Socket::INET=GLOB(0x871f034))</font>
Wed Jun  7 13:47:23 2006 (27926): <font color="green"> Attempting to start 
child (IO::Socket::INET=GLOB(0x8721494))</font>
Wed Jun  7 13:47:23 2006 (29192): <font color="green"> existing host 
msul1</font>

Wed Jun  7 13:47:23 2006 (29192): <font color="yellow">INFO: Connection, 
138.67.38.59 (csma1) connection type = client </font>
Wed Jun  7 13:47:23 2006 (29192): Setting hostid to csml1, and domain to csm
Wed Jun  7 13:47:23 2006 (29193): <font color="green"> existing host 
msul1</font>

Wed Jun  7 13:47:23 2006 (29193): <font color="yellow">INFO: Connection, 
138.67.38.59 (csma1) connection type = client </font>
Wed Jun  7 13:47:23 2006 (29193): Setting hostid to csml1, and domain to csm
Wed Jun  7 13:47:23 2006 (29192): <font color="red"> CRITICAL SSL Socket 
promotion failed: SSL accept attempt failederror:1408A0C1:SSL 
routines:SSL3_GET_CLIENT_HELLO:no shared cipher </font>
Wed Jun  7 13:47:23 2006 (29192): <font color='blue'>WARNING: Rejected client 
138.67.38.59, closing connection</font>
Wed Jun  7 13:47:23 2006 (29192): <font color='red'>CRITICAL: Disconnect from 
138.67.38.59 (csma1)</font>
Wed Jun  7 13:47:23 2006 (29193): <font color="red"> CRITICAL SSL Socket 
promotion failed: SSL accept attempt failederror:1408A0C1:SSL 
routines:SSL3_GET_CLIENT_HELLO:no shared cipher </font>
Wed Jun  7 13:47:23 2006 (29193): <font color='blue'>WARNING: Rejected client 
138.67.38.59, closing connection</font>
Wed Jun  7 13:47:23 2006 (29193): <font color='red'>CRITICAL: Disconnect from 
138.67.38.59 (csma1)</font>
Wed Jun  7 13:47:23 2006 (27926): Child 29192 died
Wed Jun  7 13:47:23 2006 (27926): Child 29193 died

Clearly there is a problem with my access servers agreeing with the library 
server regarding the appropriate keys to use.  I didn't report this before, 
because it wasn't there, I swear, but anyway, it is now.

> > Access servers no longer authenticate.  I don't see any indications on
> > the library server.  Below is the output from logs, ps, etc. from one of
> > the access servers, which make it clear why authentication isn't working.
> >  The question is why did the connections die?  I have similar output from
> > both access servers.
>
> Possibly a loncontrol restart on the access servers for fun could take
> care of it?

No such luck.  Here's an interesting set of things I tried, in fairly rapid 
succession.  Unfortunately they don't coincide with the entries above, as I 
tried the sequence below before your message appeared:

Removed certs from csml1
stop/start of loncontrol and httpd on csml1
can log in from msua1
cannot log in from csma1, csma2
stop/start loncontrol and httpd on csma1 and csma2
can now log in from both csma1 and csma2
re-ran the ssl install script on csml1
did not stop/start loncontrol and httpd
could log in from csma1--still had an insecure connection
could not log in from csma2--had to initiate connection
can log in from msua1
stop/start loncontrol and httpd on csma1 and csma2
still cannot log in from csma1 and csma2
stop/start loncontrol and httpd on csml1
still cannot log in from csma1 and csma2
stop/start loncontrol and httpd on csma1 and csma2
still cannot log in from csma1 and csma2
remove the installed cert files
can successfully log in from both csma1 and csma2, without stop/start

This all happened within about 15 minutes, so I suppose caching could have 
been an issue, but I also figured that all the stop/starting would eliminate 
that problem.

Todd

-- 
Dr. Todd Ruskell
Senior Lecturer, Department of Physics       Office:  Meyer Hall 326
Colorado School of Mines                     Phone: 303-384-2080
1523 Illinois Street                         Fax: 303-273-3919
Golden, CO 80401