Subject: [LON-CAPA-dev] lonc/lond

From: "Ron Fox" <[email-masked]>
Date: Thu, 7 Mar 2002 23:23:34 -0500

Hi all,
  I noticed there was not an open bugzilla entry for the lonc/lond issues I've been stalking so I created one and handed it to myself....

  I've committed a new version of lonc which fixes one of the many sources of instability in lonc, and simplifies some of the logic:

Lonc is a network daemon with a parent process (called PARENT below) which manages a pool of child processes each connected to a specific lond representing a persistent connection to a remote lon-capa system (called either SPECIFIC CHILD or CHILDREN below).
    In order to improve reliability, PARENT monitors the CHILDREN and attempts a limited number of restarts should a SPECIFIC CHILD die...It did this using a main loop which had (simplified) pseudo code like this:

while forever
   sleep until signal wakes me up
   for all entries in peer to pid hash with no entry in the pid to peer hash
      make_new_child for associated connection
   end for
endwhile

make_new_child maintained two hashes.  A pid to peer hash and a peer to pid hash.

a handler for the SIGCHILD signal (sent to a process when a child process exits) was established.  This handler executed pseudo code like:

deadpid = wait    # wait returns pid of dead child, and frees it from zombie state.
delete deadpid from pid to peer hash

1. This logic is overly complex for my taste and I've
   a. removed the signal handler altogether.
   b. replaced the parent main loop with logic shown in the pseudo code below:
   
while forever
   deadpid = wait     # Note other signals can interrupt this wait hence:
   if there's an entry for deadpid in the pid to peer hash
       make_new_child for the peer associated with deadpid
   endif
endwhile

2. Discovered and fixed a defect in make_new_child which essentially threw away all entries in the peer to pid hash (wrong variable used as the hash key at insertion time).

The defect  in make_new_child was causing the following problem:
  If a connection was lost or could not be formed, and the associated child exited,
lonc would not only attempt to restart the child (ok) but believed that >all< children had died, and start new children for all connections.  The duplicate children would soon discover that they could not create a server socket (the existing process already owned it), and exit starting the dance all over again, until the retry count was exhausted and lonc would give up on recreating all connections.  Subsequent to that, any child exit was total and unrecoverable (there's no current logic for resetting the retry count back to zero).

There are more problems which have probably not been seen or are seen at a lower probability than this.
  I was lucky enough to be debugging this on a system who's hosts.tab was 'too big' and therefore immediately and reproducably triggered this collapse.

Stay tuned for more news about lonc/lond as it develops.

Cheers,
RF.

---

From: Gerd Kortemeyer <[email-masked]>
Date: Fri, 08 Mar 2002 09:29:21 -0500

Ron,

Thanks!!!

There is actually only one question: why did this ever work at all?

- Gerd.

Ron Fox wrote:

> Hi all,  I noticed there was not an open bugzilla entry for the lonc/lond
> issues I've been stalking so I created one and handed it to myself....
> I've committed a new version of lonc which fixes one of the many sources of
> instability in lonc, and simplifies some of the logic: Lonc is a network
> daemon with a parent process (called PARENT below) which manages a pool of
> child processes each connected to a specific lond representing a persistent
> connection to a remote lon-capa system (called either SPECIFIC CHILD or
> CHILDREN below).    In order to improve reliability, PARENT monitors the
> CHILDREN and attempts a limited number of restarts should a SPECIFIC CHILD
> die...It did this using a main loop which had (simplified) pseudo code like
> this: while forever   sleep until signal wakes me up   for all entries in
> peer to pid hash with no entry in the pid to peer hash      make_new_child
> for associated connection   end forendwhile make_new_child maintained two
> hashes.  A pid to peer hash and a peer to pid hash. a handler for the
> SIGCHILD signal (sent to a process when a child process exits) was
> established.  This handler executed pseudo code like: deadpid = wait    #
> wait returns pid of dead child, and frees it from zombie state.delete
> deadpid from pid to peer hash 1. This logic is overly complex for my taste
> and I've   a. removed the signal handler altogether.   b. replaced the
> parent main loop with logic shown in the pseudo code below: while forever
> deadpid = wait     # Note other signals can interrupt this wait hence:   if
> there's an entry for deadpid in the pid to peer hash       make_new_child
> for the peer associated with deadpid   endifendwhile 2. Discovered and fixed
> a defect in make_new_child which essentially threw away all entries in the
> peer to pid hash (wrong variable used as the hash key at insertion
> time). The defect in make_new_child was causing the following problem:  If a
> connection was lost or could not be formed, and the associated child
> exited,lonc would not only attempt to restart the child (ok) but believed
> that >all< children had died, and start new children for all connections.
> The duplicate children would soon discover that they could not create a
> server socket (the existing process already owned it), and exit starting the
> dance all over again, until the retry count was exhausted and lonc would
> give up on recreating all connections.  Subsequent to that, any child exit
> was total and unrecoverable (there's no current logic for resetting the
> retry count back to zero). There are more problems which have probably not
> been seen or are seen at a lower probability than this.  I was lucky enough
> to be debugging this on a system who's hosts.tab was 'too big' and therefore
> immediately and reproducably triggered this collapse. Stay tuned for more
> news about lonc/lond as it develops. Cheers,RF.