[LON-CAPA-admin] Load balancing

Stuart Raeburn raeburn at msu.edu
Tue Oct 16 14:02:01 EDT 2012


Based on your current set up on capa2.ncs.mcgill.ca (which is running  
LON-CAPA 2.8.1)

> PerlSetVar     lonLoadLim      2.00
> PerlSetVar     lonUserLoadLim      0

there is no limit on users on the server, but once the server load (as  
retrieved from the first number from cat /proc/loadavg) exceeds 2.00,  
any new users attempting to log-in via capa2.ncs.mcgill.ca will have  
their encrypted log-in credentials directed to any of the "spare"  
LON-CAPA servers defined in:


which have loads less than their own "100%" load limits.

My assumption is that spares.tab contains a single entry:

which means that new log-ins will be transferred to capa.ncs.mcgill.ca  
when the loadavg on mcgilll2 exceeds 2.00

Note, however, that sessions for previously authenticated users are  
not transferred by LON-CAPA, so it is quite possible for a high load  
to exist on capa2.ncs.mcgill.ca, if there is some process running  
there that has elevated the server load for some reason.

Given that I only see two courses listed in the LON-CAPA course  
catalog for Fall 2012 for the mcgill domain --  

Intro Physics - Mechanics - Fall 2012 - PHYS 101-001
Mechanics and Waves - Fall 2012 - PHYS-131-001

I would be surprised if the high loads encountered on mcgilll2  
occurred because of the number of active user sessions was unusually  
large.  It seems more likely that there was a particular process  
running which was causing the excess load.

You can view LON-CAPA user activity on a particular server by using  
calls to scripts such as:


Note: starting with LON-CAPA 2.8 access to this script can be enabled  
for a specific IP address (e.g., for a monitoring machine), without  
the need for credentials to be supplied.  This is the practice at  
Michigan State University (MSU) where data are retrieved for display  
by MRTG for active users etc. every 5 minutes.

> What should we use for LonLoadLim and LonUserLoadLim?

LonUserLoadLim of 0 is desirable if you do not wish to limit the  
number of user sessions on a given server.  That is what we do at MSU,  
where the number of concurrent active LON-CAPA sessions usually peaks  
at around 100 on any of our access servers).

> Most of the connection I am seeing in the logs are being made by our  
>  F5 load balancer so I am thinking there is something not doing its   
> job properly.
> Does anyone have a similar setup?

I do not know of any LON-CAPA domains using an f5 load balancing  
appliance in front of their LON-CAPA servers.

For those domains where load balancing is in use for LON-CAPA  
(including MSU), a dedicated LON-CAPA machine (e.g.,  
https://loncapa.msu.edu at MSU) is used which authenticates users and  
then migrates authenticated user sessions to the least loaded of the  
access servers in the domain.  Starting with LON-CAPA 2.10 (released  
May 2010) domains have much more granular control over where their  
user's sessions may be hosted, and which other domains they may choose  
to host user sessions for.

I would strongly urge you to consider updating from your current  
LON-CAPA version -- 2.8.1, as subsequent versions provide additional  
functionality, as well as fixing a number of bugs.

Links to newer versions, and their release notes are available at:

In any event, your current LON-CAPA set-up at mcgill is not currently  
configured in a way that actually works fully with the LON-CAPA design  
of a network of servers supporting LON-CAPA inter-server traffic via  
port 5663.

Because capa.mcgill.ca was included in the official LON-CAPA cluster  
tables maintained by the LON-CAPA Academic Consoritum, starting in  
September 2007, your current set up falls somewhere between a  
standalone cluster consisting of just two servers (mcgilll1 and  
mcgilll2) and a server in the production cluster (mcgilll1  
communicating with servers from the other ~150 domains).

Evidently you have appropriate entries in  
/home/httpd/lonTabs/hosts.tab for your access and library servers,  
such that capa.ncs.mcgill.ca and capa2.ncs.mcgill.ca can communicate  
with each other.

However, other servers in the LON-CAPA network only know about the f5  
hardware balancer (externally resolved as capa.mcgill.ca), which means  
those servers are unable to connect to your library server (e.g., to  
facilitate sharing of content, or cross-institutional load-balancing).

Please contact me off-list if you wish to discuss the specific of your  
network set-up at McGill.  I would be interested to know what the  
specific needs were that led to the inclusion of the f5 load balancing  
appliance in your LON-CAPA set-up, to determine if there is a way  
those needs can continue to be addressed, while at the same  
maintaining connectivity between the mcgill LON-CAPA instance and the  
other domains in the broader LON-CAPA community.

Stuart Raeburn
LON-CAPA Academic Consortium
helpdesk at loncapa.org

Quoting Alain Plante <alain.plante at mcgill.ca>:

> Hi
> I am fairly new to loncapa and I am trying to understand if our   
> setup makes sense (this was done prior to me looking at how this   
> works) so please bear with me.
> We have capa.mcgill.ca which is the IP of a load balancer (f5) which  
>  redirects the connection to our access server capa2.ncs.mcgill.ca.
> We have a separate library server named capa.ncs.mcgill.ca which   
> should be used when load is too high on capa2. Last nigth the load   
> on capa2 was running between 30 and 75 while the load on capa kept   
> being under 1.0.
> Our current loncapa.conf is :
> PerlSetVar     lonHostID      mcgill2
> PerlSetVar     lonRole      access
> PerlSetVar     lonLoadLim      2.00
> PerlSetVar     lonExpire      86400
> PerlSetVar     lonReceipt      XXXXXXXXXXXXXXXXXXXXX
> PerlSetVar     lonUserLoadLim      0
> PerlSetVar     lonDefDomain      mcgill
> PerlSetVar     loncAllowInsecure      1
> PerlSetVar     lonSupportEMail      XXXXXXXXXXXXXXXXXXXXX
> PerlSetVar     londAllowInsecure      1
> PerlSetVar     lonAdmEMail      XXXXXXXXXXXXXXXXXXXX
> Most of the connection I am seeing in the logs are being made by our  
>  F5 load balancer so I am thinking there is something not doing its   
> job properly.
> Does anyone have a similar setup?
> What should we use for LonLoadLim and LonUserLoadLim?
> Thx
> Alain
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin

More information about the LON-CAPA-admin mailing list