[LON-CAPA-admin] Load balancing
Stuart Raeburn
raeburn at msu.edu
Tue Oct 16 14:02:01 EDT 2012
Hi,
Based on your current set up on capa2.ncs.mcgill.ca (which is running
LON-CAPA 2.8.1)
> PerlSetVar lonLoadLim 2.00
> PerlSetVar lonUserLoadLim 0
there is no limit on users on the server, but once the server load (as
retrieved from the first number from cat /proc/loadavg) exceeds 2.00,
any new users attempting to log-in via capa2.ncs.mcgill.ca will have
their encrypted log-in credentials directed to any of the "spare"
LON-CAPA servers defined in:
/home/httpd/lonTabs/spares.tab
which have loads less than their own "100%" load limits.
My assumption is that spares.tab contains a single entry:
mcgilll1
which means that new log-ins will be transferred to capa.ncs.mcgill.ca
when the loadavg on mcgilll2 exceeds 2.00
Note, however, that sessions for previously authenticated users are
not transferred by LON-CAPA, so it is quite possible for a high load
to exist on capa2.ncs.mcgill.ca, if there is some process running
there that has elevated the server load for some reason.
Given that I only see two courses listed in the LON-CAPA course
catalog for Fall 2012 for the mcgill domain --
http://capa.mcgill.ca/adm/coursecatalog
Intro Physics - Mechanics - Fall 2012 - PHYS 101-001
Mechanics and Waves - Fall 2012 - PHYS-131-001
I would be surprised if the high loads encountered on mcgilll2
occurred because of the number of active user sessions was unusually
large. It seems more likely that there was a particular process
running which was causing the excess load.
You can view LON-CAPA user activity on a particular server by using
calls to scripts such as:
http://capa.ncs.mcgill.ca/cgi-bin/userstatus.pl?simple
Note: starting with LON-CAPA 2.8 access to this script can be enabled
for a specific IP address (e.g., for a monitoring machine), without
the need for credentials to be supplied. This is the practice at
Michigan State University (MSU) where data are retrieved for display
by MRTG for active users etc. every 5 minutes.
> What should we use for LonLoadLim and LonUserLoadLim?
LonUserLoadLim of 0 is desirable if you do not wish to limit the
number of user sessions on a given server. That is what we do at MSU,
where the number of concurrent active LON-CAPA sessions usually peaks
at around 100 on any of our access servers).
> Most of the connection I am seeing in the logs are being made by our
> F5 load balancer so I am thinking there is something not doing its
> job properly.
>
> Does anyone have a similar setup?
I do not know of any LON-CAPA domains using an f5 load balancing
appliance in front of their LON-CAPA servers.
For those domains where load balancing is in use for LON-CAPA
(including MSU), a dedicated LON-CAPA machine (e.g.,
https://loncapa.msu.edu at MSU) is used which authenticates users and
then migrates authenticated user sessions to the least loaded of the
access servers in the domain. Starting with LON-CAPA 2.10 (released
May 2010) domains have much more granular control over where their
user's sessions may be hosted, and which other domains they may choose
to host user sessions for.
I would strongly urge you to consider updating from your current
LON-CAPA version -- 2.8.1, as subsequent versions provide additional
functionality, as well as fixing a number of bugs.
Links to newer versions, and their release notes are available at:
http://install.loncapa.org/
In any event, your current LON-CAPA set-up at mcgill is not currently
configured in a way that actually works fully with the LON-CAPA design
of a network of servers supporting LON-CAPA inter-server traffic via
port 5663.
Because capa.mcgill.ca was included in the official LON-CAPA cluster
tables maintained by the LON-CAPA Academic Consoritum, starting in
September 2007, your current set up falls somewhere between a
standalone cluster consisting of just two servers (mcgilll1 and
mcgilll2) and a server in the production cluster (mcgilll1
communicating with servers from the other ~150 domains).
Evidently you have appropriate entries in
/home/httpd/lonTabs/hosts.tab for your access and library servers,
such that capa.ncs.mcgill.ca and capa2.ncs.mcgill.ca can communicate
with each other.
However, other servers in the LON-CAPA network only know about the f5
hardware balancer (externally resolved as capa.mcgill.ca), which means
those servers are unable to connect to your library server (e.g., to
facilitate sharing of content, or cross-institutional load-balancing).
Please contact me off-list if you wish to discuss the specific of your
network set-up at McGill. I would be interested to know what the
specific needs were that led to the inclusion of the f5 load balancing
appliance in your LON-CAPA set-up, to determine if there is a way
those needs can continue to be addressed, while at the same
maintaining connectivity between the mcgill LON-CAPA instance and the
other domains in the broader LON-CAPA community.
Stuart Raeburn
LON-CAPA Academic Consortium
helpdesk at loncapa.org
Quoting Alain Plante <alain.plante at mcgill.ca>:
> Hi
>
> I am fairly new to loncapa and I am trying to understand if our
> setup makes sense (this was done prior to me looking at how this
> works) so please bear with me.
>
> We have capa.mcgill.ca which is the IP of a load balancer (f5) which
> redirects the connection to our access server capa2.ncs.mcgill.ca.
>
> We have a separate library server named capa.ncs.mcgill.ca which
> should be used when load is too high on capa2. Last nigth the load
> on capa2 was running between 30 and 75 while the load on capa kept
> being under 1.0.
>
> Our current loncapa.conf is :
>
> PerlSetVar lonHostID mcgill2
> PerlSetVar lonRole access
> PerlSetVar lonLoadLim 2.00
> PerlSetVar lonExpire 86400
> PerlSetVar lonReceipt XXXXXXXXXXXXXXXXXXXXX
> PerlSetVar lonUserLoadLim 0
> PerlSetVar lonDefDomain mcgill
> PerlSetVar loncAllowInsecure 1
> PerlSetVar lonSupportEMail XXXXXXXXXXXXXXXXXXXXX
> PerlSetVar londAllowInsecure 1
> PerlSetVar lonAdmEMail XXXXXXXXXXXXXXXXXXXX
>
>
> Most of the connection I am seeing in the logs are being made by our
> F5 load balancer so I am thinking there is something not doing its
> job properly.
>
> Does anyone have a similar setup?
>
> What should we use for LonLoadLim and LonUserLoadLim?
>
> Thx
>
> Alain
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin
More information about the LON-CAPA-admin
mailing list