[LON-CAPA-admin] system overload

[Bob Gonzales]

We had a pretest for our large chemistry class tonight and the system
response was terrible.  The load-balancer took minutes to authorize
you and then send you to an access server.  It eventually seemed to
stop working so students logged into access servers directly.  On the
access servers pages took minutes to load and update.

It was pretty bad experience for the freshmen students.

Here's a quick description of my domain with some load average info I
collected via top (I don't have the SA tools installed yet since I
upgraded everything over the summer):

- library server (quad core xenon with 8 GB mem and SATA HD) - load average was
around 8.0 but was only using  about 6 MB of swap
- 2 access servers (quad core xenons with 32 GB mem) - load average was
around 2.0, not using swap space
- 3 access servers (quad core xenons with 8 GB mem)  - load average was
around 1.5, not using swap space
- 1 load balancer (quad core xenon with 8 GB mem) - load average was 0.1-0.5.

We've run quizzes before without any trouble but here are some
differences in the domain from last year:

- upgraded all servers to loncapa 2.11.1
- upgraded all servers to centos 7
- library server is running software raid 1, it was not running this last year
- 950 students this year versus 850 last year
- test was comprised of simple problems, not published problems

Looking at differences from last year, 2 things stand out: the
software raid 1 and the simple problems instead of published problems?
 Are simple problems cached on the access servers?

Any help figuring this out would be very much appreciated.

Thanks,
Bob Gonzales
Chemistry Dept
Binghamton University