<div dir="ltr"><div><div>Stuart,<br><br></div>A quick look at /var/log/messages on the servers with the 'unable to contact DNS' message shows that the network is failing to start before loncontrol runs. It does try to start up again later in the boot process and is successful. This seems doesn't seem to always happen this way. Sometimes the network does start successfully the first time. <br><br></div><div>So, I think attempt to see why I have the inconsistency in start up and if I can't determine what the cause is conclusively then I will do the conversion to systemd.<br><br></div><div>Also, I will contact you about my performance issues. <br></div><div><br></div><div>Thank you for your time and effort.<br></div><div><br>Bob Gonzales<br></div><div>Binghamton University<br></div><div>Chemistry<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 9, 2015 at 1:29 PM, Stuart Raeburn <span dir="ltr"><<a href="mailto:raeburn@msu.edu" target="_blank">raeburn@msu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Bob,<br>
<br>
One of the changes between CentOS 5/6 and 7 is the use of systemd (and systemctl commands) instead of SysV to control start-up of services on boot.<br>
<br>
That said, the previously used SysV services -- including loncontrol which starts the LON-CAPA daemons and updated iptables rules -- continue to work in CentOS 7.<br>
<br>
/sbin/chkconfig --list<br>
<br>
will show those.<br>
<br>
If you look in /var/log/messages after boot you should see the lines:<br>
<br>
loncontrol: Opening firewall access on port 5663<br>
loncontrol: Starting LON-CAPA<br>
<br>
indicating that LON-CAPA was started.<br>
<br>
The message logged in lonnet.log ("unable to contact DNS defaulting to on disk file") originates in lonnet::get_dns(), which is called when information is needed about cluster membership (and the cache has not been populated).<br>
<br>
If you run:<br>
<br>
/etc/init.d/loncontrol restart<br>
<br>
after your name service is available then that will cause the daemons to be restarted (and Apache reloaded), which will include retrieval of cluster membership information (and caching). Therefore it is equivalent to the operation of /etc/init.d/loncontrol start (on boot) with name service already available.<br>
<br>
On LON-CAPA instances I manage on CentOS 7 I do not see this issue on boot, when lonnet::get_dns() is called. You might look in /var/log/messages to see when start up of your name service occurs following boot.<br>
<br>
In CentOS 7 the idea behind using systemd is to make the start up process faster by starting services in parallel, as much as possible, and you can control this process by modifying files in /usr/lib/systemd/system.<br>
<br>
If you want to convert management of loncontrol from SysV to systemd you should add a file:<br>
<br>
loncontrol.service<br>
<br>
to /usr/lib/systemd/system with the following contents:<br>
<br>
[Unit]<br>
Description=Manage LON-CAPA daemons and update iptables rules for port 5663<br>
Wants=network-online.target nss-lookup.target<br>
After=network-online.target nss-lookup.target syslog.target basic.target<br>
<br>
[Service]<br>
RemainAfterExit=yes<br>
ExecStart= /etc/init.d/loncontrol start<br>
ExecReload=/etc/init.d/loncontrol reload<br>
ExecStop=/etc/init.d/loncontrol stop<br>
StandardOutput=syslog<br>
StandardError=syslog<br>
<br>
[Install]<br>
WantedBy=multi-user.target<br>
<br>
then use the command systemctl enable loncontrol.service<br>
<br>
(Note: if you make changes to files in /usr/lib/systemd/system you should then do: systemctl daemon-reload).<br>
<br>
Once loncontrol is converted for SysV to systemd ...<br>
<br>
To start loncontrol use:<br>
systemctl start loncontrol.service<br>
<br>
to stop loncontrol use:<br>
systemctl stop loncontrol.service<br>
<br>
to restart loncontrol use:<br>
systemctl restart loncontrol.service<br>
<br>
and to reload lonc and lond use:<br>
systemctl reload loncontrol.service<br>
<br>
Output from these commands will be available using:<br>
<br>
systemctl status loncontrol.service<br>
<br>
or by looking in /var/log/messages.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
... But, since I'm still having some<span class=""><br>
serious performance issues when high numbers of students log in for a quiz,<br>
I was curious if this is something worth looking into more deeply.<br>
</span></blockquote>
<br>
The relative timing of starting of loncontrol and your named service on reboot is not related to your perfomance issues, as long as you ran /etc/init.d/loncontrol restart after the named service had become available.<br>
<br>
If you are encountering performance issues on your servers at times of peak usage you might want to modify the lonLoadLim or lonUserLoadLim values on your servers.<br>
<br>
If students log-in via a load balancer, and you want sessions to be offloaded to LON-CAPA servers elsewhere in the network, at times when all the binghamton servers are overloaded you should ensure that:<br>
<br>
(a) Domain settings for "Dedicated Load Balancer(s)" include the binghamton access servers in the "Default destinations" in the "Offloads to: primary" category.<br>
<br>
and<br>
<br>
(b) Domain settings for "User session hosting/offloading" for the loadbalancer machine/VM include the MSU access servers -- msua1, msua2, msua3, msua4 in the "default" category.<br>
<br>
It would be useful to have information about server loads etc. at these times when large numbers of students log-in (and you see these performance issues). I can provide scripts to gather load data (and display in MRTG). Contact me off-list if you are interested.<br>
<br>
<br>
Stuart Raeburn<br>
LON-CAPA Academic Consortium<div><div class="h5"><br>
<br>
<br>
Quoting Bob Gonzales <<a href="mailto:rgonzal@binghamton.edu" target="_blank">rgonzal@binghamton.edu</a>>:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
After upgrading Centos to 7.1 and Lon-capa to 2.11 this summer I often get<br>
a message like this after a reboot:<br>
<br>
Sun Nov 8 09:19:49 2015 (1709): unable to contact DNS defaulting to on<br>
disk file dns_domain.tab<br>
<br>
and then I get a lot of messages like this:<br>
<br>
Sun Nov 8 09:19:49 2015 (1052): Name <a href="http://s4.lite.msu.edu" rel="noreferrer" target="_blank">s4.lite.msu.edu</a> no IP found<br>
<br>
If I restart Lon-capa via '/etc/init.d/loncontrol restart', the messages<br>
don't appear in lonnet.log but I don't know if the same initialization<br>
happens so that might, or might not, be OK. The lonnet.log doesn't show<br>
these 'no IP found' messages when loncron does it nightly run the next day<br>
but, again, I don't know if they same initialization happens then either.<br>
<br>
<br>
I've assumed that it is the result of the name service in Centos not having<br>
finished it's startup before Lon-capa started and that it resolves itself<br>
because I can log in right after the reboot and ping various non-Lon-capa,<br>
and local lon-capa machines by name. But, since I'm still having some<br>
serious performance issues when high numbers of students log in for a quiz,<br>
I was curious if this is something worth looking into more deeply.<br>
<br>
<br>
Thanks,<br>
Bob Gonzales<br>
Binghamton University<br>
Chemistry Dept<br>
<a href="mailto:rgonzal@binghamton.edu" target="_blank">rgonzal@binghamton.edu</a><br>
<br>
</blockquote>
<br>
<br>
<br></div></div>
Stuart Raeburn, Ph.D. LON-CAPA (cell) <a href="tel:%2B1%20517%20881-3308" value="+15178813308" target="_blank">+1 517 881-3308</a><br>
Dept. Physics & Astronomy Other (cell) <a href="tel:%2B1%20734%20277-2795" value="+17342772795" target="_blank">+1 734 277-2795</a><br>
Michigan State University <a href="mailto:raeburn@msu.edu" target="_blank">raeburn@msu.edu</a><br>
MI 48824 USA<br>
<br>
_______________________________________________<br>
LON-CAPA-admin mailing list<br>
<a href="mailto:LON-CAPA-admin@mail.lon-capa.org" target="_blank">LON-CAPA-admin@mail.lon-capa.org</a><br>
<a href="http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin" rel="noreferrer" target="_blank">http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin</a><br>
</blockquote></div><br></div>