<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Problem overview:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>For approximately the last year we have had an intermittent issue where
the /home/httpd/perl/logs/lonsql_errors file grows to occupy all available disk
space - in a matter of a few minutes/hours (i.e. NOT over several days).
This error has occurred on two completely different systems (different physical
hardware and both FC6 and FC7). Recently we have seen the problem occur
when running parse_activity although we have been able to run it successfully
in the past (it is unknown if the parse_activity was to blame for any but the
two most recent occurrences of the issue). The lonsql_errors file
contains HUGE sql statements (running 'tail' on the file to only output the
last 10 lines produces 21 Mb of output (that's right, 10 lines of the file
contain 21 MEGAbytes of data).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Here's an example of the last episode:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'> [root@lon-capa ~]# ps aux | grep parse<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>www 19358 96.5
3.8 87348 80056 ?
R 15:14 33:31 parse_activity_log.pl:
9r136551aae81475cBTl1@BT loading existing data<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'> [root@lon-capa ~]# df -h<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>Filesystem
Size Used Avail Use% Mounted on<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/dev/sda2
19G 2.3G 16G 13% /<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/dev/sda6
9.5G 1.3G 7.8G 15% /var<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/dev/sda5
36G 34G 0 100% /home<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/dev/sda1
99M 17M 77M 18% /boot<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>tmpfs
1013M
0 1013M 0% /dev/shm<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'> [root@lon-capa ~]# cd /home/httpd/perl/logs/<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'> [root@lon-capa logs]# ls -lh | grep lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>-rw-r--r-- 1 www www 32G 2008-03-25 15:50
lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'> [root@lon-capa logs]# lsof | grep lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>lonsql
12846 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>lonsql 12847
www 2w
REG 8,5 34288336896
5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>lonsql
12849 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>lonsql
12851 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>lonsql
12852 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>parse_act 19358
www 2w
REG 8,5 34288336896
5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>sh
19361 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>cat
19362 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>gzip
19363 www
2w REG
8,5 34288336896 5066139 /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>At this point we did the following to rectify the situation:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/etc/init.d/loncontrol stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/etc/init.d/httpd stop<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>kill 19358 (it wasn't killed by stopping the above)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>rm /home/httpd/perl/logs/lonsql_errors<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/etc/init.d/loncontrol start<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'>/etc/init.d/httpd start<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 face=Courier><span style='font-size:8.0pt;
font-family:Courier'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>This machine is a P4 2.4Ghz with 2Gb of ram, 2Gb of swap, and the disk
space noted above (/home normally uses 1.8G of the 36G total so less than
10%). OS is Fedora - FC7 with all updates as of 3/21/08.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>A gzip compressed copy of the last 10 lines of lonsql_errors (1.5mb
compressed, 21mb uncompressed) is available at
http://lon-capa.btcatholic.org/bt-lonsql_errors.gz<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Any help would be greatly appreciated. Thanks!</span></font><font
size=1 face=Courier><span style='font-size:8.0pt;font-family:Courier'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>