[LON-CAPA-admin] Finding large video files

Mon Sep 6 15:55:13 EDT 2021

James,

1. If the large files containing uncompressed video were uploaded directly to a course then as DC you can use:

Main Menu > Status of domain servers >  "Display quotas and usage for Course/Community Content"

to display the disk usage for courses in your domain. 	
If you click on the "Usage (MB)" header the results will be sorted by usage.

Once you have the titles of the course(s) which contain a lot of uploaded content you need to get their courseIDs.

On the command line use:
/home/httpd/perl/debug/dump_db.pl -u /home/httpd/lonUsers/sc/nohist_courseids.db > /tmp/sc_courses.txt

to dump courseIDs and titles for all courses in your domain to a text file (in /tmp/sc_courses.txt).

Next, that text file look for the titles of the courses you identified which have high disk usage.
The courseID will be sc_ followed by 17 characters (starting with a number 1 .. 9) then scl1
(e.g., sc_7a23822ee54b648fdscl1).

Any files uploaded directly to that particular course would be stored in:
/home/httpd/lonUsers/sc/7/a/2/7a23822ee54b648fdscl1/userfiles/docs

If you change to that directory:

cd /home/httpd/lonUsers/sc/7/a/2/7a23822ee54b648fdscl1/userfiles/docs

you can use the following command:

du -hsx -- * | sort -rh | head -10

to display which 10 sub-directories within that directory consume the most disk space.

You can then explore each subdirectory and decide what you would like to do with each file.
If you remove file(s) you should leave a placeholder file with the same name as the file you removed which will display a message indicating how to access the file you removed.

For the longer term the best strategy would likely be for instructors to upload video content to your institution's media space, and then include a link to each video file within the LON-CAPA course instead of uploading the file directly to the course.  If video content must to be in LON-CAPA then it would be best to upload the content to an Authoring Space, and then publish, and import the published item into a course (which will create a reference to the file).  That way you won't have the problem of potentially repeated duplication of files when a course is cloned.

Stuart Raeburn
LON-CAPA Academic Consortium
________________________________________
From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
Sent: Monday, September 6, 2021 3:20 PM
To: list about administration and system updating
Subject: {Disarmed} [LON-CAPA-admin] {Disarmed} Finding large video files

Hello,

In discovering that the /home drive is full, I am now searching for files to delete.

Over the pandemic, we had a faculty member who decided to upload uncompressed video to his courses.  One semester, one of the courses was cloned.  I have gone through some of the courses, looking for these files to remove.  What would be the best way to find large files (that belong only to his courses)?  I can then decide if they need to be removed.

Thanks,

James Clawson
Laboratory Manger
University of South Carolina
Department of Physics and Astronomy
clawsonj at mailbox.sc.edu<mailto:clawsonj at mailbox.sc.edu>

________________________________
From: Lucas, Mark <lucasm at ohio.edu>
Sent: Monday, September 6, 2021 2:48 PM
To: Clawson, James <CLAWSONJ at mailbox.sc.edu>; list about administration and system updating <lon-capa-admin at mail.lon-capa.org>
Subject: Re: {Disarmed} [LON-CAPA-admin] {Disarmed} Re: Lots of errors...

James,

Stuart can suggest longer term solutions, but basically find a few log files that might be cleaned out
and stop/start your web server and loncontrol on your library server. That should fix things.

/home/httpd/perl/logs sometimes has older log files. As well, if you have /var/logs on your home
drive, you can probably find some space there.

Is the library server on a VM where you can grovel more space? (we just had to do that)
/home/httpd/lonUsers will probably be the largest culprit, and there is currently no graceful
way to archive records.

Later,
Mark

> On Sep 6, 2021, at 2:39 PM, Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org> wrote:
>
> /home is full.
>
>
>
> James Clawson
> Laboratory Manger
> University of South Carolina
> Department of Physics and Astronomy
> clawsonj at mailbox.sc.edu
>
> From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Raeburn, Stuart via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
> Sent: Monday, September 6, 2021 2:08 PM
> To: lon-capa-admin at mail.lon-capa.org <lon-capa-admin at mail.lon-capa.org>
> Subject: Re: [LON-CAPA-admin] Lots of errors...
>
> James,
>
> >
> > We are getting a lot of error reports similar to what is attached.
> >
>
> You might check to see whether the disk partition for /home is full (i.e, 100% used) on your LON-CAPA library server.  What does the command:
>
> df -h
>
> report?
>
> Stuart Raeburn
> LON-CAPA Academic Consortium
> ________________________________________
> From: Raeburn, Stuart <raeburn at msu.edu>
> Sent: Sunday, September 5, 2021 10:32 PM
> To: lon-capa-admin at mail.lon-capa.org
> Subject: Re: Lots of errors...
>
> James,
>
> I am able to transfer my user session on another LON-CAPA node in the network to the sc access server, and the sc access server is able to retrieve information about my courses etc.
>
> However, I am not able to then transfer my user session from the sc access server to the sc library server.  I am able to display the LON-CAPA course catalog for the sc domain on both the sc access server and sc library server.
>
> Based on this I would conclude that the sc access server is able to connect to other nodes in the LON-CAPA network (including the sc library server), but it seems that the sc library server is not able to make a "LON-CAPA" connection back to my home server.
>
> I don't know whether your campus firewall rules limit outbound connections for your LON-CAPA library server to IP addresses outside the sc network, so I don't know whether what I see currently is normal for your library server.
>
> From the command line, these commands (run on each server -- library and access)
>
> ps -ef |grep lond |grep sc
> ps -ef |grep lonc |grep sc
>
> should show whether there are connections between lonc (on the library node) and lond on the access node), and vice versa.
>
> For the first one you should see something like
> lond: Listening to <sc hostID>
>
> For the second you should see something like:
> Connection count: 1 Retries remaining: 5
>
> In the error report from the errorhandler web form submitted by a student, the error is:
> ... write error at /home/httpd/lib/perl//Apache/lonuserstate.pm line 541
>
> which I expect would be called when a course session was being loaded (after role selection).
>
> There are log files in /home/httpd/perl/logs.
>
> >
> > Students normally log into the access server and have been reporting an issue
> > that homework submissions are not being taken or rejected.
> >
>
> Storage of a submission is considered operation "critical", so if a connection cannot be made from the access server to the library server, any "delayed" transactions will be queued, and completed when connectivity is restored.  You will find information about delayed transactions (destined for the library server) in /home/httpd/perl/logs/lonnet.perm.log on the access server.
>
> If you have additional information to share, while troubleshooting, feel free to contact me off-list.
>
> Stuart Raeburn
> LON-CAPA Academic Consortium
> ________________________________________
> From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
> Sent: Sunday, September 5, 2021 8:48 PM
> To: lon-capa-admin at mail.lon-capa.org
> Subject: [LON-CAPA-admin] Lots of errors...
>
> Hello,
>
> We are getting a lot of error reports similar to what is attached.
>
> On another (or possibly related) note:
>
> We are having login issues with our library server but not always the access one (I've been able to get in most of the time).  Students normally log into the access server and have been reporting an issue that homework submissions are not being taken or rejected.  The screen returns to the same state as before answering (no increase in submission attempts).   I have verified this behavior.   One thought I had was that it could be that the access server is not communicating with the library server (I don't have the knowledge yet to know how to check if this is true or not), and something is wrong on the library server.
>
> As far as I can tell, this is behavior that started a few days ago (Thursday).
>
> Any help on where to start diagnosis is appreciated.
>
> James Clawson
> Laboratory Manger
> University of South Carolina
> Department of Physics and Astronomy
> clawsonj at mailbox.sc.edu<mailto:clawsonj at mailbox.sc.edu>
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:https://protect2.fireeye.com/v1/url?k=482cf89a-17b7c19f-482cb65b-86564cfb0d85-05a5963fe217332c&q=1&e=520bac02-9ae8-47fa-8eed-452b8fdd9144&u=http%3A%2F%2Fmail.lon-capa.org%2Fmailman%2Flistinfo%2Flon-capa-admin
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website: MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website: https://protect2.fireeye.com/v1/url?k=76595e1f-29c266e2-765910de-86254b3f9600-4a385164d474ba2e&q=1&e=d0cc3489-580e-4a39-8da8-6809aac80656&u=http%3A%2F%2Fmail.lon-capa.org%2Fmailman%2Flistinfo%2Flon-capa-admin<https://urldefense.com/v3/__https://protect2.fireeye.com/v1/url?k=76595e1f-29c266e2-765910de-86254b3f9600-4a385164d474ba2e&q=1&e=d0cc3489-580e-4a39-8da8-6809aac80656&u=http*3A*2F*2Fmail.lon-capa.org*2Fmailman*2Flistinfo*2Flon-capa-admin__;JSUlJSUl!!HXCxUKc!l5Uxz_FwXOA-oTgKO-wxYMSJRAZ0-dP60M62-ekScrBbKzWc_eE1hcisK2DiQg$>

--
Mark Lucas
Professor of Instruction
Department of Physics and Astronomy (College of Arts and Sciences)/Ohio Honors Program (Honors College)
252D Clippinger Lab, Ohio University, Athens, OH 45701
email: lucasm at ohio.edu
phone: (740)597-2984
Pronouns: He, Him, His