[LON-CAPA-admin] {Disarmed} {Disarmed} Finding large video files
Lucas, Mark
lucasm at ohio.edu
Mon Sep 6 16:02:44 EDT 2021
James,
I’ll preface this with a warning: removing the file may break a course. Stuart will be the expert
as to what you may be able to get away with.
It might be smarter (though more work) to replace the large video files with a very short video
file of the same filename that basically says this video was removed.
That said, I tend to use linux find to find files, but always need to search the web to remind myself. Using
https://linuxize.com/post/find-large-files-in-linux/
as a reference,
I ran
find . -xdev -type f -size +200M -print |xargs ls -lh | sort -k5,5 -h -r
on /home/httpd/lonUsers/ohiou and found a list of all files over 200M, list these nicely and sorted by size.
I just ran this and it only took about 20-30 seconds on a library server that’s been running
about as long as USC (since about 2002).
You can add the -name “*.mov” to find files with a particular file extension of .mov, for example.
“Users” are organized under lonUsers by domain/first character of user/second character/third character/username/…
For example: /home/httpd/lonUsers/ohiou/l/u/c/lucasm/….
Courses are actually users with a long, cryptic ID.
Clicking under the “more” link next to a course name at the top of the screen when in a course as course coordinator,
you will get the course ID. For example:
ohiou_48883902a5f2385e45oucapa2
All of the course-related files would be found under:
/home/httpd/lonUsers/ohiou/4/8/8/48883902a5f2385e45oucapa2
You can find courses owned by a particular user by using the course selection under domain coordinator.
It looks like you could also parse environment.hist, which has an ascii listing of changes in course
data.
I was able to search one of these files and find the string “&internal%2ecourseowner=lucasm%3aohiou&” buried in
one of the lines for a course I owned.
Again, it can be dangerous going in and removing files for a course in the filesystem. You might wait to act
on anything until Stuart can provide more information about what might be safe.
Let me know if this helps at all.
Mark
> On Sep 6, 2021, at 3:20 PM, Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org> wrote:
>
> Hello,
>
> In discovering that the /home drive is full, I am now searching for files to delete.
>
> Over the pandemic, we had a faculty member who decided to upload uncompressed video to his courses. One semester, one of the courses was cloned. I have gone through some of the courses, looking for these files to remove. What would be the best way to find large files (that belong only to his courses)? I can then decide if they need to be removed.
>
> Thanks,
>
> James Clawson
> Laboratory Manger
> University of South Carolina
> Department of Physics and Astronomy
> clawsonj at mailbox.sc.edu
>
> From: Lucas, Mark <lucasm at ohio.edu>
> Sent: Monday, September 6, 2021 2:48 PM
> To: Clawson, James <CLAWSONJ at mailbox.sc.edu>; list about administration and system updating <lon-capa-admin at mail.lon-capa.org>
> Subject: Re: {Disarmed} [LON-CAPA-admin] {Disarmed} Re: Lots of errors...
>
> James,
>
> Stuart can suggest longer term solutions, but basically find a few log files that might be cleaned out
> and stop/start your web server and loncontrol on your library server. That should fix things.
>
> /home/httpd/perl/logs sometimes has older log files. As well, if you have /var/logs on your home
> drive, you can probably find some space there.
>
> Is the library server on a VM where you can grovel more space? (we just had to do that)
> /home/httpd/lonUsers will probably be the largest culprit, and there is currently no graceful
> way to archive records.
>
> Later,
> Mark
>
>
> > On Sep 6, 2021, at 2:39 PM, Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org> wrote:
> >
> > /home is full.
> >
> >
> >
> > James Clawson
> > Laboratory Manger
> > University of South Carolina
> > Department of Physics and Astronomy
> > clawsonj at mailbox.sc.edu
> >
> > From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Raeburn, Stuart via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
> > Sent: Monday, September 6, 2021 2:08 PM
> > To: lon-capa-admin at mail.lon-capa.org <lon-capa-admin at mail.lon-capa.org>
> > Subject: Re: [LON-CAPA-admin] Lots of errors...
> >
> > James,
> >
> > >
> > > We are getting a lot of error reports similar to what is attached.
> > >
> >
> > You might check to see whether the disk partition for /home is full (i.e, 100% used) on your LON-CAPA library server. What does the command:
> >
> > df -h
> >
> > report?
> >
> > Stuart Raeburn
> > LON-CAPA Academic Consortium
> > ________________________________________
> > From: Raeburn, Stuart <raeburn at msu.edu>
> > Sent: Sunday, September 5, 2021 10:32 PM
> > To: lon-capa-admin at mail.lon-capa.org
> > Subject: Re: Lots of errors...
> >
> > James,
> >
> > I am able to transfer my user session on another LON-CAPA node in the network to the sc access server, and the sc access server is able to retrieve information about my courses etc.
> >
> > However, I am not able to then transfer my user session from the sc access server to the sc library server. I am able to display the LON-CAPA course catalog for the sc domain on both the sc access server and sc library server.
> >
> > Based on this I would conclude that the sc access server is able to connect to other nodes in the LON-CAPA network (including the sc library server), but it seems that the sc library server is not able to make a "LON-CAPA" connection back to my home server.
> >
> > I don't know whether your campus firewall rules limit outbound connections for your LON-CAPA library server to IP addresses outside the sc network, so I don't know whether what I see currently is normal for your library server.
> >
> > From the command line, these commands (run on each server -- library and access)
> >
> > ps -ef |grep lond |grep sc
> > ps -ef |grep lonc |grep sc
> >
> > should show whether there are connections between lonc (on the library node) and lond on the access node), and vice versa.
> >
> > For the first one you should see something like
> > lond: Listening to <sc hostID>
> >
> > For the second you should see something like:
> > Connection count: 1 Retries remaining: 5
> >
> > In the error report from the errorhandler web form submitted by a student, the error is:
> > ... write error at /home/httpd/lib/perl//Apache/lonuserstate.pm line 541
> >
> > which I expect would be called when a course session was being loaded (after role selection).
> >
> > There are log files in /home/httpd/perl/logs.
> >
> > >
> > > Students normally log into the access server and have been reporting an issue
> > > that homework submissions are not being taken or rejected.
> > >
> >
> > Storage of a submission is considered operation "critical", so if a connection cannot be made from the access server to the library server, any "delayed" transactions will be queued, and completed when connectivity is restored. You will find information about delayed transactions (destined for the library server) in /home/httpd/perl/logs/lonnet.perm.log on the access server.
> >
> > If you have additional information to share, while troubleshooting, feel free to contact me off-list.
> >
> > Stuart Raeburn
> > LON-CAPA Academic Consortium
> > ________________________________________
> > From: LON-CAPA-admin <lon-capa-admin-bounces at mail.lon-capa.org> on behalf of Clawson, James via LON-CAPA-admin <lon-capa-admin at mail.lon-capa.org>
> > Sent: Sunday, September 5, 2021 8:48 PM
> > To: lon-capa-admin at mail.lon-capa.org
> > Subject: [LON-CAPA-admin] Lots of errors...
> >
> > Hello,
> >
> > We are getting a lot of error reports similar to what is attached.
> >
> > On another (or possibly related) note:
> >
> > We are having login issues with our library server but not always the access one (I've been able to get in most of the time). Students normally log into the access server and have been reporting an issue that homework submissions are not being taken or rejected. The screen returns to the same state as before answering (no increase in submission attempts). I have verified this behavior. One thought I had was that it could be that the access server is not communicating with the library server (I don't have the knowledge yet to know how to check if this is true or not), and something is wrong on the library server.
> >
> > As far as I can tell, this is behavior that started a few days ago (Thursday).
> >
> > Any help on where to start diagnosis is appreciated.
> >
> > James Clawson
> > Laboratory Manger
> > University of South Carolina
> > Department of Physics and Astronomy
> > clawsonj at mailbox.sc.edu<mailto:clawsonj at mailbox.sc.edu>
> > _______________________________________________
> > LON-CAPA-admin mailing list
> > LON-CAPA-admin at mail.lon-capa.org
> > MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:https://protect2.fireeye.com/v1/url?k=482cf89a-17b7c19f-482cb65b-86564cfb0d85-05a5963fe217332c&q=1&e=520bac02-9ae8-47fa-8eed-452b8fdd9144&u=http%3A%2F%2Fmail.lon-capa.org%2Fmailman%2Flistinfo%2Flon-capa-admin
> > _______________________________________________
> > LON-CAPA-admin mailing list
> > LON-CAPA-admin at mail.lon-capa.org
> > MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:MailScanner has detected definite fraud in the website at "protect2.fireeye.com". Do not trust this website:https://protect2.fireeye.com/v1/url?k=76595e1f-29c266e2-765910de-86254b3f9600-4a385164d474ba2e&q=1&e=d0cc3489-580e-4a39-8da8-6809aac80656&u=http%3A%2F%2Fmail.lon-capa.org%2Fmailman%2Flistinfo%2Flon-capa-admin
>
> --
> Mark Lucas
> Professor of Instruction
> Department of Physics and Astronomy (College of Arts and Sciences)/Ohio Honors Program (Honors College)
> 252D Clippinger Lab, Ohio University, Athens, OH 45701
> email: lucasm at ohio.edu
> phone: (740)597-2984
> Pronouns: He, Him, His
>
>
>
>
>
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin
--
Mark Lucas
Professor of Instruction
Department of Physics and Astronomy (College of Arts and Sciences)/Ohio Honors Program (Honors College)
252D Clippinger Lab, Ohio University, Athens, OH 45701
email: lucasm at ohio.edu
phone: (740)597-2984
Pronouns: He, Him, His
More information about the LON-CAPA-admin
mailing list