[LON-CAPA-admin] Published resources not showing up on access servers
Stuart Raeburn
raeburn at msu.edu
Sun Sep 20 11:22:54 EDT 2015
Bob,
> Map not loaded: The file
> /home/httpd/html/userfiles/binghamton/71215825fc2344762binghamtonl1/default.sequence
> does not exist.
This means replication of the default.sequence file (i.e., the top
level of the course) failed, so your course coordinator role in
binghamton_71215825fc2344762binghamtonl1 could not be initialized.
> and I get sent back to the roles screen. I look on binghamtonl1 and this
> file does not exist.
Correct. On a library server that file actually resides in
/home/httpd/lonUsers/binghamton/7/1/2/71215825fc2344762binghamtonl1/userfiles
but when it is copied to an access server it is replicated to
/home/httpd/html/userfiles/binghamton/71215825fc2344762binghamtonl1/
You might check the file ownership/permissions for
/home/httpd/html/userfiles/binghamton/71215825fc2344762binghamtonl1/
on the access server, and confirm that the file:
/home/httpd/lonUsers/binghamton/7/1/2/71215825fc2344762binghamtonl1/userfiles/default.sequence exists on the library
server.
If that is the case, then look in /home/httpd/perl/logs/lonnet.log on
the access server for "Userfile repcopy failed for
uploaded/binghamton/71215825fc2344762binghamtonl1/default.sequence"
and in the Apache logs on the library server for failed requests for
/raw/uploaded/binghamton/71215825fc2344762binghamtonl1/default.sequence from
128.226.130.9
The most likely reason for failure to replicate an "uploaded" file,
such as default.sequence is for this line:
my $reqhost = $r->get_remote_host(REMOTE_NOLOOKUP);
to set $reqhost to an IP address which is not present as a key in the
hash: %iphost in lonnet.pm
You might look in /home/httpd/perl/logs/lonnet.log on the library
server for items such as: "Name loncapa2.chem.binghamton.edu no IP
found".
You could copy the following script to a file (call it check
memcached_dump.pl) and then use: perl memcached_dump.pl to see what
is cached in terms of iphost information in Memcached on one of your
LON-CAPA servers:
******
#!/usr/bin/perl
use Cache::Memcached;
use Data::Dumper;
use strict;
my $instance = "127.0.0.1:11211";
my $memd = new Cache::Memcached {
'servers' => [ $instance],
'debug' => 0,
};
my %containers;
my $stats = $memd->stats('items');
my $items = $stats->{hosts}->{$instance}->{items};
foreach my $line (split(/\r\n/,$items)) {
$line =~ s/^.*:(.*):.*$/$1/ig;
$containers{$line} = 1;
}
foreach my $container (sort(keys(%containers))) {
my $result = $memd->stats("cachedump $container 0");
my $contents = $result->{hosts}->{$instance}->{"cachedump
$container 0"};
foreach my $item (split(/\r\n/,$contents)) {
my ($name,$size) = ($item =~ /^ITEM\s+(\S+)\s+\[([^;]+)/);
next unless ($name eq 'iphost%3aiphost');
$item =~ s/^ITEM (.*) \[.*$/$1/ig;
my $val = $memd->get($item);
print "$name $size ".Dumper($val)."\n";
}
}
$memd->disconnect_all;
*****
> Looking in lonnet on the library server I see this during and after the
> reboot:
>
> Sat Sep 19 08:49:07 2015 (2980): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
> Sat Sep 19 08:49:08 2015 (2980): caching of id ->
> dns%3a%2fadm%2fdns%2fdomain failed
>
Currently you will always see those after a reboot or after restarting
loncontrol.
This message can be ignored; it occurs because
Apache::lonnet::load_hosts_tab() and Apache::lonnet::load_domain_tab()
are called in /home/httpd/perl/loncron *before* starting memcached, so
the attempt to cache the results fails. It is not an issue because
they will be cached a few seconds later, as a result of other calls.
The "no IP found" messages for hosts such as loncapa.nfcc.edu can be
ignored. That message is displayed because that server is no longer
active and that hostname no longer resolves to an IP address. (The
same goes for one or two others).
> Sat Sep 19 08:43:04 2015 (1362): Name loncapa.calpoly.edu no IP found
>
> this goes on for all the servers.
That indicates that the domain name service used by your server was
not working, at the time loncontrol was started after reboot.
However, you found that you were able to resolve that issue by using
/etc/init.d/loncontrol restart later (presumably once DNS service was
contactable), at which point the "no IP found" messages were only
logged for inactive hosts which are not resolvable.
Stuart Raeburn
LON-CAPA Academic Consortium
Quoting Bob Gonzales <rgonzal at binghamton.edu>:
> Things are getting worse for me here. I rebooted my library server this
> morning hoping that might cure the ?not found? issue. But it didn?t and I
> ended up with more weirdness.
>
>
> Now when I try to log into my test course from any of my access servers I
> get this message:
>
>
> Map not loaded: The file
> /home/httpd/html/userfiles/binghamton/71215825fc2344762binghamtonl1/default.sequence
> does not exist.
>
>
> and I get sent back to the roles screen. I look on binghamtonl1 and this
> file does not exist.
>
>
> Looking in lonnet on the library server I see this during and after the
> reboot:
>
>
> Sat Sep 19 08:40:43 2015 (15857): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
>
> Sat Sep 19 08:40:44 2015 (15857): Name loncapa.brs.cgresd.net no IP found
>
> Sat Sep 19 08:40:50 2015 (15857): Name tuvok.physast.uga.edu no IP found
>
> Sat Sep 19 08:40:50 2015 (15857): Name BCHS-CAPA.jeffco.k12.co.us no IP
> found
>
> Sat Sep 19 08:40:51 2015 (15857): Name loncapa.nfcc.edu no IP found
>
> Sat Sep 19 08:40:52 2015 (15857): caching of id -> iphost%3aiphost failed
>
> Sat Sep 19 08:40:56 2015 (8569): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:40:56 2015 (8572): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:40:56 2015 (2602): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:40:56 2015 (3204): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:40:56 2015 (8570): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:40:56 2015 (8551): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:40:56 2015 (3202): Failed to flush log buffer for
> binghamton_71215825fc2344762binghamtonl1
>
> Sat Sep 19 08:41:06 2015 (8569): Shutting down
>
> Sat Sep 19 08:41:06 2015 (3202): Failed to flush log buffer for
> binghamton_4231791afc6bb5593binghamtona6
>
> Sat Sep 19 08:41:06 2015 (2602): Shutting down
>
> Sat Sep 19 08:41:06 2015 (8572): Shutting down
>
> Sat Sep 19 08:41:06 2015 (3204): Shutting down
>
> Sat Sep 19 08:42:06 2015 (8570): Shutting down
>
> Sat Sep 19 08:43:04 2015 (1362): <font color="yellow">INFO: Read
> configuration</font>
>
> Sat Sep 19 08:43:04 2015 (1362): Detected 64bit platform (1)
>
> Sat Sep 19 08:43:04 2015 (1362): unable to contact DNS defaulting to on
> disk file dns_hosts.tab
>
> Sat Sep 19 08:43:04 2015 (1362): Name vita.ostfalia.de no IP found
>
> Sat Sep 19 08:43:04 2015 (1362): Name loncapa2.vcu.edu no IP found
>
> Sat Sep 19 08:43:04 2015 (1362): Name loncapa.et-inf.fho-emden.de no IP
> found
>
> Sat Sep 19 08:43:04 2015 (1362): Name loncapa.calpoly.edu no IP found
>
>
> this goes on for all the servers.
>
>
> Thinking that something in the OS didn?t finish starting before lon-capa, I
> restarted lon-capa via loncontrol.
>
>
> That cleared up ?unable to contact DNS defaulting to on disk file
> dns_hosts.tab? message and
>
> now I see this in lonnet which seems to always be there:
>
>
> Sat Sep 19 08:48:36 2015 (2812): <font color="yellow">INFO: Read
> configuration</font>
>
> Sat Sep 19 08:48:36 2015 (2812): Detected 64bit platform (1)
>
> Sat Sep 19 08:49:04 2015 (2825): <font color="yellow">INFO: Read
> configuration</font>
>
> Sat Sep 19 08:49:04 2015 (2825): Detected 64bit platform (1)
>
> Sat Sep 19 08:49:05 2015 (2825): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
>
> Sat Sep 19 08:49:05 2015 (2825): Name loncapa.brs.cgresd.net no IP found
>
> Sat Sep 19 08:49:05 2015 (2825): Name tuvok.physast.uga.edu no IP found
>
> Sat Sep 19 08:49:05 2015 (2825): Name BCHS-CAPA.jeffco.k12.co.us no IP found
>
> Sat Sep 19 08:49:06 2015 (2825): Name loncapa.nfcc.edu no IP found
>
> Sat Sep 19 08:49:06 2015 (2825): caching of id -> iphost%3aiphost failed
>
> Sat Sep 19 08:49:07 2015 (2980): <font color="yellow">INFO: Read
> configuration</font>
>
> Sat Sep 19 08:49:07 2015 (2980): Detected 64bit platform (1)
>
> Sat Sep 19 08:49:07 2015 (2980): <span style='color:yellow;'>INFO: Read
> file types</span>
>
> Sat Sep 19 08:49:07 2015 (2980): caching of id ->
> dns%3a%2fadm%2fdns%2fhosts failed
>
> Sat Sep 19 08:49:08 2015 (2980): caching of id ->
> dns%3a%2fadm%2fdns%2fdomain failed
>
> Sat Sep 19 08:49:08 2015 (2980): Name loncapa.brs.cgresd.net no IP found
>
> Sat Sep 19 08:49:08 2015 (2980): Name tuvok.physast.uga.edu no IP found
>
> Sat Sep 19 08:49:08 2015 (2980): Name BCHS-CAPA.jeffco.k12.co.us no IP found
>
> Sat Sep 19 08:49:08 2015 (2980): Name loncapa.nfcc.edu no IP found
>
> Sat Sep 19 08:49:08 2015 (2980): caching of id -> iphost%3aiphost failed
>
> Sat Sep 19 08:49:11 2015 (2995): <font color="yellow">INFO: Read
> configuration</font>
>
> Sat Sep 19 08:49:11 2015 (2995): Detected 64bit platform (1)
>
> Sat Sep 19 08:49:13 2015 (3012): <font color="yellow">INFO: Read
> configuration</font>
>
> Sat Sep 19 08:49:13 2015 (3012): Detected 64bit platform (1)
>
> Sat Sep 19 08:49:14 2015 (3013): Name loncapa.brs.cgresd.net no IP found
>
> Sat Sep 19 08:49:14 2015 (3013): Name tuvok.physast.uga.edu no IP found
>
> Sat Sep 19 08:49:14 2015 (3013): Name BCHS-CAPA.jeffco.k12.co.us no IP found
>
> Sat Sep 19 08:49:14 2015 (3013): Name loncapa.nfcc.edu no IP found
>
>
> I still have the ?Map not loaded: The file
> /home/httpd/html/userfiles/binghamton/71215825fc2344762binghamtonl1/default.sequence
> does not exist? message and cannot get into my test course.
>
>
> Any ideas?
>
>
> Thanks,
>
> Bob Gonzales
More information about the LON-CAPA-admin
mailing list