[LON-CAPA-admin] Resources not replicating

[Stuart Raeburn]

Hi,

> Recently, I had to switch servers when one suffered from a hard disk
> failure.

Was the "new" server previously used as a LON-CAPA server?

If so, what was switched on the access server?
IP address, hostname, lonHostID?

> <font color="blue">WARNING: LWP get: 403 Forbidden:
> /home/httpd/html/res/....*.problem</font>

That warning is reported from &repcopy() in  
/home/httpd/lib/perl/Apache/lonnet.pm on the LON-CAPA access server.

The URL which is being requested from the remote server (presumably  
the library server for the fsu domain) should be:

http://loncapa10.fsu.edu/raw/....*.problem

The entry in /etc/httpd/conf/loncapa_apache.conf which controls access  
to /raw is:

<LocationMatch "^/+raw.*">
PerlAccessHandler Apache::lonracc
</LocationMatch>

The handler routine in /home/httpd/lib/perl/Apache/lonracc.pm will  
return a FORBIDDEN (403 return code) from the following code:

(a)

my $reqhost = $r->get_remote_host(REMOTE_NOLOOKUP);
my @hostids= &Apache::lonnet::get_hosts_from_ip($reqhost);
if (!@hostids && $reqhost ne '127.0.0.1' ) {
     $r->log_reason("Unable to find a host for ".
                    $r->get_remote_host(REMOTE_NOLOOKUP));
     return FORBIDDEN;
}


and (b)

     foreach my $id (@hostids) {
         my $uri =$r->uri;
         if (($filename=~/\.meta$/) ||
             ($uri=~m|^/raw/uploaded|) ||
             (-e "$filename.$id") ||
             &subscribed($filename,$id) ) {
             return OK;
         } else {
             $return=FORBIDDEN;
             push(@ids,$id);
         }
     }
     if ($return == FORBIDDEN) {
         $r->log_reason(join(':', at ids)." not subscribed", $r->filename);
         return FORBIDDEN;
     }

If you look in the Apache error log (/etc/httpd/logs/error_log or  
/etc/httpd/logs/ssl_error_log) on the library server you should find  
which of (a) or (b) are responsible -- "Unable to find a host" or "not  
subscribed".

The fact that the .meta files appear to be replicated but the other  
files are not suggests that:

&subscribed($filename,$id)

is returning 0.

The subscribed routine in lonracc.pm returns 0 in the following cases:

(a) No $filename.subscription file found

or

(b) Invalid IP address

     my $hostname=&Apache::lonnet::hostname($id);
     my (undef,undef,undef,undef,$ip) = gethostbyname($hostname);

     return 0 if (length($ip) != 4);

or

(c) The expected pattern for: lonHostID:IP address for the requesting  
server (the access server) is not found in the subscription file for  
the resource on the library server

     $ip=inet_ntoa($ip);
     my $expr='^'.quotemeta($id).':'.quotemeta($ip).':';
     my $found=0;
     if (my $sh=Apache::File->new("$filename.subscription")) {
         while (my $subline=<$sh>) { if ($subline =~ /$expr/) { $found=1; } }
         $sh->close();
     }
     return $found;


I would check that:

     my $reqhost = $r->get_remote_host(REMOTE_NOLOOKUP);
     my @hostids= &Apache::lonnet::get_hosts_from_ip($reqhost);

return the correct values when your access server requests a file in  
/raw from the library server.

I would also check that:

     my $hostname=&Apache::lonnet::hostname($id);
     my (undef,undef,undef,undef,$ip) = gethostbyname($hostname);

on the library server return the expected hostname and IP address for  
the lonHostID of the requesting access server.

I would also check the contents of the $filename.subscription file in  
/home/httpd/html/res/fsu on the library server for one of the files  
which fails replication.

One thing I note about the entries in the cluster tables for the fsu  
servers is that they are all identified as using http, but web  
requests to the fsu servers get rewritten to https, and the identity  
of the Apache SSL certificates installed there can not be verified,  
(and I need to agree to an exception), when pointing a web browser at  
any of the fsu LON-CAPA servers.


Stuart Raeburn
LON-CAPA Academic Consortium


Quoting hkng <hkng at fsu.edu>:

> Hi,
>
> Recently, I had to switch servers when one suffered from a hard disk
> failure. The new server does not replicate the resources though the meta
> files are under the /home/httpd/html/res/ directories. Checking lonnet.log,
> there are a number of messages that say
>
> <font color="blue">WARNING: LWP get: 403 Forbidden:
> /home/httpd/html/res/....*.problem</font>
>
> 403 indicates file permission (?) but I cannot determine while file? Any
> pointers?
>
> Thanks,
> -hk
>