[LON-CAPA-admin] lonmemcached_errors

Stuart Raeburn raeburn at msu.edu
Mon Feb 9 09:38:14 EST 2015

Hi Maged,

> Failed to write, and not due to blocking: Broken pipe

I have not encountered that error myself.
Looking at the source code for memcached from:


it appears that the error you saw originates in the function:

enum transmit_result transmit()

in memcached.c, and occurs when sendmsg() used to send a message on a  
socket returns 0 or -1, i.e.,no characters sent were successfully sent.

In this instance the result is "TRANSMIT_HARD_ERROR" and the  
connection state is set to "conn_closing".

 From the command line:
ps aux |grep memcached |grep -v grep
will display information about the memcached process.

netstat |grep localhost:memcache
will show information about connections to memcache.

If memcached is not running (but other LON-CAPA daemons are running)  
it can be started using /etc/init.d/loncontrol start

The global memcache configuration is in:

The command line script: memcached-tool can be used to will display  
statistics from a running memcached instance. See: man memcached-tool.  

memcached-tool display
memcached-tool stats

There is also: memcached-tool dump
which makes a partial dump of the cache. However, a custom script is  
probably a better way to dump the current contents of memcached. (see  

LON-CAPA itself uses the Cache:Memcached perl module, see:  

According to the Cache:Memcached documentation the connect_timeout  
defaults to .25 second, and the select_timeout defaults to 1 second.

The constructor used by LON-CAPA in /home/httpd/lib/perl/Apache/lonnet.pm is:

$memcache=new Cache::Memcached({'servers'           => [''],
                                 'compress_threshold'=> 20_000,

Ubuntu 12.04 LTS includes libcache-memcached-perl 1.29-1, and as you  
noted, version 1.4.13-0ubuntu2.1 of memcached itself.

The  perl script below uses Cache::Memcached and Data::Dumper to dump  
memcached's currently stored keys and values. However, if your  
server/VM is experienceing load problems, running this will script  
will likely exacerbate that.

I recommend sending the output from this script to a file, e.g.,
perl memcached_dump.pl > memcache_snapshot.txt



use Cache::Memcached;
use Data::Dumper;

# memcached_dump.pl
# February 9, 2015

use strict;

my $instance = "";
my $memd = new Cache::Memcached {
     'servers' => [ $instance],
     'debug' => 0,

my %containers;
my $stats = $memd->stats('items');
my $items = $stats->{hosts}->{$instance}->{items};
foreach my $line (split(/\r\n/,$items)) {
     $line =~ s/^.*:(.*):.*$/$1/ig;
     $containers{$line} = 1;

foreach my $container (sort(keys(%containers))) {
      my $result = $memd->stats("cachedump $container 0");
      my $contents = $result->{hosts}->{$instance}->{"cachedump $container 0"};

      foreach my $item (split(/\r\n/,$contents)) {
          my ($name,$size) = ($item =~ /^ITEM\s+(\S+)\s+\[([^;]+)/);
          $item =~ s/^ITEM (.*) \[.*$/$1/ig;
          my $val = $memd->get($item);
          print "$name $size ".Dumper($val)."\n";


Stuart Raeburn
LON-CAPA Academic Consortium

Quoting "Abdel Messeh, Maged" <mmesseh at illinois.edu>:

> Hi All,
> Recently I have seen an error showing in lonmemcached_errors, the   
> error message which repeats itself:
> Failed to write, and not due to blocking: Broken pipe
> At the same time I see a very high disk IO, and increased number of   
> apache processes.  This happened on 2 of my access nodes.
> This also was associated with many instances of Code Ran Too Long.
> I am currently running Ubuntu 12 with memcached 1.4.13.  My access   
> nodes have 6G of memory, during this time the free amount was around  
>  0.5G-1G
> Any thoughts on why I am seeing this? And what I can do about it?
> Many thanks,
> Maged
> _______________________________________________
> LON-CAPA-admin mailing list
> LON-CAPA-admin at mail.lon-capa.org
> http://mail.lon-capa.org/mailman/listinfo/lon-capa-admin

More information about the LON-CAPA-admin mailing list