IPv6 · Monitoring · munin

Munin IPv6 neighbor state graphs

A recent issue with a Linux IPv6 firewall which saw on-link hosts appear to be flapping according to monitoring tools, highlighting a IPv6 ND table overflow problem. The short version of the solution required:

 net.ipv6.neigh.default.gc_thresh1 = 256  
 net.ipv6.neigh.default.gc_thresh2 = 1024  
 net.ipv6.neigh.default.gc_thresh3 = 2048  
To keep an eye on the neighbor table I created a series of munin scripts to produce lovely graphs like these:
9a1ad-net_neighbor_snr-day

The first is a script which collects the output from ‘ip -6 neigh’ stores it into one of five temporary files:

/etc/munin/cron_create_five

 #!/usr/bin/perl -w  
 # -*- perl -*-  
 $file_index = '/tmp/ip6_neigh.index';  
 unless (-e $file_index) {  
  open(INDEX, ">", $file_index);  
  print INDEX "1";  
 }  
 close(INDEX);  
 open(INDEX, "+<", $file_index);  
 $index = <INDEX>;  
 $index = int($index);  
 `ip -6 neigh > $file_index"."$index`;  
 seek(INDEX, 0, 0);  
 if($index >= 5) {  
  $new_index=1;  
 } else {  
  $new_index = $index+1;  
 }  
 print INDEX $new_index;  
 close(INDEX);  

The cron job to run every minute calling the above script:

/etc/cron.d/munin_ip_neigh_5_avg

 * * * * *     root     /etc/munin/cron_create_five  

Finally the munin graph plugin. Since munin updates every 5 minutes, these three scripts combine to poll ‘ip -6 neigh’ every minute and create a 5 minute average instead of a single snapshot on every update:

/usr/share/munin/plugins/net_neighbors_5_avg

 #!/usr/bin/perl -w  
 # -*- perl -*-  
 =head1 NAME  
 net_neighbors_5_avg - Plugin to monitor IPv6 neighbor numbers and state using a 5 minute average  
 =head1 CONFIGURATION  
 This plugin must run with root privileges  
 =head1 CONFIGURATION EXAMPLE  
 /etc/munin/plugin-conf.d/global or other file in that dir must contain:  
  [fw*]  
  user root  
 =head1 AUTHOR  
 Unknown snr1g11  
 =head1 LICENSE  
 GPLv2  
 =head1 MAGIC MARKERS  
  #%# family=auto  
  #%# capabilities=autoconf  
 =cut  
 if ( $ARGV[0] ) {  
   if ( $ARGV[0] eq 'autoconf' ) {  
      if ( -r '/proc/net/snmp') {  
        print "yes\n";  
        exit 0;  
      }  
      print "no\n";  
      exit 0;  
   } elsif ( $ARGV[0] eq 'config' ) {  
      print <<EOM;  
 graph_title IPv6 neighbour 5 avg count  
 graph_args -l 0  
 graph_vlabel number of hosts  
 graph_category network  
 ip6_neigh_global.label IPv6 neighbours: Globally routable  
 ip6_neigh_global.draw LINE1  
 ip6_neigh_global.type GAUGE  
 ip6_neigh_global.min 0  
 ip6_neigh_local.label IPv6 neighbours: Link Local  
 ip6_neigh_local.draw LINE1  
 ip6_neigh_local.type GAUGE  
 ip6_neigh_local.min 0  
 ip6_neigh_state_reach.label IPv6 neighbours: state REACHABLE  
 ip6_neigh_state_reach.draw LINE1  
 ip6_neigh_state_reach.type GAUGE  
 ip6_neigh_state_reach.min 0  
 ip6_neigh_state_reach.info REACHABLE indicates that address resolutinos has completed successfully. Any queued packets are transmitted immediately  
 ip6_neigh_state_stale.label IPv6 neighbours: state STALE  
 ip6_neigh_state_stale.draw LINE1  
 ip6_neigh_state_stale.type GAUGE  
 ip6_neigh_state_stale.min 0  
 ip6_neigh_state_stale.info STALE indicates the neighbor is not known to be reachable  
 ip6_neigh_state_delay.label IPv6 neighbours: state DELAY  
 ip6_neigh_state_delay.draw LINE1  
 ip6_neigh_state_delay.type GAUGE  
 ip6_neigh_state_delay.min 0  
 ip6_neigh_state_delay.info DELAY indicates the neighbor is considered no longer know to be reachable  
 ip6_neigh_state_probe.label IPv6 neighbours: state PROBE  
 EOM  
      exit 0;  
   }  
 }  
 #$ip6_neigh = '/tmp/ip6_neigh';  
 @outputs = ("/tmp/ip6_neigh.index.1", "/tmp/ip6_neigh.index.2", "/tmp/ip6_neigh.index.3", "/tmp/ip6_neigh.index.4", "/tmp/ip6_neigh.index.5");  
 $ip6_neigh_global = 0;  
 $ip6_neigh_local = 0;  
 $ip6_neigh_state_reach = 0;  
 $ip6_neigh_state_stale = 0;  
 $ip6_neigh_state_delay = 0;  
 $ip6_neigh_state_probe = 0;  
 foreach(@outputs) {    
   $ip6_neigh_global += int(`cat $_ | grep 2001 | wc -l`);  
   $ip6_neigh_local += (`cat $_ | grep fe80 | wc -l`);  
   $ip6_neigh_state_reach += (`cat $_ | grep REACHABLE | wc -l`);  
   $ip6_neigh_state_stale += (`cat $_ | grep STALE | wc -l`);  
   $ip6_neigh_state_delay += (`cat $_ | grep DELAY | wc -l`);  
   $ip6_neigh_state_probe += (`cat $_ | grep PROBE | wc -l`);  
 }  
 $ip6_neigh_global_avg = ($ip6_neigh_global / 5);  
 $ip6_neigh_local_avg = ($ip6_neigh_local / 5);  
 $ip6_neigh_state_reach_avg = ($ip6_neigh_state_reach / 5);  
 $ip6_neigh_state_stale_avg = ($ip6_neigh_state_stale / 5);  
 $ip6_neigh_state_delay_avg = ($ip6_neigh_state_delay / 5);  
 $ip6_neigh_state_probe_avg = ($ip6_neigh_state_probe / 5);  
 print "ip6_neigh_global.value $ip6_neigh_global_avg\n";  
 print "ip6_neigh_local.value $ip6_neigh_local_avg\n";  
 print "ip6_neigh_state_reach.value $ip6_neigh_state_reach_avg\n";  
 print "ip6_neigh_state_stale.value $ip6_neigh_state_stale_avg\n";  
 print "ip6_neigh_state_delay.value $ip6_neigh_state_delay_avg\n";  
 print "ip6_neigh_state_probe.value $ip6_neigh_state_probe_avg\n";  
 # vim:syntax=perl  
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s