Akips Device dependencies/alert fatigue

Kristof Vandoorsselaere

Dear Community, we are using akips in a university college. Several times per year we are experiencing planned or unplanned power outages for a building or floor. This results in a flood of status messages about devices being down.  When a full building is experiencing a power outage, the number of alerts we receive (in slack), is way to high (1 event for every single switch and AP), this is causing alert fatigue.

It would be nice if Akips could group or correlate all these alerts.

Is there any possibility to create device dependencies?  Or even better it would be nice if Akips could automatically learn dependencies (eg based on lldp) to reduce the number of alerts and prevent alert fatigue.

How are other people solving this issue? How do you prevent a flood of alerts in case of power failure.

Kind regards,

Kristof

Comments

2 comments

Date Votes
  • Comment author
    Troy Mellon

    I had to create a script that would check the status of a device's neighbor(s) before sending an alert.  I got the neighbor list from CDP and LLDP in AKiPS.  This runs as a subroutine from my alerting script.  Hope that helps.

    sub custom_upstream_neighbor_check
     
    ## Check neighbor device during alert checking
     
    {  
     
        my ($subdevice) = @_; ## Use list assignment to directly assign $subdevice
     
        my @monitored_devices = adb_result("mlist device * any group Alerting");  ## Group of devices eligible to generate alerts
    my @lldp_monitored_devices = adb_result("mlist device * any group LLDP");  ## Devices that don't do CDP
        my %good_devices = map { $_ => 1 } @monitored_devices;
        my %lldp_good_devices = map { $_ => 1 } @lldp_monitored_devices;
     
        my $page = "No";
     
    my $one_good_neighbor = "No";  ## Checking to see that there is at least one monitored neighbor to check
     
        for my $line (adb_result(sprintf("mget * %s * CISCO-CDP-MIB.cdpCacheDeviceId", $subdevice))) {
            my ($subdevice, $port_check, undef, undef, $neighbor) = split(" ", $line, 5);
            $neighbor =~ s/\..*//;
            if ((exists($good_devices{$neighbor})) and ($page ne "Yes")){  ## If the neighbor is monitored and page not set
                my (undef, undef, undef, undef, $neighbor_port_check) = split(" ", (adb_result(sprintf("mget * %s %s CISCO-CDP-MIB.cdpCacheDevicePort", $subdevice, $port_check))), 5);  ## This returns the port on the neighbor's side
                print "Checking subdevice $subdevice port $port_check Neighbor port check $neighbor $neighbor_port_check\n";
    $one_good_neighbor = "Yes";
                next if ($neighbor_port_check eq "GigabitEthernet0/0" or $neighbor_port_check eq "mgmt0");  ## Discounts management ports
     
                if ($page ne "Yes") { ## If page is already Yes, no need to check further
    my ($ping_state) = split(",", (adb_result(sprintf("get %s ping4 PING.icmpState", $neighbor))), 3);
    $page = "Yes" if ($ping_state eq "2");
                }
            }
        }
     
        if ($page ne "Yes") { ## If page is already Yes, no need to check further
     
    for my $line (adb_result(sprintf("mget * %s * LLDP-MIB.lldpRemSysName", $subdevice))) {
    my ($subdevice, $port_check, undef, undef, $neighbor) = split(" ", $line, 5);
    $neighbor =~ s/\..*//;
    if ((exists($lldp_good_devices{$neighbor})) and ($page ne "Yes")){  ## If the neighbor is monitored and page not set
    my (undef, undef, undef, undef, $neighbor_port_check) = split(" ", (adb_result(sprintf("mget * %s %s LLDP-MIB.lldpRemSysName", $subdevice, $port_check))), 5);  ## This returns the port on the neighbor's side
    print "Checking subdevice $subdevice port $port_check Neighbor port check $neighbor $neighbor_port_check\n";
    $one_good_neighbor = "Yes";
    next if ($neighbor_port_check eq "GigabitEthernet0/0" or $neighbor_port_check eq "mgmt0");
     
    if ($page ne "Yes") { ## If page is already Yes, no need to check further
    my ($ping_state) = split(",", (adb_result(sprintf("get %s ping4 PING.icmpState", $neighbor))), 3);
    $page = "Yes" if ($ping_state eq "2");
    }
    }
    }
    }
     
        if ($page ne "Yes") { ## If page is already Yes, no need to check further
     
    for my $line (adb_result("mlist device * any group z-No-Upstream-CDP")) {  ## Device has to be manually assigned to this group if the upstream device is not CDP capable
    if ($line eq $subdevice)  {
    $page = "Yes";
    print "No upstream neighbor - page $subdevice\n";
    }
    }
    }
     
     
    if ($one_good_neighbor eq "No")  {   ## If there are no CDP neighbors
    $page = "Yes";
    }
    $page = "Yes" unless defined(adb_result(sprintf("mget * %s * CISCO-CDP-MIB.cdpCacheDeviceId", $subdevice)));
     
        print "Upstream neighbor check result - ok to page $subdevice? $page\n";
        return $page;
    }
    ## End custom_upstream_neighbor_check

     

    0
  • Comment author
    Kristof Vandoorsselaere

    thanks for your solution @troy. I'm going to try it out.

    0

Please sign in to leave a comment.