Troubleshooting sudden high disk usage

 

 

 
  
Once you can see the reason that disk space has increased, you can use the below methods to address it: 
  
SYSLOG  
Message retention time 
You can set how long you want to keep Syslog and Trap messages in days and can reduce it from the default 365 days, if appropriate.

Navigate to Admin > General > Miscellaneous-> Syslog / Trap History

Note: This is a combined Syslog/Trap retention period for all Syslog/Trap devices. 

You cannot change the Syslog or Trap retention period for a single Syslog or Trap source, nor can you remove only the data from that specific source. 
  
Filter out a noisy device

To filter out a noisy device, specify rules that drop noisy or unwanted SYSLOG or Traps. This will only filter out traffic from the point that it is configured. This won't affect messages also ingested into AKIPS. 
Navigate to Admin > General > Syslog / Trap Filters 
  
Netflow  
Use the Netflow Exporters page to identify which exporter is consuming the disk space and reduce its history retention if that is appropriate for your organisation. 

Note: This can remove the historical data used by that specific Netflow Export.  
Navigate to Admin -> General -> NetFlow Exporters 
  
A picture containing chart

Description automatically generated 
 
  
LLDP Usage 
This may cause an increase in the Discover database usage. In the below output, you can see from November that the amount of space recorded for the Discover increase from almost 0% to 30%. 
  
A picture containing chart

Description automatically generated 
  
Validate that the issue is with the LLDP data collection by navigating to Admin > System > System Info > Files > File Listing  and search for walk.lldp

Below is an example where the LLDP walk file has massively increased. 

There is no workaround for this issue and the only option is to delete the file from the AKIPS CLI and disable the LLDP reports under Admin > General > Miscellaneous > LLDP
  
/home/akips/data/discover: 
-rw-r--r-- 1 akips akips uarch 469709045609 Nov 11 15:00:01 2021 walk.lldp 
  
Long Running Packet Captures 
If you cannot see what has contributed to the high disk space usage, it may be due to long running packet captures. 

When you run an SNMP Walk, we also start a packet capture in the background. 

The packet capture may not terminate correctly, and the PCAP file stored locally on your AKIPS’ server will keep expanding, potentially filling up the disk and causing AKIPS to stop.  
Verify by navigating to Admin > System > System Info > Directory Usage 

Search for tmp/walk 

The individual files are under File Listing below it on the left. 

Restarting AKIPS (Admin > System > System Shutdown > Restart Software) will:

  • stop the packet captures running and 
  • remove the log and PCAP files from the tmp/walk directory
  • free up disk space.

 

Unknown Cause  
If you cannot find the cause of your sudden disk space increase, then please open a ticket with AKIPS’ support, providing your System Logs so that we can investigate. 

  1. Navigate to Admin  > System > System Log Viewer
  2. Click download System Logs

Upload the log file to https://www.akips.com/upload and note when you first noticed that the disk space was increasing in the details section at the bottom of the uploads page.

Was this article helpful?

/

Comments

0 comments

Please sign in to leave a comment.