Linux Filesystem Monitoring with email alert


#1

Could not find a good monitoring script for filesystem space consumed, so I wrote one. Will only send one alert per violation and will send an alert cleared. Documented pretty good.

Please take a look and let me know if you find anything wrong…

Could not find a good monitoring script for filesystem space consumed, so I wrote one. Will only send one alert per violation and will send an alert cleared. Documented pretty good.

Please take a look and let me know if you find anything wrong…

Does anyone know how to disable the bolding after a pound sign on the forum that would be great as well.

#!/bin/bash
#########

fs_mon.sh - Script to monitor disk space and email if USEDPCT is over PCT_ALERT

Self contained script with no options.

Edit script to change email, EXLUDE_LIST etc…

Notes: The script uses a function loop to check each volume from the df command output.

The script checks to ensure the alert is not already tripped for the volume

and that it is over the threshold then update the EMAILTXT and FS_LOG accordingly.

It will also check and remove the volume once it falls below the threshold and email.

usage: ./fs_mon.sh execute from command line, crontab, at or OEM schedule

Variable list

ADM_EMAIL = Email list to send alert to admins.

PCT_ALERT = The percent the filesystem must be greater than or equal to to trigger the alert

FS_LOG = A logfile that keeps track of alerts

FS_LOG_CUR = A logfile that keeps track of current alert

output = Formatted output from the df -HIP command

USEDPCT = Percentage of the Filesystem that is used

CKVOL = The Filesystem volume the function was looking at with a space after it

CURVOLERR = Current Volume Error set to number if Current Volume already exists in FS_LOG

SALERT = X, Y or N Set to send alert email of not.

Written: 20181128 Drew (ASH)

20181203 ASH - Added logging to ensure error message would not get sent each time script is run.

20181207 ASH - Added removal of volume from FS_LOG when alert is cleared and email status

20181210 ASH - Added more documentation to the header and blank line removal at bottom of script

###############################################

Set admin email so that you can get email.

ADM_EMAIL=“root”

set alert level %

PCT_ALERT=80

Exclude list of unwanted monitoring, if several partions then use “|” to separate the CKVOLs.

An example: EXCLUDE_LIST="/dev/hdd1|/dev/hdc5"

EXCLUDE_LIST=""

Temporary files

FS_LOG=/tmp/fs_mon.log
EMAILTXT=/tmp/mk_amail_fs_mon.txt

Set SALERT to default of X

SALERT=X

Cleanup

rm -rf $EMAILTXT

#::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

function main_prog() {
while read output;
do
USEDPCT=$(echo $output | awk '{print 1}' | cut -d'%' -f1) CKVOL=(echo $output | awk ‘{print $2,""}’)
if [ -s $FS_LOG ] ; then
CURVOLERR=cat $FS_LOG | grep "$CKVOL"
fi
if [ “x$CURVOLERR” != “x” ] ; then
export SALERT=N
fi
if [ $USEDPCT -ge $PCT_ALERT ] && [ “$SALERT” != “N” ] ; then
export SALERT=Y
echo "Running out of space “$CKVOL USEDPCT%\" on server (hostname), $(date)” >> $EMAILTXT
echo “$CKVOL” >> $FS_LOG
fi
if [ $USEDPCT -le $PCT_ALERT ] && [ “$SALERT” = “N” ] ; then
echo “Cleared - Running out of space “$CKVOL USEDPCT%\" on server (hostname), $(date)” >> EMAILTXT export SALERT=Y # Using % as the seperater for sed never knew you could that nice feature ignores the / for volume name sed -i "s@{CKVOL}@@” $FS_LOG
fi

Set variable to X for next pass through loop

export SALERT=X
done

Send email if EMAILTXT exists

if [ -f $EMAILTXT ] ; then
cat EMAILTXT | mail -s "Alert: Check disk space on (hostname)" $ADM_EMAIL
fi
}

if [ “EXCLUDE_LIST" != "" ] ; then df -HP | grep -vE "^Filesystem|tmpfs|cdrom|{EXCLUDE_LIST}” | awk ‘{print $5 " " $6}’ | main_prog
else
df -HP | grep -vE “^Filesystem|tmpfs|cdrom” | awk ‘{print $5 " " $6}’ | main_prog
fi

Cleanup to remove blank lines from the FS_LOG file

sed -i ‘/^$/d’ $FS_LOG

#END