Man­ag­ing Log Files with the Logro­tate Util­ity

Log files, though use­ful to trou­bleshoot and to track usage, tend to use up valu­able disk space. Over time, they be­come large and un­wieldy, so pin­point­ing an event be­comes dif­fi­cult. Logro­tate per­forms the func­tion of archiv­ing a log file and start­ing a n

OpenSource For You - - Admin - By: Man­ish Sharma The au­thor has a master’s in com­puter ap­pli­ca­tions and is cur­rently work­ing as a tech­nol­ogy ar­chi­tect at In­fosys, Chandi­garh. He can be reached at cloudtechgig@gmail.com

Logro­tate has been de­signed to ease the ad­min­is­tra­tion of sys­tems that gen­er­ate large num­bers of log files in any for­mat. It al­lows au­to­matic ro­ta­tion, com­pres­sion, re­moval and mail­ing of log files. Each log file may be han­dled daily, ev­ery week, ev­ery month, or when it grows too large (ro­ta­tion on the ba­sis of a file’s size).

The ap­pli­ca­tion and the servers gen­er­ate too many logs, mak­ing the task of trou­bleshoot­ing or gain­ing busi­ness in­sights from these logs, a dif­fi­cult one. Many a time, there’s the is­sue of servers run­ning on low disk space be­cause of the very large log files on them.

Servers with huge log files cre­ate prob­lems when the re­siz­ing of vir­tual ma­chines needs to be done. Trou­bleshoot­ing based on large files may take up a lot of time and valu­able mem­ory. The logro­tate util­ity is ex­tremely use­ful to solve all such prob­lems. It helps in tak­ing back­ups of log files on an hourly, daily, weekly, monthly or yearly ba­sis with ad­di­tional choice of log backup with com­pres­sion. Also, file back­ups can be taken by set­ting a limit on the file size, like 100MB, for in­stance. So, after the log file reaches a size of 100MB, the file will be ro­tated.

The syn­op­sis is as fol­lows:

logro­tate [-dv] [-f|--force] [-s|--state file] con­fig_­file

Any num­ber of con­fig­u­ra­tion files can be given on the com­mand line, and one file can in­clude an­other con­fig file. A sim­ple logro­tate con­fig­u­ra­tion looks like what’s shown be­low:

/var/log/mes­sages { ro­tate 5 weekly com­press old­dir /var/log/backup/mes­sages/ missin­gok }

Here, ev­ery week, the /var/log/mes­sages file will be com­pressed and backed up to the /var/log/backup/ mes­sages/ folder, and only five ro­tated log files will be kept around in the sys­tem.

In­stalling logro­tate

Log ro­ta­tion is a util­ity that comes pre­in­stalled in Linux servers like Ubuntu, Cen­tOS, Red Hat, etc. Check the folder at path /etc/logro­tate.d. If it is not in­stalled, then you can in­stall it man­u­ally by us­ing the fol­low­ing com­mands.

For Ubuntu, type:

sudo apt-get in­stall logro­tate

For Cen­tOS, type: sudo yum in­stall logro­tate

Con­fig­ur­ing logro­tate

When logro­tate runs, it reads its con­fig­u­ra­tion files to de­cide where Fig­ure 1: The logro­tate util­ity to find the log files that it needs to ro­tate, how of­ten the files should be ro­tated and how many archived logs to keep. There are pri­mar­ily two ways to write a logro­tate script and con­fig­ure it to run ev­ery day, ev­ery week, ev­ery month, and so on.

1. Con­fig­u­ra­tion can be done in the de­fault global con­fig­u­ra­tion file /etc/logro­tate.conf; or

2. By cre­at­ing sep­a­rate con­fig­u­ra­tion files in the di­rec­tory/etc/logro­tate.d/ for each ser­vice/ap­pli­ca­tion. Per­son­ally, I think the lat­ter op­tion is a bet­ter way to write logro­tate con­fig­u­ra­tions, as each con­fig­u­ra­tion is sep­a­rate from the other. Some dis­tri­bu­tions use a vari­a­tion and scripts that run logro­tate daily can be found at any of the fol­low­ing paths:

/etc/cron.daily/logro­tate

/etc/cron.daily/logro­tate.cron

/etc/logro­tate.d/

One logro­tate con­fig­u­ra­tion (file­name: Tom­cat) file given be­low will be used to com­press and take daily back­ups of all Tom­cat .log files and catalina.out files and after ro­ta­tion, the Tom­cat ser­vice will get restarted. With this con­fig­u­ra­tion it is clear that mul­ti­ple log file back­ups can be taken in one go. Mul­ti­ple log files should be de­lim­ited with space.

/home/tom­cat/logs/*.log /home/tom­cat/logs/catalina.out { missin­gok copy­trun­cate daily com­press ro­tate 10 old­dir /var/log/backup/tom­cat/ shared­scripts postro­tate

/home/tom­cat/bin/catal­ins.sh res­tart > / dev/null end­script }

To check if the con­fig­u­ra­tion is func­tion­ing prop­erly, the com­mand given be­low with the –v op­tion can be used. Op­tion -v means ‘ver­bose’ so that we can view the progress made by the logro­tate util­ity.

The types of di­rec­tives

Given be­low are some use­ful di­rec­tives that can be in­cluded in the logro­tate con­fig­u­ra­tion file.

Missin­gok: Con­tin­ues ex­e­cut­ing the next con­fig­u­ra­tion in the file even if the log file is miss­ing, in­stead of throw­ing an er­ror. nomissin­gok: Throws an er­ror if the log file is miss­ing. com­press: Com­presses the log file in the .tar.gz for­mat. The file can com­press in an­other for­mat us­ing the com­press­cmd di­rec­tive.

com­press­cmd: Spec­i­fies the com­mand to use for log file com­pres­sion.

com­pres­sext: Spec­i­fies the ex­ten­sion to use on the com­pressed log file. Only ap­pli­ca­ble if the com­press op­tion is en­abled dur­ing con­fig­u­ra­tion.

copy: Makes a copy of the log file but it does not make any mod­i­fi­ca­tion in the orig­i­nal file. It is just like tak­ing a snapshot of the log file.

copy­trun­cate: Copies the orig­i­nal file con­tent and then trun­cates it. This is use­ful when some pro­cesses are writ­ing to the log file and can’t be stopped.

da­te­ext: Adds a date ex­ten­sion (de­fault YYYYMMDD), to back up the log file. Also see no­da­te­ext.

date­for­mat for­mat_string: Spec­i­fies the ex­ten­sion for da­te­ext. Only %Y%m %d and %s spec­i­fiers are al­lowed.

Ifempty: Ro­tates the log file even if it is empty.

Also see no­tifempty.

old­dir <di­rec­tory>: Ro­tated log files get moved in the spec­i­fied di­rec­tory. Over­rides noold­dir.

shared­scripts: This says that post­script will run once for mul­ti­ple con­fig­u­ra­tion files hav­ing the same log di­rec­tory.

For ex­am­ple, the di­rec­tory struc­ture /home/tom­cat/logs/*.log is the same for all log files placed in the logs folder, and in this case, post­script will run only once.

post­scripts: This runs when­ever a log is ro­tated in the con­fig­u­ra­tion file spec­i­fied block. The num­ber of post­script ex­e­cu­tions for logs placed in the same di­rec­tory can be over­rid­den with shared­scripts di­rec­tives.

Di­rec­tives are also re­lated to the in­ter­vals at which log files are ro­tated. They tell logro­tate how of­ten the log files should be ro­tated. The avail­able op­tions are:

1. Hourly (copy the file /etc/cron.daily/logro­tate into the / etc/cron.hourly/ di­rec­tory)

2. Daily

3. Weekly

4. Monthly

5. Yearly

Log files may also be ro­tated on the ba­sis of file size. We can in­struct logro­tate to ro­tate files when the size of the file is greater than, let’s say, 100KB, 100MB, 10GB, etc.

Some di­rec­tives tell logro­tate what num­ber of ro­tated files to keep be­fore delet­ing the old ones. In the fol­low­ing ex­am­ple, it will keep four ro­tated log files. ro­tate 4

You can also use di­rec­tives to re­move ro­tated logs that are older than X num­ber of days. The age is only checked if the log file is to be ro­tated. The files are mailed, in­stead of be­ing deleted, to the con­fig­ured ad­dress if mail­last and mail are con­fig­ured.

One can get the full list of com­mands used in logro­tate con­fig­u­ra­tion files by check­ing the man page:

man logro­tate

Logro­tate is one of the best util­i­ties avail­able in the Linux OS. It is ideal to take back­ups of ap­pli­ca­tions, servers or any logs. By writ­ing a script in the post­script sec­tion, we can move or copy back­ups of log files in Ama­zon s3 buck­ets as well.

Newspapers in English

Newspapers from India

© PressReader. All rights reserved.