Logd

About

Log Daemon - also known as logd or logrotate. This is a tool for IT-System Administrators which needs to archive, copy or delete data by a criteria, for example by a specific date.

Available Functions:

• ZipAndDelete - Used to compress data (optional recursive), then copy the data to a structured directory target (optional with a date indicator) and delete the source after.
  Good for log file archiving.

• Zip - Used to compress data (optional recursive), copy them to a structured directory (optional with a date indicator).
  Good for system backups

• Delete - Used to delete data (optional recursive). Good for temp files

• See below for more functions

Available Options:

• Uses a XML configuration file
• Target directory named with Year/Date
• Copy results via network to remote server using smb,sftp or other protocols.

Requirements

Logd requires Perl, either Perl from CPAN (Linux) or Active State (Windows)

• Required Modules:
  • POSIX
  • XML::Simple
  • HTTP::Date
  • File::Path
  • File::Find
  • File::stat
  • Digest::MD5
  • Archive::Zip
  • Archive::Tar;
  • File::Copy
  • Net::SFTP::Foreign

Note: When using the SFTP option then please install the Putty Suite including plink. When finished then please run a manual psftp transfer once in order to store the key files to the cache.

Installation

• With Debian:

apt-get install libxml-simple-perl 
apt-get install libarchive-zip-perl
apt-get install libnet-sftp-foreign-perl

• If you plan to psftp data to a remote host then install:

apt-get install putty
apt-get install plink

Note: Please follow the putty documentation about to create and use of a public and private key. We do not document this here because there are a lot of documetns in the internet about this topic.

• Create directories

# mkdir /usr/local/logd
# mkdir /usr/local/logd/tmp
# mkdir /var/log/logd
# mkdir /etc/logd

Usage

Windows

• The config paramter is optional, if not given then we expect the config within the working directory

>c:\perl\bin\perl.exe c:\mypath\logd.pl c:\mypath\optional\logd.xml

Linux

• The config paramter is optional, if not given then we expect the config within the working directory, these are the default paths:

#/usr/local/logd/logd.pl /etc/logd/logd.xml

Operating Mode

logd knows two operating modes yet, the default 1 is the default if not specified else, mode 1 supports the job mode ZipAndDelete, Zip, Delete, Move and Copy.
Mode 2 has the idea to operate like the linux logrotate tool but with different options, we will use a datestamp as prefix (yyyymmdd) and keep the original filename on the archive name, sample:

• Original logfile

/var/log/anylog.log

• Will be compressed and moved to:

/archive/logd/yyyyddmm-anylog.log.zip

• If multiple unique logs are occuring per day then we name the target:

/archive/logd/yyyyddmm#1-anylog.log.zip

• Continuing multiple logs per day:

/archive/logd/yyyyddmm#2-anylog.log.zip

Mode 2 is moving the logs into a temporary directory first (performance reason), then the syslog daemon gets opitonaly restartet to release the actual handles and then logd starts to compress the logs out of the temporary directory into the final target, when done then the log gets deleted from the temproary directory.

Security

logd knows two additional security features with opreating mode 2:

• Security Feature: MD5File, this option is creating a additional file inlcluded in the archive, the file contains:

Name	Value
File	The source file
MD5	The MD5 Hash of the source file
ModSizeBytes	The logfile size (Bytes)
ModSizeMegaBytes	The logfile size (MB)
ModDateTime	Modification Time
ModEpocheTime	Modification Epoche Time

• Security Feature: RandomDelayMin and RandomDelayMax. A possible attacker should not know at what time logd is starting, note that the syslog or syslog-ng daemon should be restarted after the log move is complete, within this time window a attacker could start processes and try to hide them from syslogging.

Configuration (logd.xml)

• logd.xml

Root	Job Root	Attribute	Value	Required	Operation Mode	Description
Config				Yes	1/2	XML Root attribute
	Setting			No	1/2	Job Root attribute
		WriteLog	String	No	1/2	Write Log
		LogPath	String	No	1/2	The path of the log, please leave a '/'
		RandomDelayMin	Integer	No	1/2	Minimum startup delay in seconds
		RandomDelayMax	Integer	No	1/2	Maximum startup delay in seconds
	Logpath		String	Yes	1/2	Job Root attribute , this is a ARRAY
		Name	String	Yes	1/2	Name of the Job can be any name can be equal names when using multiple times
		Mode	String	Yes	1	Job Mode, availble modes: ZipAndDelete Zip Tgz Delete Copy Move Sample: Mode="ZipAndDelete"
		MaxFileage	Intger	Yes	1	Amount of days of the file(s), we use the filesystem modification time
		Recursive	Boolean (0/1)	Yes	1/2	Recursive mode, only valid with ZipParents="1"
		Path	String	Yes	1/2	UNC Path to the logs, Note: THIS IS A RECOMMENDED UNIQUE STYLE Windows Recommended Sample: Target="C:/web/log/" Windows Recommended Sample UNC PATH: Target="//hostname/web/log/" Windows Sample UNC Path - NOT RECOMMENDED: Target="\\\\hostname\\\web\\log\\" Linux Sample: Target="/web/log/"
		Target	String	Yes	1/2	The target where the files are supposed to processed to, available modes: ZipAndDelete Zip Sample: Target="//host//archive/log//"
		IncludeMask	String	Yes	1/2	This is a CSV list to include files to the proces, example: IncludeMask=".log,.txt" This examples includes .log or .txt files
		ExcludeMask	String	No	1/2	This is a CSV list to exclude files to the proces, example: ExcludMask=".exe,.dat,RECYCLER,SYSTEM" This examples includes .exe or .dat files OR possible system files
		ZipParents	Boolean (0/1)	Yes	1/2	Keep the original path, recommendeded setting is 0 not needed with mode=tgz
		CreateDateDir	Boolean (0/1)	Yes	1/2	This param will create subdirectories in a date scheme, like: //server/backup/log/2012 //server/backup/log/2012/01 //server/backup/log/2012/02 ... This applies to: ZipAndDelete Zip Move Copy Not working with protocol=sftp
		CheckTargetDir	Boolean (0/1)	No	1	Create the same directory structure on the target as on the source, valid with copy and move, not compatible with CreateDateDir.
		DirectoryCleanUp	Boolean (0/1)	No	1	Delete empty directories (no files within the given directory), used with Delete or Move
		LeaveFiles	Integer	Yes	1	leave the given amount of files of the backup, this is a security feature, it keeps a minimum of files before deleting everything (think of config files)
		OperatingMode	Boolean (0/1)	No	1/2	The operating mode knows two options: 1 The default if not specified else 2 This mode runs simular then logrotate from linux, see further description
		TempPath	Boolean (0/1)	No	1/2	This is the temporary path for the logrotate mode or with the sftp protocol using operationmode 1
		Logsuffix	Boolean (0/1)	No	1/2	Use any text to be set within the archive name
		UseTodaysDate	Boolean (0/1)	No	1/2	Use todays date (yyyymmdd) instead of the cretation date of the file
		UseHostname	Boolean (0/1)	No	2	Use my hostname within the archive name
		RestartDaemon	Boolean (0/1)	No	2	Command to be applied after the log move
		MD5File	Boolean (0/1)	No	2	Additional MD5 File which will be included within the archive
		Protocol	string	No	1	Additional SFTP transfer, works only with zip and tgz
		User	string	No	1	User for the SFTP Server
		Password	string	No	1	Password for the SFTP Server (depreciated) - try to use keyfiles instead
		Server	string	No	1	SFTP Server
		KeyFile	string	No	1	Private keyfile to be used for the transfer

Samples

Zip and delete files

There are two jobs, the first is zipping and deliting files while the other delete files.

• 1:Zip all files older then 5 days from //myHost/myPath/Log/myApp, compress them to d:/myHost/myPath/ and delete the files after they have been compressed, no recursive walk
• 2:Delete all files older then 5 days, walk recursive

<?xml version="1.0"?>
 <CONFIG>
  <Setting WriteLog="1"/>
  <LogPath 
   Name="myAppLog" 
   Mode="ZipAndDelete"  
   Path="//myHost/myPath/Log/myApp" 
   Target="d:/myHost/myPath/" 
   CreateDateDir="1" 
   ZipParents="0" 
   Recursive="0" 
   MaxFileage="5" 
   ExcludeMask=".exe,.dat,-book" 
   IncludeMask=".log,txt"  
  /> 
  <LogPath 
   Name="myLog" 
   Mode="Delete"  
   Path="//myHost/myPath/Log/myApp" 
   CreateDateDir="0" 
   ZipParents="0" 
   Recursive="1" 
   MaxFileage="5" 
   ExcludeMask=".exe,.dat,-book" 
   IncludeMask=".log,txt"  
  />
 </CONFIG>

Copy files

Copy files, create target directory with a datestamp and use the today date as suffix

<LogPath
  Name="myBackup"
  Path="/usr/local/mysource/"
  Target="/usr/local/backup/"
  IncludeMask="sql.gz,tar.gz"
  ExcludeMask=""
  MaxFileage="0"
  Mode="Copy"
  CreateDateDir="1"
  Recursive="0"
  UseTodaysDate="1"
/>

Move files

Move files, create the target directory and use the today date as suffix. Use DirectoryCleanup to delete empty (history) directories.

<LogPath
 Name="MyTest"
 Path="c:/archive/"
 Target="c:/backup/"
 IncludeMask="zip"
 ExcludeMask=""
 MaxFileage="0"
 Mode="Move"
 CreateDateDir="0"
 Recursive="1"
 UseTodaysDate="0"
 CheckTargetDir="1"
 DirectoryCleanUp="1"
/>

Logrotate sample (Operating Mode 2)

Good to use on cloud servers

<?xml version="1.0"?>
 <CONFIG>
  <Setting WriteLog="1" LogPath="/tmp/" RandomDelayMin="600" RandomDelayMax="3600" />
  <LogPath 
  	Name="Web Logs" 
  	Path="/var/log/apache/" 
  	Target="/arcive/apache/" 
  	IncludeMask=".log" 
  	ExcludeMask="" 
  	MaxFileage="0" 
  	Mode="0" 
  	Recursive="0" 
  	ZipParents="0" 
  	CreateDateDir="1"
  	OperatingMode="2" 
  	TempPath="/tmp/apache/"
  	RestartDaemon="/etc/init.d/syslog restart"
  	MD5File="1"
  />
</CONFIG>

Delete files older then 10 days but leave minimum 5 files

<LogPath 
 Name="CUSTOMERBACKUP" 
 Path="//myHost/customer/reports/backup/" 
 Target="0" IncludeMask=".log,txt,.gz" 
 ExcludeMask=".exe,.dat,-book,RECYCLER,SYSTEM" 
 MaxFileage="10"  
 Mode="Delete" 
 Recursive="0" 
 ZipParents="0" 
 CreateDateDir="0" 
 LeaveFiles="5"
/>

TAR and Gzip /etc and transfer using sftp

This is good to run on a cloud server, the script will compress and copy date using psftp.

<LogPath 
	Name="ETC" 
	Path="/etc/" 
	IncludeMask="." 
	ExcludeMask="" 
	MaxFileage="0" 
       Mode="tgz" 
	CreateDateDir="0"
	Recursive="1" 
	Protocol="sftp"
	TempPath="/usr/local/dev/logd/test/tmp/"
	User="hhbackup"
	Server="svn"
	KeyFile="/root/.ssh/private.ppk"
/>

Backup Data

• Backup data from host myHost, copy the archive to myTarget

<LogPath 
 Name="Backup myHost" 
 Path="//myHost/d$/data/" 
 Target="//myTarget/backups/myHost/" 
 IncludeMask="inventory.txt,database,anything" 
 ExcludeMask=".exe,.dat,RECYCLER,SYSTEM" 
 MaxFileage="0" 
 Mode="ZIP" 
 Recursive="0" 
 ZipParents="0" 
 CreateDateDir="1"
/>

Download the script