- 1 About
- 2 Requirements
- 3 Installation
- 4 Usage
- 5 Operating Mode
- 6 Configuration (logd.xml)
- 7 Samples
Log Daemon - also known as logd or logrotate. This is a tool for IT-System Administrators which needs to archive, copy or delete data by a criteria, for example by a specific date.
- ZipAndDelete - Used to compress data (optional recursive), then copy the data to a structured directory target (optional with a date indicator) and delete the source after.
Good for log file archiving.
- Zip - Used to compress data (optional recursive), copy them to a structured directory (optional with a date indicator).
Good for system backups
- Delete - Used to delete data (optional recursive). Good for temp files
- See below for more functions
- Uses a XML configuration file
- Target directory named with Year/Date
- Copy results via network to remote server using smb,sftp or other protocols.
Logd requires Perl, either Perl from CPAN (Linux) or Active State (Windows)
- Required Modules:
Note: When using the SFTP option then please install the Putty Suite including plink. When finished then please run a manual psftp transfer once in order to store the key files to the cache.
- With Debian:
apt-get install libxml-simple-perl apt-get install libarchive-zip-perl apt-get install libnet-sftp-foreign-perl
- If you plan to psftp data to a remote host then install:
apt-get install putty apt-get install plink
Note: Please follow the putty documentation about to create and use of a public and private key. We do not document this here because there are a lot of documetns in the internet about this topic.
- Create directories
# mkdir /usr/local/logd # mkdir /usr/local/logd/tmp # mkdir /var/log/logd # mkdir /etc/logd
- The config paramter is optional, if not given then we expect the config within the working directory
>c:\perl\bin\perl.exe c:\mypath\logd.pl c:\mypath\optional\logd.xml
- The config paramter is optional, if not given then we expect the config within the working directory, these are the default paths:
logd knows two operating modes yet, the default 1 is the default if not specified else, mode 1 supports the job mode ZipAndDelete, Zip, Delete, Move
Mode 2 has the idea to operate like the linux logrotate tool but with different options, we will use a datestamp as prefix (yyyymmdd) and keep the original filename on the archive name, sample:
- Original logfile
- Will be compressed and moved to:
- If multiple unique logs are occuring per day then we name the target:
- Continuing multiple logs per day:
Mode 2 is moving the logs into a temporary directory first (performance reason), then the syslog daemon gets opitonaly restartet to release the actual handles and then logd starts to compress the logs out of the temporary directory into the final target, when done then the log gets deleted from the temproary directory.
logd knows two additional security features with opreating mode 2:
- Security Feature: MD5File, this option is creating a additional file inlcluded in the archive, the file contains:
|File||The source file|
|MD5||The MD5 Hash of the source file|
|ModSizeBytes||The logfile size (Bytes)|
|ModSizeMegaBytes||The logfile size (MB)|
|ModEpocheTime||Modification Epoche Time|
- Security Feature: RandomDelayMin and RandomDelayMax. A possible attacker should not know at what time logd is starting, note that the syslog or syslog-ng daemon should be restarted after the log move is complete, within this time window a attacker could start processes and try to hide them from syslogging.
|Root||Job Root||Attribute||Value||Required||Operation Mode||Description|
|Config||Yes||1/2||XML Root attribute|
|Setting||No||1/2||Job Root attribute|
|LogPath||String||No||1/2||The path of the log, please leave a '/'|
|RandomDelayMin||Integer||No||1/2||Minimum startup delay in seconds|
|RandomDelayMax||Integer||No||1/2||Maximum startup delay in seconds|
|Logpath||String||Yes||1/2||Job Root attribute , this is a ARRAY|
|Name||String||Yes||1/2|| Name of the Job
|Mode||String||Yes||1|| Job Mode, availble modes:|
|MaxFileage||Intger||Yes||1||Amount of days of the file(s), we use the filesystem modification time|
|Recursive||Boolean (0/1)||Yes||1/2||Recursive mode, only valid with ZipParents="1"|
|Path||String||Yes||1/2|| UNC Path to the logs, Note: THIS IS A RECOMMENDED UNIQUE STYLE|
|Target||String||Yes||1/2|| The target where the files are supposed to processed to, available modes:|
|IncludeMask||String||Yes||1/2|| This is a CSV list to include files to the proces, example:|
This examples includes .log or .txt files
|ExcludeMask||String||No||1/2|| This is a CSV list to exclude files to the proces, example:|
This examples includes .exe or .dat files OR possible system files
|ZipParents||Boolean (0/1)||Yes||1/2|| Keep the original path, recommendeded setting is 0|
not needed with mode=tgz
|CreateDateDir||Boolean (0/1)||Yes||1/2|| This param will create subdirectories in a date scheme, like:
//server/backup/log/2012 //server/backup/log/2012/01 //server/backup/log/2012/02 ...
This applies to:
Not working with protocol=sftp
|CheckTargetDir||Boolean (0/1)||No||1||Create the same directory structure on the target as on the source, valid with copy and move, not compatible with CreateDateDir.|
|DirectoryCleanUp||Boolean (0/1)||No||1||Delete empty directories (no files within the given directory), used with Delete or Move|
|LeaveFiles||Integer||Yes||1|| leave the given amount of files of the backup, this is a security feature, it keeps a minimum of
files before deleting everything (think of config files)
|OperatingMode||Boolean (0/1)||No||1/2|| The operating mode knows two options:
|TempPath||Boolean (0/1)||No||1/2||This is the temporary path for the logrotate mode or with the sftp protocol using operationmode 1|
|Logsuffix||Boolean (0/1)||No||1/2||Use any text to be set within the archive name|
|UseTodaysDate||Boolean (0/1)||No||1/2||Use todays date (yyyymmdd) instead of the cretation date of the file|
|UseHostname||Boolean (0/1)||No||2||Use my hostname within the archive name|
|RestartDaemon||Boolean (0/1)||No||2||Command to be applied after the log move|
|MD5File||Boolean (0/1)||No||2||Additional MD5 File which will be included within the archive|
|Protocol||string||No||1||Additional SFTP transfer, works only with zip and tgz|
|User||string||No||1||User for the SFTP Server|
|Password||string||No||1||Password for the SFTP Server (depreciated) - try to use keyfiles instead|
|KeyFile||string||No||1||Private keyfile to be used for the transfer|
Zip and delete files
There are two jobs, the first is zipping and deliting files while the other delete files.
- 1:Zip all files older then 5 days from //myHost/myPath/Log/myApp, compress them to d:/myHost/myPath/ and delete the files after they have been compressed, no recursive walk
- 2:Delete all files older then 5 days, walk recursive
<?xml version="1.0"?> <CONFIG> <Setting WriteLog="1"/> <LogPath Name="myAppLog" Mode="ZipAndDelete" Path="//myHost/myPath/Log/myApp" Target="d:/myHost/myPath/" CreateDateDir="1" ZipParents="0" Recursive="0" MaxFileage="5" ExcludeMask=".exe,.dat,-book" IncludeMask=".log,txt" /> <LogPath Name="myLog" Mode="Delete" Path="//myHost/myPath/Log/myApp" CreateDateDir="0" ZipParents="0" Recursive="1" MaxFileage="5" ExcludeMask=".exe,.dat,-book" IncludeMask=".log,txt" /> </CONFIG>
Copy files, create target directory with a datestamp and use the today date as suffix
<LogPath Name="myBackup" Path="/usr/local/mysource/" Target="/usr/local/backup/" IncludeMask="sql.gz,tar.gz" ExcludeMask="" MaxFileage="0" Mode="Copy" CreateDateDir="1" Recursive="0" UseTodaysDate="1" />
Move files, create the target directory and use the today date as suffix. Use DirectoryCleanup to delete empty (history) directories.
<LogPath Name="MyTest" Path="c:/archive/" Target="c:/backup/" IncludeMask="zip" ExcludeMask="" MaxFileage="0" Mode="Move" CreateDateDir="0" Recursive="1" UseTodaysDate="0" CheckTargetDir="1" DirectoryCleanUp="1" />
Logrotate sample (Operating Mode 2)
Good to use on cloud servers
<?xml version="1.0"?> <CONFIG> <Setting WriteLog="1" LogPath="/tmp/" RandomDelayMin="600" RandomDelayMax="3600" /> <LogPath Name="Web Logs" Path="/var/log/apache/" Target="/arcive/apache/" IncludeMask=".log" ExcludeMask="" MaxFileage="0" Mode="0" Recursive="0" ZipParents="0" CreateDateDir="1" OperatingMode="2" TempPath="/tmp/apache/" RestartDaemon="/etc/init.d/syslog restart" MD5File="1" /> </CONFIG>
Delete files older then 10 days but leave minimum 5 files
<LogPath Name="CUSTOMERBACKUP" Path="//myHost/customer/reports/backup/" Target="0" IncludeMask=".log,txt,.gz" ExcludeMask=".exe,.dat,-book,RECYCLER,SYSTEM" MaxFileage="10" Mode="Delete" Recursive="0" ZipParents="0" CreateDateDir="0" LeaveFiles="5" />
TAR and Gzip /etc and transfer using sftp
This is good to run on a cloud server, the script will compress and copy date using psftp.
<LogPath Name="ETC" Path="/etc/" IncludeMask="." ExcludeMask="" MaxFileage="0" Mode="tgz" CreateDateDir="0" Recursive="1" Protocol="sftp" TempPath="/usr/local/dev/logd/test/tmp/" User="hhbackup" Server="svn" KeyFile="/root/.ssh/private.ppk" />
- Backup data from host myHost, copy the archive to myTarget
<LogPath Name="Backup myHost" Path="//myHost/d$/data/" Target="//myTarget/backups/myHost/" IncludeMask="inventory.txt,database,anything" ExcludeMask=".exe,.dat,RECYCLER,SYSTEM" MaxFileage="0" Mode="ZIP" Recursive="0" ZipParents="0" CreateDateDir="1" />