Logd

From Coolscript
Jump to: navigation, search


About

Log Daemon - also known as logd or logrotate. This is a tool for IT-System Administrators which needs to archive, copy or delete data by a criteria, for example by a specific date.

Available Functions:

  • ZipAndDelete - Used to compress data (optional recursive), then copy the data to a structured directory target (optional with a date indicator) and delete the source after.
    Good for log file archiving.
  • Zip - Used to compress data (optional recursive), copy them to a structured directory (optional with a date indicator).
    Good for system backups
  • Delete - Used to delete data (optional recursive). Good for temp files
  • See below for more functions


Available Options:

  • Uses a XML configuration file
  • Target directory named with Year/Date
  • Copy results via network to remote server using smb,sftp or other protocols.



Requirements

Logd requires Perl, either Perl from CPAN (Linux) or Active State (Windows)

  • Required Modules:
    • POSIX
    • XML::Simple
    • HTTP::Date
    • File::Path
    • File::Find
    • File::stat
    • Digest::MD5
    • Archive::Zip
    • Archive::Tar;
    • File::Copy
    • Net::SFTP::Foreign


Note: When using the SFTP option then please install the Putty Suite including plink. When finished then please run a manual psftp transfer once in order to store the key files to the cache.

Installation

  • With Debian:
apt-get install libxml-simple-perl 
apt-get install libarchive-zip-perl
apt-get install libnet-sftp-foreign-perl
  • If you plan to psftp data to a remote host then install:
apt-get install putty
apt-get install plink

Note: Please follow the putty documentation about to create and use of a public and private key. We do not document this here because there are a lot of documetns in the internet about this topic.


  • Create directories
# mkdir /usr/local/logd
# mkdir /usr/local/logd/tmp
# mkdir /var/log/logd
# mkdir /etc/logd




Usage


Windows

  • The config paramter is optional, if not given then we expect the config within the working directory
>c:\perl\bin\perl.exe c:\mypath\logd.pl c:\mypath\optional\logd.xml


Linux

  • The config paramter is optional, if not given then we expect the config within the working directory, these are the default paths:
#/usr/local/logd/logd.pl /etc/logd/logd.xml


Operating Mode

logd knows two operating modes yet, the default 1 is the default if not specified else, mode 1 supports the job mode ZipAndDelete, Zip, Delete, Move and Copy.
Mode 2 has the idea to operate like the linux logrotate tool but with different options, we will use a datestamp as prefix (yyyymmdd) and keep the original filename on the archive name, sample:

  • Original logfile
/var/log/anylog.log
  • Will be compressed and moved to:
/archive/logd/yyyyddmm-anylog.log.zip
  • If multiple unique logs are occuring per day then we name the target:
/archive/logd/yyyyddmm#1-anylog.log.zip
  • Continuing multiple logs per day:
/archive/logd/yyyyddmm#2-anylog.log.zip

Mode 2 is moving the logs into a temporary directory first (performance reason), then the syslog daemon gets opitonaly restartet to release the actual handles and then logd starts to compress the logs out of the temporary directory into the final target, when done then the log gets deleted from the temproary directory.

Security

logd knows two additional security features with opreating mode 2:

  • Security Feature: MD5File, this option is creating a additional file inlcluded in the archive, the file contains:
Name Value
File The source file
MD5 The MD5 Hash of the source file
ModSizeBytes The logfile size (Bytes)
ModSizeMegaBytes The logfile size (MB)
ModDateTime Modification Time
ModEpocheTime Modification Epoche Time


  • Security Feature: RandomDelayMin and RandomDelayMax. A possible attacker should not know at what time logd is starting, note that the syslog or syslog-ng daemon should be restarted after the log move is complete, within this time window a attacker could start processes and try to hide them from syslogging.



Configuration (logd.xml)

  • logd.xml


Root Job Root Attribute Value Required Operation Mode Description
Config Yes 1/2 XML Root attribute
Setting No 1/2 Job Root attribute
WriteLog String No 1/2 Write Log
LogPath String No 1/2 The path of the log, please leave a '/'
RandomDelayMin Integer No 1/2 Minimum startup delay in seconds
RandomDelayMax Integer No 1/2 Maximum startup delay in seconds
Logpath String Yes 1/2 Job Root attribute , this is a ARRAY
Name String Yes 1/2 Name of the Job
  • can be any name
  • can be equal names when using multiple times
Mode String Yes 1 Job Mode, availble modes:
  • ZipAndDelete
  • Zip
  • Tgz
  • Delete
  • Copy
  • Move

Sample:

Mode="ZipAndDelete"
MaxFileage Intger Yes 1 Amount of days of the file(s), we use the filesystem modification time
Recursive Boolean (0/1) Yes 1/2 Recursive mode, only valid with ZipParents="1"
Path String Yes 1/2 UNC Path to the logs, Note: THIS IS A RECOMMENDED UNIQUE STYLE
  • Windows Recommended Sample:
Target="C:/web/log/"
  • Windows Recommended Sample UNC PATH:
Target="//hostname/web/log/"
  • Windows Sample UNC Path - NOT RECOMMENDED:
Target="\\\\hostname\\\web\\log\\"
  • Linux Sample:
Target="/web/log/"
Target String Yes 1/2 The target where the files are supposed to processed to, available modes:
  • ZipAndDelete
  • Zip


Sample:

Target="//host//archive/log//"
IncludeMask String Yes 1/2 This is a CSV list to include files to the proces, example:
IncludeMask=".log,.txt"

This examples includes .log or .txt files

ExcludeMask String No 1/2 This is a CSV list to exclude files to the proces, example:
ExcludMask=".exe,.dat,RECYCLER,SYSTEM"

This examples includes .exe or .dat files OR possible system files

ZipParents Boolean (0/1) Yes 1/2 Keep the original path, recommendeded setting is 0

not needed with mode=tgz

CreateDateDir Boolean (0/1) Yes 1/2 This param will create subdirectories in a date scheme, like:
//server/backup/log/2012
//server/backup/log/2012/01
//server/backup/log/2012/02
...

This applies to:

  • ZipAndDelete
  • Zip
  • Move
  • Copy

Not working with protocol=sftp

CheckTargetDir Boolean (0/1) No 1 Create the same directory structure on the target as on the source, valid with copy and move, not compatible with CreateDateDir.
DirectoryCleanUp Boolean (0/1) No 1 Delete empty directories (no files within the given directory), used with Delete or Move
LeaveFiles Integer Yes 1 leave the given amount of files of the backup, this is a security feature, it keeps a minimum of

files before deleting everything (think of config files)

OperatingMode Boolean (0/1) No 1/2 The operating mode knows two options:
  • 1 The default if not specified else
  • 2 This mode runs simular then logrotate from linux, see further description
TempPath Boolean (0/1) No 1/2 This is the temporary path for the logrotate mode or with the sftp protocol using operationmode 1
Logsuffix Boolean (0/1) No 1/2 Use any text to be set within the archive name
UseTodaysDate Boolean (0/1) No 1/2 Use todays date (yyyymmdd) instead of the cretation date of the file
UseHostname Boolean (0/1) No 2 Use my hostname within the archive name
RestartDaemon Boolean (0/1) No 2 Command to be applied after the log move
MD5File Boolean (0/1) No 2 Additional MD5 File which will be included within the archive
Protocol string No 1 Additional SFTP transfer, works only with zip and tgz
User string No 1 User for the SFTP Server
Password string No 1 Password for the SFTP Server (depreciated) - try to use keyfiles instead
Server string No 1 SFTP Server
KeyFile string No 1 Private keyfile to be used for the transfer


Samples

Zip and delete files

There are two jobs, the first is zipping and deliting files while the other delete files.

  • 1:Zip all files older then 5 days from //myHost/myPath/Log/myApp, compress them to d:/myHost/myPath/ and delete the files after they have been compressed, no recursive walk
  • 2:Delete all files older then 5 days, walk recursive
<?xml version="1.0"?>
 <CONFIG>
  <Setting WriteLog="1"/>
  <LogPath 
   Name="myAppLog" 
   Mode="ZipAndDelete"  
   Path="//myHost/myPath/Log/myApp" 
   Target="d:/myHost/myPath/" 
   CreateDateDir="1" 
   ZipParents="0" 
   Recursive="0" 
   MaxFileage="5" 
   ExcludeMask=".exe,.dat,-book" 
   IncludeMask=".log,txt"  
  /> 
  <LogPath 
   Name="myLog" 
   Mode="Delete"  
   Path="//myHost/myPath/Log/myApp" 
   CreateDateDir="0" 
   ZipParents="0" 
   Recursive="1" 
   MaxFileage="5" 
   ExcludeMask=".exe,.dat,-book" 
   IncludeMask=".log,txt"  
  />
 </CONFIG>

Copy files

Copy files, create target directory with a datestamp and use the today date as suffix

<LogPath
  Name="myBackup"
  Path="/usr/local/mysource/"
  Target="/usr/local/backup/"
  IncludeMask="sql.gz,tar.gz"
  ExcludeMask=""
  MaxFileage="0"
  Mode="Copy"
  CreateDateDir="1"
  Recursive="0"
  UseTodaysDate="1"
/>

Move files

Move files, create the target directory and use the today date as suffix. Use DirectoryCleanup to delete empty (history) directories.

<LogPath
 Name="MyTest"
 Path="c:/archive/"
 Target="c:/backup/"
 IncludeMask="zip"
 ExcludeMask=""
 MaxFileage="0"
 Mode="Move"
 CreateDateDir="0"
 Recursive="1"
 UseTodaysDate="0"
 CheckTargetDir="1"
 DirectoryCleanUp="1"
/>


Logrotate sample (Operating Mode 2)

Good to use on cloud servers

<?xml version="1.0"?>
 <CONFIG>
  <Setting WriteLog="1" LogPath="/tmp/" RandomDelayMin="600" RandomDelayMax="3600" />
  <LogPath 
  	Name="Web Logs" 
  	Path="/var/log/apache/" 
  	Target="/arcive/apache/" 
  	IncludeMask=".log" 
  	ExcludeMask="" 
  	MaxFileage="0" 
  	Mode="0" 
  	Recursive="0" 
  	ZipParents="0" 
  	CreateDateDir="1"
  	OperatingMode="2" 
  	TempPath="/tmp/apache/"
  	RestartDaemon="/etc/init.d/syslog restart"
  	MD5File="1"
  />
</CONFIG>

Delete files older then 10 days but leave minimum 5 files

<LogPath 
 Name="CUSTOMERBACKUP" 
 Path="//myHost/customer/reports/backup/" 
 Target="0" IncludeMask=".log,txt,.gz" 
 ExcludeMask=".exe,.dat,-book,RECYCLER,SYSTEM" 
 MaxFileage="10"  
 Mode="Delete" 
 Recursive="0" 
 ZipParents="0" 
 CreateDateDir="0" 
 LeaveFiles="5"
/>

TAR and Gzip /etc and transfer using sftp

This is good to run on a cloud server, the script will compress and copy date using psftp.

<LogPath 
	Name="ETC" 
	Path="/etc/" 
	IncludeMask="." 
	ExcludeMask="" 
	MaxFileage="0" 
       Mode="tgz" 
	CreateDateDir="0"
	Recursive="1" 
	Protocol="sftp"
	TempPath="/usr/local/dev/logd/test/tmp/"
	User="hhbackup"
	Server="svn"
	KeyFile="/root/.ssh/private.ppk"
/>

Backup Data

  • Backup data from host myHost, copy the archive to myTarget
<LogPath 
 Name="Backup myHost" 
 Path="//myHost/d$/data/" 
 Target="//myTarget/backups/myHost/" 
 IncludeMask="inventory.txt,database,anything" 
 ExcludeMask=".exe,.dat,RECYCLER,SYSTEM" 
 MaxFileage="0" 
 Mode="ZIP" 
 Recursive="0" 
 ZipParents="0" 
 CreateDateDir="1"
/>



Download the script
Download