Setup Splunk with Index and SearchHead Cluster

From Coolscript
Revision as of 07:17, 5 July 2023 by Admin (talk | contribs) (Created page with " =Purpose= *Setup a demo Lab with a clustered Splunk environment, based on CLI commands '''<br> *NOTE: Only to be used temporary in test environments''' =TestSetup= ==Host...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Purpose

  • Setup a demo Lab with a clustered Splunk environment, based on CLI commands
  • NOTE: Only to be used temporary in test environments

TestSetup

Host Environment

  • /etc/hosts
#List of required Splunk instances with IP and Hostname in order of the installation
#-----------------------------------------------------------------------------------
#Index Cluster
192.168.178.109 splunk-cluster-master
192.168.178.110 splunk-idx-cluster-indexer-0
192.168.178.111 splunk-idx-cluster-indexer-1
192.168.178.112 splunk-idx-cluster-indexer-2
#Search Head Cluster
192.168.178.113 splunk-sh-cluster-search-head-0
192.168.178.114 splunk-sh-cluster-search-head-1
192.168.178.115 splunk-sh-cluster-search-head-2
192.168.178.116 splunk-sh-cluster-deployer
#Monitoring
192.168.178.117 splunk-mc-monitoring-console
#Universal Forwarder
192.168.178.118 splunk-uf-01
192.168.178.120 splunk-uf-02

Templates

The one in bold is used 

splunk-8.2.11-e73c56f930c5-linux-2.6-amd64.deb
splunk-8.2.11-e73c56f930c5-x64-release.msi
splunk-9.0.5-e9494146ae5c-linux-2.6-amd64.deb
splunkforwarder-9.0.5-e9494146ae5c-linux-2.6-amd64.deb
splunkforwarder-9.0.5-e9494146ae5c-x64-release.msi

General

Environment requirements

  • Debian Bookworm, Bridged VMs for each Splunk instance (x 11), each with 2CPUs, 4GB RAM, 64GB HD
  • Working as root
  • Fully network access to each other, no port restrictions
  • Internet access everywhere (No Proxies!)

Splunk disclaimer

Important Note that this installation is based on the trial License which Splunk set at the first start.
Once started it counts for 60 days before the setup becomes unusable anymore.

Quickstart Reference

  • Debian specific but may work on Ubuntu too
apt-get install curl
dpkg -i splunk-8.2.11-e73c56f930c5-linux-2.6-amd64.deb
/opt/splunk/bin/splunk start --accept-license --answer-yes
/opt/splunk/bin/splunk enable web-ssl
/opt/splunk/bin/splunk set web-port 443
/opt/splunk/bin/splunk restart

Quickdelete Reference

  • Debian specific but may work on Ubuntu too
dpkg --purge --force-all splunk
rm -rf /opt/splunk

Cluster

Cluster Master

  • Start with the cluster master, note replication_factor and search_factor, the secret requires a min length
root@splunk-cluster-master:/opt/splunk/bin/splunk edit cluster-config -mode master -replication_factor 3 -search_factor 2 -secret idxSymmKey
root@splunk-cluster-master:/opt/splunk/bin/splunk restart

Cluster Peer 0

root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk enable listen 9997
root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100
root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk restart

Cluster Peer 1

root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk enable listen 9997
root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100
root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk restart

Cluster Peer 2

root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk enable listen 9997
root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100
root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk restart

Search Head

Note: the secret requires a min length otherwise the setup will not work

SH0

root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-0:8089 -replication_port 9200 -secret shclusterabcd
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart

SH1

root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-1:8089 -replication_port 9200 -secret shclusterabcd
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk restart

SH2

root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-2:8089 -replication_port 9200 -secret shclusterabcd
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk restart

Bootstrap

root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list https://splunk-sh-cluster-search-head-0:8089,https://splunk-sh-cluster-search-head-1:8089,https://splunk-sh-cluster-search-head-2:8089
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart
  • It makes sense to check the setup after the above was applied. Check kvstore status and the mc

Add SH0 to Index Cluster

root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart

Add SH1 to Index Cluster

root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk restart

Add SH2 to Index Cluster

root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk restart

Universal Forwarder

Check This: https://docs.splunk.com/Documentation/Splunk/9.0.5/DistSearch/SHCandindexercluster

UF0

  • /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log]
disabled=false
sourcetype=auth
 
[monitor:///var/log/syslog]
disabled=false
sourcetype=syslog
  • /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup=indexer1

[tcpout:indexer1]
server=192.168.178.110:9997,192.168.178.111:9997,192.168.178.112:9997

UF1

NOTE THAT UF1 USES ONLY ONE INDEXER

  • /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log]
disabled=false
sourcetype=auth
 
[monitor:///var/log/syslog]
disabled=false
sourcetype=syslog
  • /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup=indexer1

[tcpout:indexer1]
server=192.168.178.110:9997

Testing

Happy testing

  • Aftermath
    • Switch to Distributed Mode from the mc web page of the https://splunk-cluster-master
    • Switch off the MC on all Splunk Instances except the cluster-master
/opt/splunk/bin/splunk disable app splunk_monitoring_console
  • consideration
    • Remove the MC app from the SH Cluster(s) beforehand, just to make sure its being removed
    • The MC must run only once clusterwide on: https://splunk-cluster-master - nowwhere else

Reference

Retrieved from "https://coolscript.org/index.php?title=Setup_Splunk_with_Index_and_SearchHead_Cluster&oldid=952"