Setup Splunk with Index and SearchHead Cluster
Jump to navigation
Jump to search
Purpose
- Setup a demo Lab with a clustered Splunk environment, based on CLI commands
- NOTE: Only to be used temporary in test environments
TestSetup
Host Environment
- /etc/hosts
#List of required Splunk instances with IP and Hostname in order of the installation #----------------------------------------------------------------------------------- #Index Cluster 192.168.178.109 splunk-cluster-master 192.168.178.110 splunk-idx-cluster-indexer-0 192.168.178.111 splunk-idx-cluster-indexer-1 192.168.178.112 splunk-idx-cluster-indexer-2 #Search Head Cluster 192.168.178.113 splunk-sh-cluster-search-head-0 192.168.178.114 splunk-sh-cluster-search-head-1 192.168.178.115 splunk-sh-cluster-search-head-2 192.168.178.116 splunk-sh-cluster-deployer #Monitoring 192.168.178.117 splunk-mc-monitoring-console #Universal Forwarder 192.168.178.118 splunk-uf-01 192.168.178.120 splunk-uf-02
Templates
The one in bold is used
splunk-8.2.11-e73c56f930c5-linux-2.6-amd64.deb splunk-8.2.11-e73c56f930c5-x64-release.msi splunk-9.0.5-e9494146ae5c-linux-2.6-amd64.deb splunkforwarder-9.0.5-e9494146ae5c-linux-2.6-amd64.deb splunkforwarder-9.0.5-e9494146ae5c-x64-release.msi
General
Environment requirements
- Debian Bookworm, Bridged VMs for each Splunk instance (x 11), each with 2CPUs, 4GB RAM, 64GB HD
- Working as root
- Fully network access to each other, no port restrictions
- Internet access everywhere (No Proxies!)
Splunk disclaimer
Important Note that this installation is based on the trial License which Splunk set at the first start.
Once started it counts for 60 days before the setup becomes unusable anymore.
Quickstart Reference
- Debian specific but may work on Ubuntu too
apt-get install curl dpkg -i splunk-8.2.11-e73c56f930c5-linux-2.6-amd64.deb /opt/splunk/bin/splunk start --accept-license --answer-yes /opt/splunk/bin/splunk enable web-ssl /opt/splunk/bin/splunk set web-port 443 /opt/splunk/bin/splunk restart
Quickdelete Reference
- Debian specific but may work on Ubuntu too
dpkg --purge --force-all splunk rm -rf /opt/splunk
Cluster
Cluster Master
- Start with the cluster master, note replication_factor and search_factor, the secret requires a min length
root@splunk-cluster-master:/opt/splunk/bin/splunk edit cluster-config -mode master -replication_factor 3 -search_factor 2 -secret idxSymmKey root@splunk-cluster-master:/opt/splunk/bin/splunk restart
Cluster Peer 0
root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk enable listen 9997 root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100 root@splunk-idx-cluster-indexer-0:/opt/splunk/bin/splunk restart
Cluster Peer 1
root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk enable listen 9997 root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100 root@splunk-idx-cluster-indexer-1:/opt/splunk/bin/splunk restart
Cluster Peer 2
root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk enable listen 9997 root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey -replication_port 9100 root@splunk-idx-cluster-indexer-2:/opt/splunk/bin/splunk restart
Search Head
Note: the secret requires a min length otherwise the setup will not work
SH0
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-0:8089 -replication_port 9200 -secret shclusterabcd root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart
SH1
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-1:8089 -replication_port 9200 -secret shclusterabcd root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk restart
SH2
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk init shcluster-config -mgmt_uri https://splunk-sh-cluster-search-head-2:8089 -replication_port 9200 -secret shclusterabcd root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk restart
Bootstrap
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list https://splunk-sh-cluster-search-head-0:8089,https://splunk-sh-cluster-search-head-1:8089,https://splunk-sh-cluster-search-head-2:8089 root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart
- It makes sense to check the setup after the above was applied. Check kvstore status and the mc
Add SH0 to Index Cluster
root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey root@splunk-sh-cluster-search-head-0:/opt/splunk/bin/splunk restart
Add SH1 to Index Cluster
root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey root@splunk-sh-cluster-search-head-1:/opt/splunk/bin/splunk restart
Add SH2 to Index Cluster
root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://splunk-cluster-master:8089 -secret idxSymmKey root@splunk-sh-cluster-search-head-2:/opt/splunk/bin/splunk restart
Universal Forwarder
Check This: https://docs.splunk.com/Documentation/Splunk/9.0.5/DistSearch/SHCandindexercluster
UF0
- /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log] disabled=false sourcetype=auth [monitor:///var/log/syslog] disabled=false sourcetype=syslog
- /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout] defaultGroup=indexer1 [tcpout:indexer1] server=192.168.178.110:9997,192.168.178.111:9997,192.168.178.112:9997
UF1
NOTE THAT UF1 USES ONLY ONE INDEXER
- /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/auth.log] disabled=false sourcetype=auth [monitor:///var/log/syslog] disabled=false sourcetype=syslog
- /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout] defaultGroup=indexer1 [tcpout:indexer1] server=192.168.178.110:9997
Testing
Happy testing
- Aftermath
- Switch to Distributed Mode from the mc web page of the https://splunk-cluster-master
- Switch off the MC on all Splunk Instances except the cluster-master
/opt/splunk/bin/splunk disable app splunk_monitoring_console
- consideration
- Remove the MC app from the SH Cluster(s) beforehand, just to make sure its being removed
- The MC must run only once clusterwide on: https://splunk-cluster-master - nowwhere else
Reference
- https://github.com/juanlittledevil/splunk-cluster-training/blob/master/README.md
- https://docs.splunk.com/Documentation/Splunk/9.0.4/Indexer/MultisiteCLI
- Nice: https://www.aplura.com/assets/pdf/shc_cheatsheet.pdf
- https://kinneygroup.com/blog/the-rules-of-the-search-head-cluster-shc/
- https://blog.avotrix.com/splunk-search-head-cluster-configuration/
- https://ashvinpandey.medium.com/steps-to-create-search-head-cluster-in-splunk-19106909e148