Covering off some of the latest announcements at Splunk's user conference (.conf), an Add-on created to Splunk config files and also the presentation delivered at .conf18 on SplDevOps!

1. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Splunk User Group Edinburgh

2. © 2018 SPLUNK INC.
Harry McLaren
● Managing Consultant at ECS Security
● Splunk Enablement Lead & Member of Splunk Trust
● Leader of the Splunk User Group Edinburgh

3. © 2018 SPLUNK INC.
Introduction to ECS Security
Splunk Partner - UK
– Security Consultancy & Managed SOC Provider
– Splunk Revolution Award & Splunk Partner of the Year

4. © 2018 SPLUNK INC.
Agenda
• Housekeeping: Event Overview & House Rules
• Splunk .conf18 Updates (Harry McLaren)
• Security
• IT Ops
• Others (Docker)
• Splunking .conf Files (Tomasz Dziwok)
• Break
• SplDevOps (Harry McLaren & Ilias Diamantakos)

5. © 2018 SPLUNK INC.
Splunk [Official] User Group
“The overall goal is to create an authentic, ongoing
user group experience for our users, where
they contribute and get involved”
● Technical Discussions
● Sharing Environment
● Build Trust
● No Sales!

6. © 2018 SPLUNK INC.
Splunk .conf18
Updates
Harry McLaren

7. © 2018 SPLUNK INC.
Introducing Splunk Enterprise Security 5.2
Generally Available: 16/10/18

8. © 2018 SPLUNK INC.
Event Sequencing
Define Attacker Techniques via Multiple Matching Events
▶ The Event Sequencing Engine runs as a real-
time search and listens for incoming notable
events and risk modifiers that are triggered by
correlation searches.
▶ Transitions can also be configured to
aggregate notable events or risk modifiers that
may happen after a transition match is found.

9. © 2018 SPLUNK INC.
Event Sequencing
Define Attacker Techniques via Multiple Matching Events

10. © 2018 SPLUNK INC.
Use Case Library
ES Content Updates Type Function Integrated

11. © 2018 SPLUNK INC.
Investigation Workbench
Two New Artifact Types - File Name & URL

12. © 2018 SPLUNK INC.
Introducing Splunk Phantom Version 4.0
Security Orchestration, Automation, & Response (SOAR) Platform
▶ Clustering support for added performance and redundancy
• Enables Phantom to scale horizontally using additional instances for added performance
and redundancy
▶ Indicator View for threat intelligence style analysis
• Provides a new and important way to visualize security data on the Phantom platform.
Data is presented in the view organized by indicator, versus event, for easier threat-
intelligence style analysis.
▶ Native Splunk search support
• Splunk is now the default search engine shipped with the Phantom product. Users are
able to use their existing or new external Splunk instances to achieve a single source for
security data storage.
Elasticsearch engine remains an external option for those that prefer it.

13. © 2018 SPLUNK INC.
Introducing Splunk User Behaviour Analytics 4.2
Generally Available: 16/10/18
▶ User Feedback for machine learning
models provides anomaly customization and
improved threat detection accuracy
▶ Improved data ingestion performance by
up to 10x, with the new Splunk-to-Kafka UBA
ingestion connector.
Kafka ingestion does not require UBA to run
real-time indexed search queries on core
Splunk, rather uses micro-batched queries.
▶ Native single-sign-on authentication
support for multiple identity providers Okta,
Microsoft ADFS and Ping Identity

14. © 2018 SPLUNK INC.
Introducing Splunk ITSI 4.0
Predictive Analytics for Real-Time Insights
▶ KPI Predictions
We’re excited to deliver deeper insights into a
potential health degradation with KPI
Predictions. These utilize the breadth of data in
the platform to help predict KPIs like customer
experience, application workload, and
infrastructure health, in order to identify issues
or outages in advance.
▶ Predictive Cause Analysis
This new feature helps you drill down into the
specific services underlying a predicted issue to
proactively remediate and resolve it before
customer experience is impacted.

15. © 2018 SPLUNK INC.
Introducing Splunk SmartStore
Cut the Cord by Decoupling Compute and Storage
▶ Allowing compute and
storage tiers to be
independently scaled.
▶ Automatically evaluates
users’ data access
patterns to determine
which data needs to be
accessible for real-time
analytics and which data
should reside in lower
cost, long-term storage.

16. © 2018 SPLUNK INC.
Introducing Dynamic Data: Active Archive
Data Retention Options in Splunk Cloud
▶ Data Management
• Splunk provides complete lifecycle management of the
archive on your behalf and remains the custodian of your
data. Just like your Active Searchable data, Splunk
manages all aspects of archive availability, durability,
security and privacy requirements on your behalf.
▶ Data Restore
• Enables you to request a slice of your data to be restored
back into your Splunk Cloud instance. The entire workflow
is fully integrated into Splunk Web so your archived data is
available at your fingertips with predictable time between
retrieval to search.

17. © 2018 SPLUNK INC.
Other Features!
Selection of Interesting New Releases!
▶ Dark Mode heightens visual contrast within Splunk dashboards.
▶ Workload Management enables users to prioritize the allocation of compute and
memory resources used by Splunk on searches and alerts to ensure users’ most
critical analytics are completed first.
▶ Guided Data Onboarding is a new graphical user interface helping customers
move data into Splunk Cloud or Splunk Enterprise and guiding them through the
best onboarding methodology based on their specific architecture.
▶ Logs to Metrics helps configure and convert log events to metrics, enabling
users to take advantage of breakthrough performance when monitoring and
alerting on metrics with the Splunk platform.
▶ Health Report gives Splunk administrators immediate visibility into the overall
health status of their Splunk environments.

18. © 2018 SPLUNK INC.
Introducing Splunk Next
Splunk Works the Way Your Data Works
▶ Feedback from Splunk Customers
• Make it easier to access data with Splunk no matter where it lives or what format it is in.
• Make it easier to automate the actions and outcomes in order to drive the business forward.
• Make it possible for all kinds of people to ask questions of Splunk and get to answers, no matter
their role or where they might be in the world.
▶ What Does Splunk Next Do For You?
• Ask Questions: Open customers to a broader set of data sources.
• Get Answers: Empower a broader set of customers from IT and Security to Lines of Business.
• Take Action: Operate on data wherever it lives.

19. © 2018 SPLUNK INC.
Splunk Next
Experimental, Pre-release Features (Alpha/Beta)
▶ Splunk Developer Cloud: Write Splunk applications natively in the cloud.
▶ Splunk Business Flow: Analytics-driven approach into customer/user’s interactions and identify
ways to optimize those interactions and processes.
▶ Splunk Data Fabric Search: Seamlessly search across massive amounts of data and federated
searches across multiple instances.
▶ Splunk Data Stream Processor: Refine, modify and adjust data mid-stream and within
milliseconds before the data reaches its destination.
▶ Splunk Cloud Gateway: Secure cloud service with end-to-end encryption for easy mobile
engagement through a simple to install Splunk app for Mobile.
▶ Splunk Mobile: Actionable alerts and mobile-friendly dashboards on mobile devices through our
Splunk Mobile App.
▶ Splunk Natural Language: Query a system and ask question of Splunk without knowing SPL
▶ Splunk TV: View Splunk on any peripheral device instead of having to purchase a dedicated PC
▶ Splunk Augmented Reality: Enjoy direct access to the Splunk dashboard and live augmented
reality Splunk-powered gauges on top of real-world objects.

20. © 2018 SPLUNK INC.
Splunk on Docker
Containers are now a First-Class Citizen
▶ Splunk Support now covers Splunk Enterprise 7.2 deployments in Docker
containers, enabling customers to quickly deploy and scale Splunk based on their
organizations’ demands.

21. © 2018 SPLUNK INC.
Splunking .conf Files
Tomasz Dziwok

22. Splunk Configuration
Management
Tomasz Dziwok
ECS

23. > whoami | fields name, occupation
 ECS
 Associate Security Consultant / Engineer

24. Why configuration change
control/monitoring?
 Accountability
 Failure recovery
 Historical reference
 Training
 Enforcing change approval
 …

25. What’s the absolute best way to do this?

26. This is the best way to do it…

27. There is no best way…
 Version control; git
 Automation; jenkins
 Orchestration; ansible

28. Where to start?
 I recommend
 version control of configuration
 monitor .conf files for changes
 ideally store backup of configuration somewhere
 git
 can be simple
 monitor file changes
 But perhaps something easier…

29. Configuration Monitoring App
 DEMO

30. Open Source & SplunkBase
 https://gitlab.com/ecs_public_projects
 https://splunkbase.splunk.com/
 First open source software
 Open to feedback
 Will maintain on a best effort basis
 PRs welcome!

31. Thank you…
 Questions
 Comments
 Feedback

32. © 2018 SPLUNK INC.
Break
10mins

33. © 2018 SPLUNK INC.
SplDevOps
Harry McLaren & Ilias Diamantakos

34. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
SplDevOps
Making Splunk Development a Breeze with a
Deep Dive on DevOps, Containerization,
Version Control & Automation
Harry McLaren, Ilias Diamantakos, Tomasz Dziwok
October 2018 | Version 1.3

35. © 2018 SPLUNK INC.
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward-looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2018 Splunk Inc. All rights reserved.
Forward-Looking Statements

36. v
© 2018 SPLUNK INC.
HARRY MCLAREN
Splunk Enablement Lead, Managing Consultant
ILIAS DIAMANTAKOS
Splunk Engineer, Associate Consultant
Who Are We?
cyberharibu
ilias-diamantakos

37. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Best Security
Company
of the Year
Employer: ECS (UK Splunk Elite Partner)

38. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
What’s It All About?
▶ Customer Challenges
▶ What Do We Want?
▶ Our Idea to Deploy Splunk
▶ Technical Deep Dive
▶ Project Roadmap
▶ Key Takeaways
~40mins

39. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Customer Challenges
“The expansion of
Splunk has increased
operational complexity,
as we manage it
manually and can’t
keep on top of project
change requests.”
– High-Street Retailer
“We require a full route-
to-live to maintain
system integrity and
can’t deploy changes
fast enough in our
current setup.”
– National Bank
“Multiple developers
within the same DEV
environment, causes
repeated configuration
conflicts and delays to
planned changes.”
– National Building Society

40. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
What Do We Want?
Enterprises Want to Respond Quickly, Safely & With Less Risk
Rapid Changes
to Splunk Software
01
Orchestrated
Deployment
02
Fragile
Route-to-Live

Fail Safe,
Fast Backout
03
Development at Scale
• Enterprise Scale Development
• Synchronous Changes /
Multiple Admins & Developers
• Splunk Defined via Code
• Familiar Approach (AKA: DevOps/Agile)
Reduction in Custom Config
• Every ’Custom’ Configuration
Introduces Disparity
• Inconsistent Dev, Test, Pre-Prod, Prod
• Testing is “Best Endeavors”
• Increased Risk, Changes Batched

41. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Splunk for Agility
Supporting Agile Methodology by Default
Schema at Read,
Supporting Multiple Use Cases
Analytic Tools Exposed to UI,
Empowering Users to Experiment
Plain Text Configuration Files,
Documented & Supported
Splunk API is Enumerated,
Dev Licenses, Labs Encouraged
SPL
Web UI
Plain Text Config
Open API
Monitor
InvestigateIntelligence

42. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Our Idea
“SplDevOps” Became the Solution
Version Control
 Git[Lab] Utilized
 Multiple Projects/Branches
 Key Releases Tagged
Full Route-to-Live
 Multi-Stage Environments
 Dev > Pre-Prod > Prod
• Automated Testing
Agile Development
 Short Sprints
 Test Driven Development
 Issue Management
& Feature Backlog
Configuration Management
 Orchestrated Deployment
 Centralized Config
 Ansible used via SSH

43. © 2018 SPLUNK INC.
Project: Internal Monitoring
Ask: Deploy Splunk Internally for SecOps & ITOps Use Cases

44. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Automation Engine Containerization Version Control
Brief Background
Let’s talk tools!

45. © 2018 SPLUNK INC.
What Tool Fits Where?
a new Splunk infrastructure the DevOps way!
▶ Identical environments & route-to-live
• Development, Pre-production, Production
▶ Eliminate fear driven development
• It’s ok to make mistakes!
▶ Minimize direct production changes
• Always go through route-to-live
• Transparent change control
▶ Modern means of disaster recovery
▶ Security driven
← Ansible + Git + Docker + Python
← Docker + Git
← GitLab
← Ansible (IaC)
← Ansible Vault

46. © 2018 SPLUNK INC.
How We Wanted It To Look
Spoiler Alert: This is also the end result
dev pre-prod prod
IX: Splunk Indexer SH: Splunk Search Head DS: Splunk Deployment Server AS: Ansible Server

47. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Multiple Repositories
Of /opt/splunk/etc for each instance
Ansible & DS
IX
splunk_ix
SH
splunk_sh
Syslog Collector
syslog
ansible splunk_ds

48. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Git Workflow
aka “the change process”

49. © 2018 SPLUNK INC.
Everything starts from
our DevEnv
So let’s spin one up

50. © 2018 SPLUNK INC.
What’s going on in the background

51. © 2018 SPLUNK INC.
What’s going on in the background

52. © 2018 SPLUNK INC.
How It Looks

53. © 2018 SPLUNK INC.
How It Looks

54. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Let’s Share Secrets
No really, we are sharing!
▶ How to version sensitive information
• Encryption
▶ How to decrypt automatically
• Ansible Vault
▶ How to store Ansible Vault Password
• More encryption

55. © 2018 SPLUNK INC.
Let’s Decrypt
One password to rule them all
Ansible Server

56. © 2018 SPLUNK INC.
Use Case Scenario
Demo time

57. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
How it should have been done
Integrating with our change process

58. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
How it should have been done
Integrating with our change process

59. © 2018 SPLUNK INC.
How it gets deployed

60. © 2018 SPLUNK INC.
How it gets deployed

61. © 2018 SPLUNK INC.
How it gets deployed

62. © 2018 SPLUNK INC.
Lessons We Learned
Not everything was easy…
▶ Multiple repositories
• What goes where?
• Many lines of history
▶ Identical code for different environments
• There are always exceptions (Eventgen, production API calls)
▶ Data for different environments
• Production data is sensitive
▶ Automated deployment of code
• When do you restart?

63. © 2018 SPLUNK INC.
Deployment Results
Did it work!?

64. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Full
Route-to-Live
Implemented
in Production
Users &
Admins
Educated &
Empowered
Everything
Under Version
Control
Promoting
Changes in
~ 5mins
(Dev>Prod)
Foundations
Built for Future
Development
End Result
Prototype Success, Production Rollout

65. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Adaptable Framework
Expressed in Software
(Python + Git + Ansible)
Environment Agnostic &
Scales to Clustered
Deployments& Hybrid
Cloud Architecture
User Friendly &
End-to-End Integrated
with Issue/Change
Management
Roadmap
Introducing “Splunk Compiler” (v2.0+)

66. © 2018 SPLUNK INC.© 2018 SPLUNK INC.
Splunk Supports
Experimentation
by Default
Agile/DevOps
Methodologies
are Compatible
Doesn’t Require
Automation
Expertise
Version Control
BEFORE
Software
Orchestration
Key Takeaways
Remember Four Things…

67. © 2018 SPLUNK INC.
Don't forget to rate this session
in the .conf18 mobile app
Thank You

68. © 2018 SPLUNK INC.
Get Involved!
● Splunk User Group Edinburgh
– https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html
– https://www.linkedin.com/groups/12013212
● Splunk’s Slack Group
– Register via http://splunk-usergroups.signup.team/
– Channel: #edinburgh
● Present & Share at the User Group?
Connect:
‣ Harry McLaren | harry.mclaren@ecssecurity.co.uk | @cyberharibu | harrymclaren.co.uk
‣ ECS | enquiries@ecs.co.uk | @ECS_IT | ecs.co.uk