On July 19, 2024, CrowdStrike, a global cybersecurity company, experienced a significant outage caused by a faulty software update. This incident impacted millions of Windows machines across multiple industries, including transportation, defense, manufacturing, and finance. CrowdStrike has released an official statement and is posting updates on their blog. Microsoft has also published a blog with remediations, which we encourage you to review.
This blog is intended to help existing Splunk customers who are also customers of CrowdStrike gain visibility into how the CrowdStrike outage may be impacting their organizations.
With that in mind, we’d like to share a few detections and additional resources that may help you gain further visibility into how this event is impacting your organization.
Splunk Security Analytics
These detections are designed for use within Splunk Enterprise Security, our SIEM solution that supports our broader unified security and observability platform. For technical support, please reach out to us via the Support Portal.
Find machines with the broken update
This search requires the Splunk Add-on for CrowdStrike, an app that is used with data from Falcon Data Replicator. For more information on the sourcetypes produced by this app, refer to the Splunk documentation.
This search is designed to identify file downloads of the defective CrowdStrike content update on Windows systems by querying CrowdStrike sensor logs. It looks for events where the target file name contains the substring "C-00000291" and ends with ".sys", which might indicate a bad update file, specifically during the LFODownloadConfirmation event. The search is confined to a specific time range that CrowdStrike has provided.
index=* sourcetype="crowdstrike:events:sensor"
TargetFileName="*C-00000291*.sys*" event_simpleName=LFODownloadConfirmation
event_platform=Win earliest=1721362140 latest=1721366820
| stats count min(_time) as firstTime max(_time) as lastTime by aid aip
event_simpleName event_platform TargetFileName DownloadPath
| rename aid as device_id, aip as src_ip , TargetFileName as file_name
| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(firstTime)
| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(lastTime)
Sample logs from sourcetype="crowdstrike:events:sensor"
{
"event_simpleName": "LFODownloadConfirmation",
"ConfigStateHash": "3600681180",
"aip": "18.82.155.193",
"DownloadServer": "lfodown01-b.cloudsink.net",
"DownloadPath":
"metahash+/cfs/channelfiles/0000000291/bd9a5c40318869df823085f4449d4c9a/C-00000291-00000000-00000032.sys",
"DownloadPort": "443",
"ConfigBuild": "1007.3.0018408.1",
"event_platform": "Win",
"Entitlements": "15",
"name": "LFODownloadConfirmationV1",
"EventOrigin": "17",
"CompletionEventId": "Event_ChannelDataDownloadCompleteV1",
"id": "3aab720a-f12d-139d-883c-693105c5a42c",
"EffectiveTransmissionClass": "0",
"aid": "69483d8c3f0bb47758126f30921102e8",
"timestamp": "1721366804209",
"cid": "bd9a5c40318869df823085f4449d4c9a",
"TargetFileName": "C-00000291-00000000-00000032.sys"
}This data comes from a production environment. We have anonymized the sensitive fields “aip”, “id”, “cid”, and the “cid” value embedded in the “DownloadPath” field.
Identify machines that have not checked in with CrowdStrike
This search requires the Splunk Add-on for CrowdStrike and the CrowdStrike Devices Inventory for Splunk.
This search identifies Windows machines that have not checked in with CrowdStrike using data from the crowdstrike_devices lookup table and calculates a delta of when it was last seen from the date of the outage July 19, 2024, 04:09:00 UTC.
| inputlookup crowdstrike_devices
| rex field="category" "os_name: (?[^\|]+)"
| rex field="category" "dvc_type: (?[^\|]+)"
| rex field="category" "cs_last_seen: (?[^\|]+)"
| eval
cs_last_seen=strptime(cs_last_seen, "%x %T %Z"),
current_time=now(),
delta=abs(cs_last_seen - current_time)
| where match(os, "(?i)windows") AND cs_last_seen>=1721362140 AND cs_last_seen<=1721366820
| convert timeformat="%x %T %Z" ctime(cs_last_seen) ctime(current_time)
| eval delta_days=floor(delta/86400),
delta_hours=floor((delta%86400)/3600),
delta_minutes=floor((delta%3600)/60),
last_checked_in=mvappend(delta_days."d", delta_hours."h", delta_minutes."m")
| eval last_seen=mvjoin(last_checked_in, " ")
| rename nt_host as host, cs_last_seen as last_seen_by_crowdstrike
| table host, mac, ip, os, dvc_type, city, country, last_seen_by_crowdstrike, last_seen
| sort - delta
Additional Apps and Resources for Tracking Host Check-ins
In addition to these detections, there are several apps built by members of the Splunk Community and articles that can help with tracking host check-ins, which is particularly useful in this scenario:
Community-Built Apps on Splunkbase
- Meta Woot!
- TrackMe
- Broken Hosts App for Splunk
- Alerts for Splunk Admins
Articles and Documentation
- Splunk Docs: Monitoring Console
- Splunk Docs: Deployment Server
- Lantern article: Hosts logging data in a certain timeframe
- Post from duanewaddle.com: Proving a Negative
Summary
We’ve provided this security content to help Splunk customers gain further visibility into how the CrowdStrike outage may be impacting their organizations. This and the community’s response in general —sharing information, providing solutions, and supporting affected organizations —demonstrates the strength of our industry when faced with unexpected challenges.
While this event serves as a reminder of the potential risks in software updates — especially for products with deployed agents — it also underscores the overall benefits of automatic update processes in maintaining robust security postures.
Moving forward, this incident will likely drive improvements in update deployment and quality assurance processes across the industry. It reinforces the importance of maintaining robust incident response plans and the value of community collaboration in addressing cybersecurity challenges.
Contributors
We would like to thank Kevin Jarvis, Jonathan Heckinger, Emmanuel Jamison, Lou Stella, Bhavin Patel, Rod Soto, Eric McGinnis, Tyne Darke, Anne Ortel and Jose Hernandez for authoring this post, and Zachary Christensen for sharing how to identify machines that have not checked into Splunk.
Splunk Threat Research Team
The Splunk Threat Research Team is an active part of a customer’s overall defense strategy by enhancing Splunk security offerings with verified research and security content such as use cases, detection searches, and playbooks. We help security teams around the globe strengthen operations by providing tactical guidance and insights to detect, investigate and respond against the latest threats. The Splunk Threat Research Team focuses on understanding how threats, actors, and vulnerabilities work, and the team replicates attacks which are stored as datasets in theAttack Data repository.
Our goal is to provide security teams with research they can leverage in their day to day operations and to become the industry standard for SIEM detections. We are a team of industry-recognized experts who are encouraged to improve the security industry by sharing our work with the community via conference talks, open-sourcing projects, and writing white papers or blogs. You will also find us presenting our research at conferences such as Defcon, Blackhat, RSA, and many more.
Read moreSplunk Security Content.
