kestrel-lang

.. image:: https://github.com/opencybersecurityalliance/kestrel-lang/raw/develop/logo/logo_w_text.svg :width: 432 :alt: Kestrel Threat Hunting Language

.. image:: https://img.shields.io/pypi/pyversions/kestrel-lang :target: https://www.python.org/ :alt: Python 3

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg :target: https://github.com/psf/black :alt: Code Style: Black

.. image:: https://codecov.io/gh/opencybersecurityalliance/kestrel-lang/branch/develop/graph/badge.svg?token=HM4ax10IW3 :target: https://codecov.io/gh/opencybersecurityalliance/kestrel-lang :alt: Code Coverage

.. image:: https://img.shields.io/pypi/v/kestrel-lang :target: https://pypi.python.org/pypi/kestrel-lang :alt: Latest Version

.. image:: https://img.shields.io/pypi/dm/kestrel-lang :target: https://pypistats.org/packages/kestrel-lang :alt: PyPI Downloads

.. image:: https://readthedocs.org/projects/kestrel/badge/?version=latest :target: https://kestrel.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status

|

[News] Kestrel session at Black Hat USA 2023_

---

Kestrel is a threat hunting language aiming to make cyber threat hunting fast by providing a layer of abstraction to build reusable, composable, and shareable hunt-flow. Starting with:

#. Black Hat USA 2022 session recording_ #. Black Hat USA 2022 Kestrel hunting lab_ #. Kestrel live tutorial in a cloud sandbox_

The Goal

Software developers write Python or Swift than machine code to quickly turn business logic into applications. Threat hunters write Kestrel to quickly turn threat hypotheses into hunt-flow. We see threat hunting as an interactive procedure to create customized intrusion detection systems on the fly, and hunt-flow is to hunts as control-flow is to ordinary programs.

What does it mean by hunt fast?

• Do NOT write the same TTP pattern in different data source queries.
• Do NOT write one-time-use adapaters to connect hunt steps.
• Do NOT waste your existing analytic scripts/programs in future hunts.
• Do construct your hunt-flow from smaller reuseable hunt-flow.
• Do share your huntbook with your future self and your colleagues.
• Do get interactive feedback and revise hunt-flow on the fly.

|

.. image:: https://github.com/opencybersecurityalliance/data-bucket-kestrel/raw/main/images/github_homepage_animation.gif :width: 90% :target: https://www.youtube.com/watch?v=tASFWZfD7l8 :alt: Kestrel Hunting Demo

Kestrel in a Nutshell

.. image:: https://github.com/opencybersecurityalliance/kestrel-lang/raw/develop/docs/images/overview.png :width: 100% :alt: Kestrel overview.

• Kestrel language: a threat hunting language for a human to express what to hunt.

  • expressing the knowledge of what in patterns, analytics, and hunt flows.
  • composing reusable hunting flows from individual hunting steps.
  • reasoning with human-friendly entity-based data representation abstraction.
  • thinking across heterogeneous data and threat intelligence sources.
  • applying existing public and proprietary detection logic as analytic hunt steps.
  • reusing and sharing individual hunting steps, hunt-flow, and entire huntbooks.

• Kestrel runtime: a machine interpreter that deals with how to hunt.

  • compiling the what against specific hunting platform instructions.
  • executing the compiled code locally and remotely.
  • assembling raw logs and records into entities for entity-based reasoning.
  • caching intermediate data and related records for fast response.
  • prefetching related logs and records for link construction between entities.
  • defining extensible interfaces for data sources and analytics execution.

Basic Concepts and Howto

Visit Kestrel documentation_ to learn Kestrel:

• Learn concepts and syntax:

  • A comprehensive introduction to Kestrel_
  • The two key concepts of Kestrel_
  • Interactive tutorial with quiz_
  • Language reference book_

• Hunt in your environment:

  • Kestrel runtime installation_
  • How to connect to your data sources_
  • How to execute an analytic hunt step in Python/Docker_
  • How to use Kestrel via API_
  • How to launch Kestrel as a Docker container_

Kestrel Huntbooks And Analytics

• Kestrel huntbook_: community-contributed Kestrel huntbooks
• Kestrel analytics_: community-contributed Kestrel analytics

Kestrel Hunting Blogs

#. Building a Huntbook to Discover Persistent Threats from Scheduled Windows Tasks_ #. Practicing Backward And Forward Tracking Hunts on A Windows Host_ #. Building Your Own Kestrel Analytics and Sharing With the Community_ #. Setting Up The Open Hunting Stack in Hybrid Cloud With Kestrel and SysFlow_ #. Try Kestrel in a Cloud Sandbox_ #. Fun with securitydatasets.com and the Kestrel PowerShell Deobfuscator_ #. Kestrel Data Retrieval Explained_

Talks And Demos

Talk summary (visit Kestrel documentation on talks_ to learn details):

• 2023/08 Black Hat USA 2023_
• 2022/12 Infosec Jupyterthon 2022_ [IJ'22 live hunt recording_]
• 2022/08 Black Hat USA 2022_ [BH'22 recording_ | BH'22 hunting lab_]
• 2022/06 Cybersecurity Automation Workshop_
• 2022/04 SC eSummit on Threat Hunting & Offense Security_ (free to register/playback)
• 2021/12 Infosec Jupyterthon 2021_ [IJ'21 live hunt recording_]
• 2021/11 BlackHat Europe 2021_
• 2021/10 SANS Threat Hunting Summit 2021: [SANS'21 session recording]
• 2021/05 RSA Conference 2021: [RSA'21 session recording]

Connecting With The Community

• Join Kestrel slack channel:

  • Get a slack invitation_ to join Open Cybersecurity Alliance workspace_

    .. image:: https://opencyberallia.wpengine.com/wp-content/uploads/2022/03/OCA-logo-e1646689234325.png :width: 20% :alt: OCA logo

  • Join the kestrel channel to ask questions and connect with other hunters

• Contribute to the language development (Apache License 2.0_):

  • Create a GitHub Issue_ to report bugs and suggest new features
  • Follow the contributing guideline_ to submit your pull request
  • Refer to the governance documentation_ regarding PR merge, release, and vulnerability disclosure

• Share your huntbook and analytics:

  • Kestrel huntbook_
  • Kestrel analytics_

.. _Kestrel live tutorial in a cloud sandbox: https://mybinder.org/v2/gh/opencybersecurityalliance/kestrel-huntbook/HEAD?filepath=tutorial .. _Kestrel documentation: https://kestrel.readthedocs.io/

.. _A comprehensive introduction to Kestrel: https://kestrel.readthedocs.io/en/stable/overview/ .. _The two key concepts of Kestrel: https://kestrel.readthedocs.io/en/stable/language/tac.html#key-concepts .. _Interactive tutorial with quiz: https://mybinder.org/v2/gh/opencybersecurityalliance/kestrel-huntbook/HEAD?filepath=tutorial .. _Kestrel runtime installation: https://kestrel.readthedocs.io/en/stable/installation/runtime.html .. _How to connect to your data sources: https://kestrel.readthedocs.io/en/stable/installation/datasource.html .. _How to execute an analytic hunt step in Python/Docker: https://kestrel.readthedocs.io/en/stable/installation/analytics.html .. _Language reference book: https://kestrel.readthedocs.io/en/stable/language/commands.html .. _How to use Kestrel via API: https://kestrel.readthedocs.io/en/stable/source/kestrel.session.html .. _How to launch Kestrel as a Docker container: https://kestrel.readthedocs.io/en/stable/deployment/ .. _Kestrel documentation on talks: https://kestrel.readthedocs.io/en/stable/talks.html

.. _Kestrel huntbook: https://github.com/opencybersecurityalliance/kestrel-huntbook .. _Kestrel analytics: https://github.com/opencybersecurityalliance/kestrel-analytics

.. _Building a Huntbook to Discover Persistent Threats from Scheduled Windows Tasks: https://opencybersecurityalliance.org/huntbook-persistent-threat-discovery-kestrel/ .. _Practicing Backward And Forward Tracking Hunts on A Windows Host: https://opencybersecurityalliance.org/backward-and-forward-tracking-hunts-on-a-windows-host/ .. _Building Your Own Kestrel Analytics and Sharing With the Community: https://opencybersecurityalliance.org/kestrel-custom-analytics/ .. _Setting Up The Open Hunting Stack in Hybrid Cloud With Kestrel and SysFlow: https://opencybersecurityalliance.org/kestrel-sysflow-open-hunting-stack/ .. _Try Kestrel in a Cloud Sandbox: https://opencybersecurityalliance.org/try-kestrel-in-a-cloud-sandbox/ .. _Fun with securitydatasets.com and the Kestrel PowerShell Deobfuscator: https://opencybersecurityalliance.org/fun-with-securitydatasets-com-and-the-kestrel-powershell-deobfuscator/ .. _Kestrel Data Retrieval Explained: https://opencybersecurityalliance.org/kestrel-data-retrieval-explained/

.. _RSA Conference 2021: https://www.rsaconference.com/Library/presentation/USA/2021/The%20Game%20of%20Cyber%20Threat%20Hunting%20The%20Return%20of%20the%20Fun .. _RSA'21 session recording: https://www.youtube.com/watch?v=-Xb086R0JTk .. _SANS Threat Hunting Summit 2021: https://www.sans.org/blog/a-visual-summary-of-sans-threat-hunting-summit-2021/ .. _SANS'21 session recording: https://www.youtube.com/watch?v=gyY5DAWLwT0 .. _BlackHat Europe 2021: https://www.blackhat.com/eu-21/arsenal/schedule/index.html#an-open-stack-for-threat-hunting-in-hybrid-cloud-with-connected-observability-25112 .. _Infosec Jupyterthon 2021: https://infosecjupyterthon.com/2021/agenda.html .. _IJ'21 live hunt recording: https://www.youtube.com/embed/nMnHBnYfIaI?start=20557&end=22695 .. _Infosec Jupyterthon 2022: https://infosecjupyterthon.com/2022/agenda.html .. _IJ'22 live hunt recording: https://www.youtube.com/embed/8Mw1yyYkeqM?start=23586&end=26545 .. _SC eSummit on Threat Hunting & Offense Security: https://www.scmagazine.com/esummit/automating-the-hunt-for-advanced-threats .. _Cybersecurity Automation Workshop: http://www.cybersecurityautomationworkshop.org/ .. _Black Hat USA 2023: https://www.blackhat.com/us-23/arsenal/schedule/index.html#identity-threat-hunting-with-kestrel-33662 .. _Black Hat USA 2022: https://www.blackhat.com/us-22/arsenal/schedule/index.html#streamlining-and-automating-threat-hunting-with-kestrel-28014 .. _BH'22 recording: https://www.youtube.com/watch?v=tf1VLIpFefs .. _Black Hat USA 2022 session recording: https://www.youtube.com/watch?v=tf1VLIpFefs .. _BH'22 hunting lab: https://mybinder.org/v2/gh/opencybersecurityalliance/black-hat-us-2022/HEAD?filepath=demo .. _Black Hat USA 2022 Kestrel hunting lab: https://mybinder.org/v2/gh/opencybersecurityalliance/black-hat-us-2022/HEAD?filepath=demo

.. _slack invitation: https://join.slack.com/t/open-cybersecurity/shared_invite/zt-19pliofsm-L7eSSB8yzABM2Pls1nS12w .. _Open Cybersecurity Alliance workspace: https://open-cybersecurity.slack.com/ .. _GitHub Issue: https://github.com/opencybersecurityalliance/kestrel-lang/issues .. _contributing guideline: CONTRIBUTING.rst .. _governance documentation: GOVERNANCE.rst .. _Apache License 2.0: LICENSE.md