python-haystack memory forensics ################################
|travis| |coverage| |landscape| |pypi| |docs|
Quick usage guide <docs/Haystack_basic_usage.ipynb>_ in the docs/ folder.
Haystack-reverse CLI <docs/Haystack_reverse_CLI.ipynb>_ in the docs/ folder.
More documentation <http://python-haystack.readthedocs.io/en/latest/>_
python-haystack is an heap analysis framework, focused on searching and reversing of C structure in allocated memory.
The first function/API is the SEARCH function. It gives the ability to search for known record types in a process memory dump or live process's memory.
The second function/API is the REVERSE function in the extension python-haystack-reverse <https://github.com/trolldbois/python-haystack-reverse>_
It aims at helping an analyst in reverse engineering the memory records types present in a process heap.
It focuses on reconstruction, classification of classic C structures from memory.
It attempts to recreate types definition.
A few entry points exists to handle the format your memory dump.
haystack-live-dumphaystack-find-heap allows to show details on Windows HEAP.haystack-search search CLIhaystack-show show CLI for specific record type at a specific addressYou can use the following URL to designate your memory handler/dump:
dir:///path/to/my/haystack/fump/folder to use the haystack dump formatdmp:///path/to/my/minidump/file use the minidump format (microsoft?)frida://name_or_pid_of_process_to_attach_to use frida to access a live process memorylive://name_or_pid_of_process_to_attach_to ptrace a live processrekall:// load a rekall imagevolatility:// load a volatility imagecuckoo:// load a memory dump produced by Cuckoo (beta might need patch)On Windows, the most straightforward is to get a Minidump. The Microsoft Sysinternals
suite of tools provide either a CLI (procdump.exe) or a GUI (Process explorer).
Using one of these (with full memory dump option) you will produce a file
that can be used with the haystack-minidump-xxx list of entry points.
While technically you could use many third party tool, haystack actually
need memory mapping information to work with.
So there is a dumping tool included haystack-live-dump:
.. code-block:: bash
# haystack-live-dump <pid> myproc.dump
haystack-live-dump capture a process memory dump to a folder (haystack format)haystack-rekall-dump dump a specific process to a haystack process dumphaystack-volatility-dump dump a specific process to a haystack process dumpYou can easily reproduce the format of the dump, its a folder/archive containing each memory map in a separate file :
Or you can code a haystack.abc.IMemoryMapping implementation for your favorite format.
Otherwise, if you already have a system memory dump from Volatility or Rekall,
you can use the haystack-rekall-xxx or haystack-volatility-xxx families of
entry points to extract a specific process memory into a file.
The entry point haystack-find-heap allows to show details on Windows HEAP.
It should support:
and show details of the Look Aside List (LAL) and Low Fragmentation Heap (LFH) frontend.
You might be surprised to see that sometimes, a single process can mix the two types of HEAP (32 & 64).
To search for a specific record, you will first need to define that record type. A [quick usage guide](docs/Haystack basic usage.ipynb) is available to go over the basic steps to go from a C Header file to a Python ctypes definition. Or you can do it yourself, with traditional Python ctypes records.
The search api is available through the haystack-xxx-search family of scripts but
also in an API so that you can embed that search in your own code.
In short, the haystack search will iterate over every offset of the program's memory to try and find 'valid' offset for that specific record type.
The validity of the record is determined by type constraints such as:
.. code-block:: bash
$ python haystack/cli.py dir:///home/user/project/python-haystack/test/src/test-ctypes6.32.dump ctypes6_gen32.struct_usual
The following constraints are supported:
Example:
.. code-block:: python
[struct_name]
myfield: [1,0xff]
ptr_field: NotNull
You can take a look a haystack/allocators/win32/winxpheap32.constraints, where
the constraints of a Windows XP HEAP x32 are defined.
Obviously, the more constraints, the better the results will be.
You can also create more complex constraints using python code by implementing
a haystack.abc.interface.IRecordTypeDynamicConstraintsValidator class and feeding it to
the ModuleConstraints.set_dynamic_constraints
sslsnoop repository needs an update to be compatible with releases > v0.30 - pending
For example, this will dump the session_state structures + pointed children structures as an python object that we can play with. Lets assume we have an ssh client or server as pid 4042:
.. code-block:: bash
$ sudo haystack-live-search --pickled 4042 sslsnoop.ctypes_openssh.session_state search > instance.pickled
$ sudo haystack-live-search --pickled 4042 sslsnoop.ctypes_openssh.session_state refresh 0xb8b70d18 > instance.pickled
$ sudo haystack-live-search --pickled <pid> <your ctypes Structure> search
This is not working right now
There is also an attempt at a Graphical UI python-haystack-gui <https://github.com/trolldbois/python-haystack-gui>_
See the quick usage guide <docs/Haystack_basic_usage.ipynb>_
The most easy way is to use ctypeslib to generate ctypes records from C Headers.
Or define your python ctypes record by hand.
@ see sslsnoop in the Pypi repo. openssl and nss structures are generated.
@ see ctypes-kernel on my github. Linux kernel structure are generated from a build kernel tree. (VMM is abitch)
The basic functionality is to search in a process' memory for a specific C Record.
The extended reverse engineering functionality aims at reversing structures from memory/heap analysis.
You add some constraints on the record fields expected values. Pointers are always constrained to valid memory space.
passe-partout <http://www.hsc.fr/ressources/breves/passe-partout.html.fr>_ originally.
since I started in March 2011, I have uncovered several other related
previous work.
Most of them are in the docs/ folder.
Other related work are mona.py from Immunity, some other Mandiant stuff...
In a nutshell, this is probably not an original idea. But yet, I could not find a operational standalone lib for live memory extraction for my sslsnoop PoC, so....
Related work <https://github.com/trolldbois/python-haystack/wiki/State-of-art-reference>_
http://ntinfo.biz/ xntsv32
.. |pypi| image:: https://img.shields.io/pypi/v/haystack.svg?style=flat-square&label=latest%20stable%20version :target: https://pypi.python.org/pypi/haystack :alt: Latest version released on PyPi
.. |coverage| image:: https://img.shields.io/coveralls/trolldbois/python-haystack/master.svg?style=flat-square&label=coverage :target: https://coveralls.io/github/trolldbois/python-haystack?branch=master :alt: Test coverage
.. |travis| image:: https://img.shields.io/travis/trolldbois/python-haystack/master.svg?style=flat-square&label=travis-ci :target: http://travis-ci.org/trolldbois/python-haystack :alt: Build status of the master branch on Mac/Linux
.. |landscape| image:: https://landscape.io/github/trolldbois/python-haystack/master/landscape.svg?style=flat :target: https://landscape.io/github/trolldbois/python-haystack/master :alt: Code Health
.. |docs| image:: https://readthedocs.org/projects/python-haystack/badge/?version=latest :target: https://readthedocs.org/projects/python-haystack/badge/?version=latest :alt: Documentation status
FAQs
Search C Structures in a process' memory
We found that haystack demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
PyPI confirms no security flaws were exploited in the Ultralytics supply chain attack and highlights improvements for safer package publishing.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.