jspcap

jspcap

  The jspcap project is an open source Python program focus on PCAP parsing and analysis, which works as a stream PCAP file extractor. With support of jsformat, it shall support multiple output report formats.

Note that the whole project only supports Python 3.6 or later.

• About
  • Module Structure
    • Foundation
    • Interface
    • Reassembly
    • IPSuite
    • Protocols
    • Utilities
    • CoreKit
    • DumpKit
  • Engine Comparison
• Installation
• Usage
  • Documentation
    • Interfaces
    • Macros
      • Formats
      • Layers
      • Engines
    • Protocols
  • CLI Usage
• Samples
  • Usage Samples
  • CLI Samples
• TODO

About

  jspcap is an independent open source library, using only jsformat as its formatted output dumper.

There is a project called jspcapy works on jspcap, which is a command line tool for PCAP extraction.

  Unlike popular PCAP file extractors, such as Scapy, dpkt, pyshark, and etc, jspcap uses streaming strategy to read input files. That is to read frame by frame, decrease occupation on memory, as well as enhance efficiency in some way.

Module Structure

  In jspcap, all files can be described as following six parts.

• Foundation (jspcap.foundation) -- synthesise file I/O and protocol analysis, coordinate information exchange in all network layers
• Interface (jspcap.interface) -- user interface for the jspcap library, which standardise and simplify the usage of this library
• Reassembly (jspcap.reassembly) -- base on algorithms described in RFC 815, implement datagram reassembly of IP and TCP packets
• IPSuite (jspcap.ipsuite) -- collection of constructors for Internet Protocol Suite
• Protocols (jspcap.protocols) -- collection of all protocol family, with detail implementation and methods
• Utilities (jspcap.utilities) -- collection of four utility functions and classes
• CoreKit (jspcap.corekit) -- core utilities for jspcap implementation
• DumpKit (jspcap.dumpkit) -- dump utilities for jspcap implementation

Engine Comparison

  Besides, due to complexity of jspcap, its extraction procedure takes around 0.01 seconds per packet, which is not ideal enough. Thus, jspcap introduced alternative extraction engines to accelerate this procedure. By now, jspcap supports Scapy, DPKT, and PyShark. Plus, jspcap supports two strategies of multiprocessing (server & pipeline). For more information, please refer to the document.

Engine	Performance (seconds per packet)
default	0.014525251388549805
server	0.12124489148457845
pipeline	0.014450424114863079
scapy	0.002443440357844035
dpkt	0.0003609057267506917
pyshark	0.0792640733718872

 

Installation

Note that jspcap only supports Python versions since 3.6

  Simply run the following to install the latest from PyPI:

pip install jspcap

  Or install from the git repository:

$ git clone https://github.com/JarryShaw/jspcap.git
$ python setup.py install

 

Usage

Documentation

Interfaces

NAME	DESCRIPTION
extract	extract a PCAP file
analyse	analyse application layer packets
reassemble	reassemble fragmented datagrams
trace	trace TCP packet flows

Macros

Formats

NAME	DESCRIPTION
JSON	JavaScript Object Notation (JSON) format
PLIST	macOS Property List (PLIST) format
TREE	Tree-View text format
PCAP	PCAP format

Layers

NAME	DESCRIPTION
RAW	no specific layer
LINK	data-link layer
INET	internet layer
TRANS	transport layer
APP	application layer

Engines

NAME	DESCRIPTION
jsPCAP	the default engine
MPServer	the multiprocessing engine with server process strategy
MPPipeline	the multiprocessing engine with pipeline strategy
DPKT	the DPKT engine
Scapy	the Scapy engine
PyShark	the PyShark engine

Protocols

NAME	DESCRIPTION
Raw	Raw Packet Data
ARP	Address Resolution Protocol
Ethernet	Ethernet Protocol
L2TP	Layer Two Tunnelling Protocol
OSPF	Open Shortest Path First
RARP	Reverse Address Resolution Protocol
VLAN	802.1Q Customer VLAN Tag Type
AH	Authentication Header
HIP	Host Identity Protocol
HOPOPT	IPv6 Hop-by-Hop Options
IP	Internet Protocol
IPsec	Internet Protocol Security
IPv4	Internet Protocol version 4
IPv6	Internet Protocol version 6
IPv6_Frag	Fragment Header for IPv6
IPv6_Opts	Destination Options for IPv6
IPv6_Route	Routing Header for IPv6
IPX	Internetwork Packet Exchange
MH	Mobility Header
TCP	Transmission Control Protocol
UDP	User Datagram Protocol
HTTP	Hypertext Transfer Protocol

  Documentation can be found in submodules of jspcap. Or, you may find usage sample in the test folder. For further information, please refer to the source code -- the docstrings should help you :)

ps: help function in Python should always help you out.

CLI Usage

The following part was originally described in jspcapy, which is now deprecated and merged into this repository.

  As it shows in the help manual, it is quite easy to use:

$ jspcapy --help
usage: jspcapy [-h] [-V] [-o file-name] [-f format] [-j] [-p] [-t] [-a] [-v]
               [-F] [-E PKG] [-P PROTOCOL] [-L LAYER]
               input-file-name

PCAP file extractor and formatted exporter

positional arguments:
  input-file-name       The name of input pcap file. If ".pcap" omits, it will
                        be automatically appended.

optional arguments:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit
  -o file-name, --output file-name
                        The name of input pcap file. If format extension
                        omits, it will be automatically appended.
  -f format, --format format
                        Print a extraction report in the specified output
                        format. Available are all formats supported by
                        jsformat, e.g.: json, plist, and tree.
  -j, --json            Display extraction report as json. This will yield
                        "raw" output that may be used by external tools. This
                        option overrides all other options.
  -p, --plist           Display extraction report as macOS Property List
                        (plist). This will yield "raw" output that may be used
                        by external tools. This option overrides all other
                        options.
  -t, --tree            Display extraction report as tree view text. This will
                        yield "raw" output that may be used by external tools.
                        This option overrides all other options.
  -a, --auto-extension  If output file extension omits, append automatically.
  -v, --verbose         Show more information.
  -F, --files           Split each frame into different files.
  -E PKG, --engine PKG  Indicate extraction engine. Note that except default
                        or jspcap engine, all other engines need support of
                        corresponding packages.
  -P PROTOCOL, --protocol PROTOCOL
                        Indicate extraction stops after which protocol.
  -L LAYER, --layer LAYER
                        Indicate extract frames until which layer.

  Under most circumstances, you should indicate the name of input PCAP file (extension may omit) and at least, output format (json, plist, or tree). Once format unspecified, the name of output file must have proper extension (*.json, *.plist, or *.txt), otherwise FormatError will raise.

  As for verbose mode, detailed information will print while extraction (as following examples). And auto-extension flag works for the output file, to indicate whether extensions should be appended.

 

Samples

Usage Samples

  As described in test folder, jspcap is quite easy to use, with simply three verbs as its main interface. Several scenarios are shown as below.

• extract a PCAP file and dump the result to a specific file (with no reassembly)

  import jspcap
  # dump to a PLIST file with no frame storage (property frame disabled)
  plist = jspcap.extract(fin='in.pcap', fout='out.plist', format='plist', store=False)
  # dump to a JSON file with no extension auto-complete
  json = jspcap.extract(fin='in.cap', fout='out.json', format='json', extension=False)
  # dump to a folder with each tree-view text file per frame
  tree = jspcap.extract(fin='in.pcap', fout='out', format='tree', files=True)
  

• extract a PCAP file and fetch IP packet (both IPv4 and IPv6) from a frame (with no output file)

  >>> import jspcap
  >>> extraction = jspcap.extract(fin='in.pcap', nofile=True)
  >>> frame0 = extraction.frame[0]
  # check if IP in this frame, otherwise ProtocolNotFound will be raised
  >>> flag = jspcap.IP in frame0
  >>> tcp = frame0[jspcap.IP] if flag else None
  

• extract a PCAP file and reassemble TCP payload (with no output file nor frame storage)

  import jspcap
  # set strict to make sure full reassembly
  extraction = jspcap.extract(fin='in.pcap', store=False, nofile=True, tcp=True, strict=True)
  # print extracted packet if HTTP in reassembled payloads
  for packet in extraction.reassembly.tcp:
  for reassembly in packet.packets:
      if jspcap.HTTP in reassembly.protochain:
          print(reassembly.info)
  

CLI Samples

  The CLI (command line interface) of jspcap has two different access.

• through console scripts -- use command name jspcapy directly (as shown in samples)
• through Python module -- python -m jspcap [...] works exactly the same as above

Here are some usage samples:

• export to a macOS Property List (Xcode has special support for this format)

$ jspcapy in --format plist --verbose
🚨Loading file 'in.pcap'
 - Frame   1: Ethernet:IPv6:ICMPv6
 - Frame   2: Ethernet:IPv6:ICMPv6
 - Frame   3: Ethernet:IPv4:TCP
 - Frame   4: Ethernet:IPv4:TCP
 - Frame   5: Ethernet:IPv4:TCP
 - Frame   6: Ethernet:IPv4:UDP
🍺Report file stored in 'out.plist'

• export to a JSON file (with no format specified)

$ jspcapy in --output out.json --verbose
🚨Loading file 'in.pcap'
 - Frame   1: Ethernet:IPv6:ICMPv6
 - Frame   2: Ethernet:IPv6:ICMPv6
 - Frame   3: Ethernet:IPv4:TCP
 - Frame   4: Ethernet:IPv4:TCP
 - Frame   5: Ethernet:IPv4:TCP
 - Frame   6: Ethernet:IPv4:UDP
🍺Report file stored in 'out.json'

• export to a text tree view file (without extension autocorrect)

$ jspcapy in --output out --format tree --verbose
🚨Loading file 'in.pcap'
 - Frame   1: Ethernet:IPv6:ICMPv6
 - Frame   2: Ethernet:IPv6:ICMPv6
 - Frame   3: Ethernet:IPv4:TCP
 - Frame   4: Ethernet:IPv4:TCP
 - Frame   5: Ethernet:IPv4:TCP
 - Frame   6: Ethernet:IPv4:UDP
🍺Report file stored in 'out'

 

TODO

• specify Raw packet
• interface verbs
• review docstrings
• merge jspcapy
• write documentation
• implement IP and MAC address containers
• implement option list extractors
• implement more protocols