github.com/m-lab/tcp-info

tcp-info

The tcp-info tool executes a polling loop that tracks the measurement statistics of every open TCP socket on a system. Data is written, in JSONL format (refered to internally as ArchivedRecord), to files compressed using zstd. This tool forms the basis of a lot of measurements on the Kubernetes-based Measurement Lab platform.

We expect most people will run this tool using a docker container. To invoke, with data written to ~/data, and prometheus metrics published on port 7070:

docker run --network=host -v ~/data:/home/ -it measurementlab/tcp-info -prom=7070

Fast tcp-info collector in Go

This repository uses the netlink API to collect inet_diag messages, partially parses them, and caches the intermediate representation. It then detects differences from one scan to the next, and queues connections that have changed for logging. It logs the intermediate representation through external zstd processes to one file per connection.

The previous version uses protobufs, but we have discontinued that largely because of the increased maintenance overhead, and risk of losing unparsed data. Instead, we are now using ArchivedRecord which is partially parsed netlink messages, mostly in base64 encoded blobs, marshaled to JSONL format, with one JSON object per line.

To run the tests or the collection tool, you will also require zstd, which can be installed with:

bash <(curl -fsSL https://raw.githubusercontent.com/horta/zstd.install/master/install)

OR

sudo apt-get update && sudo apt-get install -y zstd

Example sidecar

The tcp-info eventsocket interface allows sidecar services to receive "open" and "close" events on a unix domain socket connection. A simple reference implementation cmd/example-eventsocket-client can be started using docker-compose.

docker-compose up

New TCP events are processed by the example-eventsocket-client sidecar and logged to stderr. You may trigger a TCP connection from within the TCPINFO container using a command like:

docker exec -it tcp-info_tcpinfo_1 wget www.google.com

Parse library and command line tools

CSV tool

The cmd/csvtool directory contains a tool for parsing ArchivedRecord and producing CSV files. Currently reads netlink-jSONL from stdin and writes CSV to stdout.

Code Layout

• inetdiag - code related to include/uapi/linux/inet_diag.h. All structs will be in structs.go
• tcp - Should include ONLY the code related to include/uapi/linux/tcp.h
• parse - code related to parsing the messages in inetdiag and tcp.
• zstd - zstd reader and writer.
• saver - code related to writing ParsedMessages to files.
• cache - code to cache netlink messages and detect changes.
• collector - code related to collecting netlink messages from the kernel.

Dependencies (as of March 2019)

• saver: inetdiag, cache, parse, tcp, zstd
• collector: parse, saver, inetdiag, tcp
• main.go: collector, saver, parse (just for sanity check)
• cache: parse
• parse: inetdiag

And (almost) all package use metrics.

Layers for main.go (each layer depends only on items to right, or lower layers)

1. main.go
2. collector > saver > cache
3. netlink > inetdiag
4. tcp, zstd, metrics

Layers for parse package

1. parse (used by command line tools, etl)
2. netlink > inetdiag
3. tcp, zstd, metrics