@meeshkanml/http-types-kafka

http-types-kafka

Tools for writing ts-http-types to Kafka in Node.js, powered by kafka.js.

Quick start

First create the topic you're writing to:

$ kafka-topics.sh --bootstrap-server localhost:9092 --topic express_recordings --create --partitions 3 --replication-factor 1

Note that you may need to change script name depending on how you installed Kafka.

Create a HttpTypesKafkaProducer and connect to Kafka:

// Create a `KafkaConfig` instance (from kafka.js)
const brokers = ["localhost:9092"];
const kafkaConfig: KafkaConfig = {
  clientId: "client-id",
  brokers,
};

// Specify the topic
const kafkaTopic = "express_recordings";

// Create the producer
const producer = HttpTypesKafkaProducer.create({ kafkaConfig, topic: kafkaTopic });

// Connect to Kafka
await producer.connect();

Send a single HttpExchange to Kafka:

const exchange: HttpExchange = ...;
await producer.send(exchange);

Send multiple HttpExchanges:

const exchanges: HttpExchange[] = ...;
await producer.sendMany(exchanges);

Send recordings from a JSON lines file, where every line is a JSON-encoded HttpExchange:

await producer.sendFromFile("recordings.jsonl");

Finally, disconnect:

await producer.disconnect();

Delete the topic if you're done:

$ kafka-topics.sh --bootstrap-server localhost:9092 --topic express_recordings --delete

Command-line interface

See available commands:

$ http-types-kafka

Producer

First create the destination topic in Kafka.

To send recordings from recordings.jsonl to Kafka, run:

$ http-types-kafka producer --file=recordings.jsonl --topic=my_recordings

Development

Install dependencies:

$ yarn

Build a package in lib:

$ yarn compile

Run tests:

$ ./docker-start.sh  # Start Kafka and zookeeper
$ yarn test
$ ./docker-stop.sh  # Once you're done

Package for npm:

$ npm pack

Publish to npm:

$ npm publish --access public --dry-run

Remove --dry-run once you're sure you want to publish.

Working with local Kafka

First start kafka and zookeeper:

# See `docker-compose.yml`
docker-compose up

Create a topic called http_types_kafka_test:

docker exec kafka1 kafka-topics --bootstrap-server kafka1:9092 --topic http_types_kafka_test --create --partitions 3 --replication-factor 1

Check the topic exists:

docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --list

Describe the topic:

docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --describe --topic http_types_kafka_test

Using kafkacat

List topics:

kafkacat -b localhost:9092 -L

Push data to topic from file with snappy compression:

tail -f tests/resources/recordings.jsonl | kafkacat -b localhost:9092 -t http_types_kafka_test -z snappy

Consume messages from topic to console:

kafkacat -b localhost:9092 -t http_types_kafka_test -C