kafkajs

What is kafkajs?

KafkaJS is a modern Apache Kafka client for Node.js. It is designed to be simple, reliable, and performant, making it easy to interact with Kafka brokers and manage Kafka topics, producers, and consumers.

What are kafkajs's main functionalities?

Producer

This code sample demonstrates how to create a Kafka producer using KafkaJS. The producer connects to the Kafka broker, sends a message to a specified topic, and then disconnects.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
});

const producer = kafka.producer();

const run = async () => {
  await producer.connect();
  await producer.send({
    topic: 'test-topic',
    messages: [
      { value: 'Hello KafkaJS user!' }
    ]
  });
  await producer.disconnect();
};

run().catch(console.error);

Consumer

This code sample demonstrates how to create a Kafka consumer using KafkaJS. The consumer connects to the Kafka broker, subscribes to a specified topic, and logs each message received.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
});

const consumer = kafka.consumer({ groupId: 'test-group' });

const run = async () => {
  await consumer.connect();
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value.toString(),
      });
    },
  });
};

run().catch(console.error);

Admin

This code sample demonstrates how to use the admin client in KafkaJS to create a new topic. The admin client connects to the Kafka broker, creates a topic with specified configurations, and then disconnects.

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
});

const admin = kafka.admin();

const run = async () => {
  await admin.connect();
  await admin.createTopics({
    topics: [
      { topic: 'test-topic', numPartitions: 1 }
    ]
  });
  await admin.disconnect();
};

run().catch(console.error);

Other packages similar to kafkajs

node-rdkafka

node-rdkafka is a high-performance Node.js client for Apache Kafka based on the C/C++ library librdkafka. It offers more advanced features and better performance compared to KafkaJS but can be more complex to set up and use.

kafka-node

kafka-node is another popular Kafka client for Node.js. It is simpler and easier to use compared to node-rdkafka but may not offer the same level of performance and advanced features as node-rdkafka or KafkaJS.

no-kafka

no-kafka is a pure JavaScript client for Apache Kafka. It is designed to be simple and easy to use, similar to KafkaJS, but it may not be as actively maintained or feature-rich as KafkaJS.

README

A modern Apache Kafka client for node.js. This library is compatible with Kafka 0.10+.
Native support for Kafka 0.11 features.

KafkaJS is battle-tested and ready for production.

Features

• Producer
• Consumer groups with pause, resume, and seek
• Transactional support for producers and consumers
• Message headers
• GZIP compression
• Snappy and LZ4 compression through plugins
• Plain, SSL and SASL_SSL implementations
• Support for SCRAM-SHA-256 and SCRAM-SHA-512
• Support for AWS IAM authentication
• Admin client

Read something on the website that didn't work with the latest stable version?
Check the pre-release versions - the website is updated on every merge to master.

Getting Started

npm install kafkajs
# yarn add kafkajs

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
})

const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group' })

const run = async () => {
  // Producing
  await producer.connect()
  await producer.send({
    topic: 'test-topic',
    messages: [
      { value: 'Hello KafkaJS user!' },
    ],
  })

  // Consuming
  await consumer.connect()
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value.toString(),
      })
    },
  })
}

run().catch(console.error)

Documentation

Learn more about using KafkaJS on the official site!

• Getting Started
• A Brief Intro to Kafka
• Configuring KafkaJS
• Example Producer
• Example Consumer

Contributing

KafkaJS is an open-source project where development takes place in the open on GitHub. Although the project is maintained by a small group of dedicated volunteers, we are grateful to the community for bugfixes, feature development and other contributions.

See Developing KafkaJS for information on how to run and develop KafkaJS.

Help wanted 🤝

We welcome contributions to KafkaJS, but we also want to see a thriving third-party ecosystem. If you would like to create an open-source project that builds on top of KafkaJS, please get in touch and we'd be happy to provide feedback and support.

Here are some projects that we would like to build, but haven't yet been able to prioritize:

• Dead Letter Queue - Automatically reprocess messages
• Schema Registry - Seamless integration with the schema registry to encode and decode AVRO Now available! thanks to @erikengervall
• Metrics - Integrate with the instrumentation events to expose commonly used metrics

Acknowledgements

Thanks to Sebastian Norde for the V1 logo ❤️

Thanks to Tracy (Tan Yun) for the V2 logo ❤️

Sponsored by:

License

See LICENSE for more details.

Changelog

[1.12.0] - 2020-01-30

Added

• Force refresh of metadata when topic is not present #556
• Expose ConsumerRunConfig type #615
• Randomize order of seed brokers #632

Changed

• Support TLS SNI by default #512
• Changed typing of logLevel argument of logCreator #538
• Add type boolean in ssl KafkaConfig #557
• Allow logging Fetch response payload buffers #573
• Remove default null for logCreator #595
• Add error names to ensure error names work with webpack + uglify #602
• Merge TopicMessages by topic in producer sendBatch #626

Fixed

• Skip control records without auto-resolve #511
• Handle empty member assignment #567
• Only fetch for partitions with initialized offsets #582
• Get correct next offset for compacted topics #577
• TS type definition for removing instrumentation event listeners #608
• Fixed IHeaders definition to accept plain strings #547
• Make TS type ProducerBatch fields optional #610
• Fix typings for logger getters #620