Socket
Socket
Sign inDemoInstall

kafka-streams

Package Overview
Dependencies
120
Maintainers
1
Versions
43
Alerts
File Explorer

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

    kafka-streams

kafka-streams for Node.js


Version published
Maintainers
1
Created

Changelog

Source

2019-05-07, Version 4.9.0

  • upgraded dependencies
  • fixed bug where kstream merge was not passing kafkaStreams reference
  • fixed code formatting
  • fixed word-count example
  • fixed produceToTopic example
  • fixed consumeOneProduceTwo example
  • adjusted consumeFromTopic example
  • ported most-subject back to 5.3.0 to fix major window bug
  • fixed window example

Readme

Source

node-kafka-streams

Greenkeeper badge Build Status npm version

npm install --save kafka-streams
const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(/* .. */);

kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");

CHANGES: The latest version brings a lot of changes, please check here before updating.

API Overview

You might also like

README Overview

Prerequisites

  • Kafka broker should be version >= 0.11.x
  • Node.js should be version >= 8.x.x

Aim of this Library

  • this is not a 1:1 port of the official JAVA kafka-streams
  • the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
  • stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

Description

kafka-streams :octopus: equivalent for nodejs :sparkles::turtle::rocket::sparkles: build on super fast :fire: observables using most.js :metal:

ships with sinek :pray: for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API :goberserk:

the lib is based on sinek, which is based on kafka-node's ConsumerGroups

Port Progress Overview

  • core structure
  • KStream base - stream as a changelog
  • KTable base - stream as a database
  • KStream & KTable cloning
  • complex stream join structure
  • advanced joins see
  • windows (for joins) see
  • flink like window operations
  • word-count example
  • more examples
  • local-storage for etl actions
  • local-storage factory (one per action)
  • KStorage example for any DB that supports atomic actions
  • backing-up local-storage via kafka
  • kafka client implementation
  • KTable replay to Kafka (produce)
  • stream for topic message production only
  • sinek implementation
  • backpressure mode for KafkaClient
  • auto-json payloads (read-map/write-map)
  • auto producer partition and keyed-message handling
  • documentation
  • API description
  • higher join & combine examples
  • embed native client librdkafka for more performance
  • SSL
  • SASL
  • Kerberos

Operator Implementations

  • map
  • asyncMap
  • constant
  • scan
  • timestamp
  • tap
  • filter
  • skipRepeats
  • skipRepeatsWith
  • slice
  • take
  • skip
  • takeWhile
  • skipWhile
  • until
  • since
  • reduce
  • chainReduce
  • forEach (observe)
  • chainForEach
  • drain
  • _zip
  • _merge
  • _join
  • _combine
  • _sample
  • throttle
  • debounce
  • delay
  • multicast
  • A description of the operators can be found here
  • Missing an operator? Feel free to open an issue :cop:

Additional Operators

  • mapStringToArray
  • mapArrayToKV
  • mapStringToKV
  • mapParse
  • mapStringify
  • atThroughput
  • mapWrapKafkaPayload
  • mapToFormat
  • mapFromFormat
  • Want more? Feel free to open an issue :cop:

Stream Action Implementations

  • countByKey
  • sumByKey
  • min
  • max
  • Want more? Feel free to open an issue :cop:

Join Operations

Operation description

KStream Status

  • merge
  • outerJoin
  • innerJoin
  • leftJoin
  • branch

KTable Status

  • merge
  • outerJoin
  • innerJoin
  • leftJoin

KTable <-> KStream Status

  • merge
  • outerJoin
  • innerJoin
  • leftJoin

Window Operations

KStream

  • window
  • advanced window
  • rolling window

More

Can I use this library yet?

Yes.

Are we ready for production yet?

Probably, yes. :smile:

Even More

Forks or Stars give motivation :bowtie:

Keywords

FAQs

Last updated on 07 May 2019

Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc