duramq

Pigeon

A Ruby gem that implements the outbox pattern for Kafka message publishing to ensure message durability and delivery reliability.

Installation

Add this line to your application's Gemfile:

gem 'pigeon'

And then execute:

$ bundle install

Or install it yourself as:

$ gem install pigeon

Usage

Configuration

Configure the gem with your Karafka and Kafka details using dry-configurable:

Pigeon.configure do |config|
  config.client_id = "my-application"
  config.kafka_brokers = ["kafka1:9092", "kafka2:9092"]
  config.max_retries = 5
  config.retry_delay = 60 # seconds
  config.max_retry_delay = 3600 # 1 hour
  config.encrypt_payload = false
  config.retention_period = 7 # days
  
  # Additional Karafka-specific configuration
  config.karafka_config = {
    delivery: :async,
    kafka: {
      'bootstrap.servers': 'kafka1:9092,kafka2:9092',
      'request.required.acks': 1
    }
  }
end

You can also access configuration values directly:

# Get the current configuration
client_id = Pigeon.config.client_id
max_retries = Pigeon.config.max_retries

Publishing Messages

# Simple publishing
Pigeon.publisher.publish(
  topic: "my-topic",
  payload: { user_id: 123, action: "signup" }
)

# With additional options
Pigeon.publisher.publish(
  topic: "my-topic",
  payload: { user_id: 123, action: "signup" },
  key: "user-123",
  headers: { "source" => "web-app" },
  sync: true, # attempt immediate publishing
  partition: 1
)

Processing Messages

# Process pending messages
stats = Pigeon.processor.process_pending(batch_size: 100)
puts "Processed: #{stats[:processed]}, Succeeded: #{stats[:succeeded]}, Failed: #{stats[:failed]}"

# Process a specific message
success = Pigeon.processor.process_message("message-id")

# Clean up old processed messages
cleaned = Pigeon.processor.cleanup_processed(older_than: 14) # days
puts "Cleaned up #{cleaned} messages"

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

Contributing

Bug reports and pull requests are welcome on GitHub.

License

The gem is available as open source under the terms of the MIT License.