sqs-consumer

What is sqs-consumer?

The sqs-consumer npm package is a simple to use library for building Amazon Simple Queue Service (SQS) based applications. It allows you to create consumers that process messages from SQS queues with ease, handling message polling, error handling, and visibility timeout extensions.

What are sqs-consumer's main functionalities?

Creating a Consumer

This code sample demonstrates how to create a basic consumer that reads messages from an SQS queue and logs the message body to the console.

const { Consumer } = require('sqs-consumer');
const AWS = require('aws-sdk');

const app = Consumer.create({
  queueUrl: 'https://sqs.us-east-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    console.log(message.Body);
  },
  sqs: new AWS.SQS()
});

app.on('error', (err) => {
  console.error(err.message);
});

app.start();

Error Handling

This code sample shows how to handle errors that occur during message processing. The 'error' event is emitted for general errors, while the 'processing_error' event is specifically for errors that occur during message handling.

const { Consumer } = require('sqs-consumer');
const AWS = require('aws-sdk');

const app = Consumer.create({
  queueUrl: 'https://sqs.us-east-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    throw new Error('Processing error');
  },
  sqs: new AWS.SQS()
});

app.on('error', (err) => {
  console.error('Error occurred:', err.message);
});

app.on('processing_error', (err) => {
  console.error('Processing error:', err.message);
});

app.start();

Batch Processing

This code sample demonstrates how to process messages in batches. The 'handleMessageBatch' function receives an array of messages, allowing for more efficient processing.

const { Consumer } = require('sqs-consumer');
const AWS = require('aws-sdk');

const app = Consumer.create({
  queueUrl: 'https://sqs.us-east-1.amazonaws.com/account-id/queue-name',
  handleMessageBatch: async (messages) => {
    messages.forEach(message => console.log(message.Body));
  },
  sqs: new AWS.SQS()
});

app.on('error', (err) => {
  console.error(err.message);
});

app.start();

Other packages similar to sqs-consumer

aws-sdk

The aws-sdk package is the official AWS SDK for JavaScript, which includes support for SQS. It provides a more comprehensive set of tools for interacting with AWS services, including SQS. However, it requires more boilerplate code to achieve the same functionality as sqs-consumer.

sqs-producer

The sqs-producer package is designed for sending messages to SQS queues, complementing the sqs-consumer package. While sqs-consumer focuses on consuming messages, sqs-producer is used for producing messages, making them a good pair for full SQS queue management.

node-sqs

The node-sqs package is another library for interacting with SQS. It provides both producing and consuming capabilities but is less specialized and streamlined compared to sqs-consumer, which is focused solely on consuming messages.

README

sqs-consumer

Build SQS-based applications without the boilerplate. Just define an async function that handles the SQS message processing.

Installation

npm install sqs-consumer --save-dev

Note This library assumes you are using AWS SDK v3. If you are using v2, please install v5.8.0:

npm install sqs-consumer@5.8.0 --save-dev

Usage

import { Consumer } from 'sqs-consumer';

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    // do some work with `message`
  }
});

app.on('error', (err) => {
  console.error(err.message);
});

app.on('processing_error', (err) => {
  console.error(err.message);
});

app.start();

• The queue is polled continuously for messages using long polling.
• Messages are deleted from the queue once the handler function has completed successfully.
• Throwing an error (or returning a rejected promise) from the handler function will cause the message to be left on the queue. An SQS redrive policy can be used to move messages that cannot be processed to a dead letter queue.
• By default messages are processed one at a time – a new message won't be received until the first one has been processed. To process messages in parallel, use the batchSize option detailed below.

You can also find some examples of sqs-consumer implemented in various ways within the examples directory.

Credentials

By default the consumer will look for AWS credentials in the places specified by the AWS SDK. The simplest option is to export your credentials as environment variables:

export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...

If you need to specify your credentials manually, you can use a pre-configured instance of the SQS Client client.

import { Consumer } from 'sqs-consumer';
import { SQSClient } from '@aws-sdk/client-sqs';

const app = Consumer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  handleMessage: async (message) => {
    // ...
  },
  sqs: new SQSClient({
    region: 'my-region',
    credentials: {
      accessKeyId: 'yourAccessKey',
      secretAccessKey: 'yourSecret'
    }
  })
});

app.on('error', (err) => {
  console.error(err.message);
});

app.on('processing_error', (err) => {
  console.error(err.message);
});

app.on('timeout_error', (err) => {
  console.error(err.message);
});

app.start();

AWS IAM Permissions

Consumer will receive and delete messages from the SQS queue. Ensure sqs:ReceiveMessage, sqs:DeleteMessage, sqs:DeleteMessageBatch, sqs:ChangeMessageVisibility and sqs:ChangeMessageVisibilityBatch access is granted on the queue being consumed.

API

Consumer.create(options)

Creates a new SQS consumer.

Options

Option	Type	Description
queueUrl	String	The SQS queue URL
region	String	The AWS region (default eu-west-1)
handleMessage	Function	An async function (or function that returns a Promise) to be called whenever a message is received. Receives an SQS message object as its first argument.
handleMessageBatch	Function	An async function (or function that returns a Promise) to be called whenever a batch of messages is received. Similar to handleMessage but will receive the list of messages, not each message individually. If both are set, handleMessageBatch overrides handleMessage. In the case that you need to ack only some of the messages, return an array with the successful messages only.
handleMessageTimeout	Number	Time in ms to wait for handleMessage to process a message before timing out. Emits timeout_error on timeout. By default, if handleMessage times out, the unprocessed message returns to the end of the queue.
attributeNames	Array	List of queue attributes to retrieve (i.e. ['All', 'ApproximateFirstReceiveTimestamp', 'ApproximateReceiveCount']).
messageAttributeNames	Array	List of message attributes to retrieve (i.e. ['name', 'address']).
batchSize	Number	The number of messages to request from SQS when polling (default 1). This cannot be higher than the AWS limit of 10.
visibilityTimeout	Number	The duration (in seconds) that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request.
heartbeatInterval	Number	The interval (in seconds) between requests to extend the message visibility timeout. On each heartbeat the visibility is extended by adding visibilityTimeout to the number of seconds since the start of the handler function. This value must less than visibilityTimeout.
terminateVisibilityTimeout	Boolean	If true, sets the message visibility timeout to 0 after a processing_error (defaults to false).
waitTimeSeconds	Number	The duration (in seconds) for which the call will wait for a message to arrive in the queue before returning (defaults to 20).
authenticationErrorTimeout	Number	The duration (in milliseconds) to wait before retrying after an authentication error (defaults to 10000).
pollingWaitTimeMs	Number	The duration (in milliseconds) to wait before repolling the queue (defaults to 0).
sqs	SQSClient	An optional SQS Client object to use if you need to configure the client manually
shouldDeleteMessages	Boolean	Default to true, if you don't want the package to delete messages from sqs set this to false

consumer.start()

Start polling the queue for messages.

consumer.stop()

Stop polling the queue for messages.

consumer.isRunning

Returns the current polling state of the consumer: true if it is actively polling, false if it is not.

Events

Each consumer is an EventEmitter and emits the following events:

Event	Params	Description
error	err, [message]	Fired when an error occurs interacting with the queue. If the error correlates to a message, that message is included in Params
processing_error	err, message	Fired when an error occurs processing the message.
timeout_error	err, message	Fired when handleMessageTimeout is supplied as an option and if handleMessage times out.
message_received	message	Fired when a message is received.
message_processed	message	Fired when a message is successfully processed and removed from the queue.
response_processed	None	Fired after one batch of items (up to batchSize) has been successfully processed.
stopped	None	Fired when the consumer finally stops its work.
empty	None	Fired when the queue is empty (All messages have been consumed).

Contributing

See contributing guidelines.