📮 Email queue extending NodeMailer with multi SMTP transports and horizontally scaled apps support
"Mail-Time" is NPM package for mail queue management. Build on top of the nodemailer package. Mail-Time made for single-server and horizontally scaled multi-server setups in mind.
Every MailTime instance can have type configured as Server or Client. Server type of MailTime is great for creating an emailing micro-service app.
The main difference between Server and Client type is that the Server handles the queue and sends email. While the Client only adds emails into the queue.
Redundant solution for email transmission.
Issue - mitigate a single point of failure via persistent queue and re-send attempts
|----------------| |------| |------------------|
| Other mailer | ------> | SMTP | ------> | ^_^ Happy user |
|----------------| |------| |------------------|
The scheme above will work as long as SMTP service is available
or connection between your server and SMPT is up. Once network
failure occurs or SMTP service is down - users won't be happy
|----------------| \ / |------| |------------------|
| Other mailer | --X---> | SMTP | ------> | 0_o Disappointed |
|----------------| / \ |------| |------------------|
^- email lost in vain
Single SMTP solution may work in case of network or other failures
As long as MailTime has not received confirmation what email is sent
it will keep the letter in the queue and retry to send it again
|----------------| / |------| |------------------|
| Mail Time | --X---> | SMTP | ------> | ^_^ Happy user |
|---^------------| / |------| |------^-----------|
\-------------/ ^- We will try later /
\- put it back into queue /
\----------Once connection is back ------/
Rotate email transports by using multiple SMTP providers. MailTime support two strategies backup (rotate when failed) and balancer (round-robin rotation)
|--------|
/--X--| SMTP 1 |
/ ^ |--------|
/ \--- Retry with next provider
|----------------|/ |--------| |------------------|
| Mail Time | ---X--> | SMTP 2 | /->| ^_^ Happy user |
|----------------|\ ^ |--------| / |------------------|
\ \--- Retry /
\ |--------| /
\---->| SMTP 3 |--/
|--------|
It is common to have horizontally scaled "Cluster" of servers for load-balancing and for durability.
Most modern application has scheduled or recurring emails. For example, once a day — with recent news and updates. It won't be an issue with a single server setup — the server would send emails at a daily interval via timer or CRON. But in "Cluster" implementation — each server will attempt to send the same email. MailTime built to avoid sending the same email multiple times to a user from horizontally scaled applications.
For the maximum durability and agility each Application Server can run MailTime in the "Server" mode:
|===================THE=CLUSTER===================| |=QUEUE=|
| |----------| |----------| |----------| | | | |--------|
| | App | | App | | App | | | |-->| SMTP 1 |------\
| | Server 1 | | Server 2 | | Server 3 | | | | |--------| \
| |-----\----| |----\-----| |----\-----| | | | |-------------|
| \---------------\----------------\----------> | |--------| | ^_^ |
| | | |-->| SMTP 2 |-->| Happy users |
| Each "App Server" | | | |--------| |-------------|
| runs MailTime as a "Server" | | | /
| for the maximum durability | | | |--------| /
| | | |-->| SMTP 3 |-----/
| | | | |--------|
|=================================================| |=======|
To split roles MailTime can run on a dedicated machine as micro-service. This case is great for private email servers with implemented authentication via rDNS and PTR records:
|===================THE=CLUSTER===================| |=QUEUE=| |===Mail=Time===|
| |----------| |----------| |----------| | | | | | |--------|
| | App | | App | | App | | | | | Micro-service |-->| SMTP 1 |------\
| | Server 1 | | Server 2 | | Server 3 | | | | | running | |--------| \
| |-----\----| |----\-----| |----\-----| | | | | MailTime as | |-------------|
| \---------------\----------------\----------> | | "Server" only | |--------| | ^_^ |
| | | | | sending |-->| SMTP 2 |-->| Happy users |
| Each "App Server" runs MailTime as | | | | emails | |--------| |-------------|
| a "Client" only placing emails to the queue. | | <-------- | /
| | | --------> | |--------| /
| | | | | |-->| SMTP 3 |-----/
| | | | | | |--------|
|=================================================| |=======| |===============|
backup and balancing. Use this feature to reduce the cost of SMTP services and add extra layer of durability. When one of the transports is failing to send an email — mail-time will switch to the next oneTo implement Server functionality — begin with installing nodemailer, although this package meant to be used with nodemailer, it's not added as the dependency, as nodemailer not needed by Client, and to give freedom to choose nodemailer's version to fit every project needs:
npm install --save nodemailer
Install MailTime package:
# for node@>=14.20.0
npm install --save mail-time
# for node@<14.20.0
npm install --save mail-time@=1.3.4
# for node@<8.9.0
npm install --save mail-time@=0.1.7
Setup Nodemailer's transports, Queue storage, and MailTime instance
See steps 1-4 below to learn about different parts of MailTime library and how it can get used. From configuration options to sending email
mail-time packagemail-time server
mail-time as client// import as ES Module
import { MailTime, MongoQueue, RedisQueue } from 'mail-time';
// require as CommonJS
const { MailTime, MongoQueue, RedisQueue } = require('mail-time');
For compatibility and flexibility MailTime has no dependency on nodemailer it should be installed and imported manually. Create one or more "SMTP transports" before initializing new MailTime instance.
For details and full list of options available in .createTransport() see nodemailer docs
// transports.js
import nodemailer from 'nodemailer';
// Use DIRECT transport
// and enable sending email from localhost
// install "nodemailer-direct-transport" NPM package:
import directTransport from 'nodemailer-direct-transport';
const transports = [];
const directTransportOpts = {
pool: false,
direct: true,
name: 'mail.example.com',
from: 'no-reply@example.com',
};
transports.push(nodemailer.createTransport(directTransport(directTransportOpts)));
// IMPORTANT: Add `.options` to a newly created transport,
// this is necessary to make sure options are available to MailTime package:
transports[0].options = directTransportOpts;
// Private SMTP
transports.push(nodemailer.createTransport({
host: 'smtp.example.com',
from: 'no-reply@example.com',
auth: {
user: 'no-reply',
pass: 'xxx'
},
}));
// Google Apps SMTP
transports.push(nodemailer.createTransport({
host: 'smtp.gmail.com',
from: 'no-reply@mail.example.com',
auth: {
user: 'no-reply@mail.example.com',
pass: 'xxx'
},
}));
// Mailing service (SparkPost as example)
transports.push(nodemailer.createTransport({
host: 'smtp.sparkpostmail.com',
port: 587,
from: 'no-reply@mail2.example.com',
auth: {
user: 'SMTP_Injection',
pass: 'xxx'
},
}));
export { transports };
mail-timeCreate new instance of MailTime in the Server mode, — it will be able to send and add emails to the queue.
Connecting to Redis before initiating new MailTime instance:
// mail-queue.js
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
import { transports } from './transports.js';
// Use REDIS_URL environment variable to store connection string to MongoDB
// example: "REDIS_URL=redis://127.0.0.1:6379/myapp node mail-micro-service.js"
const redisClient = await createClient({ url: process.env.REDIS_URL }).connect();
const mailQueue = new MailTime({
transports,
queue: new MongoQueue({
client: redisClient,
}),
josk: {
adapter: {
type: 'redis',
client: redisClient,
}
},
template: MailTime.Template // Use default template
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"Awesome App" <${transport.options.from}>`;
},
onError(error, email, details) {
console.log(`Email "${email.mailOptions.subject}" wasn't sent to ${email.mailOptions.to}`, error, details);
},
onSent(email, details) {
console.log(`Email "${email.mailOptions.subject}" successfully sent to ${email.mailOptions.to}`, details);
},
});
export { mailQueue };
Connecting to MongoDB before initiating new MailTime instance:
// mail-queue.js
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { transports } from './transports.js';
// Use MONGO_URL environment variable to store connection string to MongoDB
// example: "MONGO_URL=mongodb://127.0.0.1:27017/myapp node mail-micro-service.js"
const mongodb = (await MongoClient.connect(process.env.MONGO_URL)).db('database');
const mailQueue = new MailTime({
transports,
queue: new MongoQueue({
db: mongodb,
}),
josk: {
adapter: {
type: 'mongo',
db: mongodb,
}
},
template: MailTime.Template // Use default template
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"Awesome App" <${transport.options.from}>`;
},
onError(error, email, details) {
console.log(`Email "${email.mailOptions.subject}" wasn't sent to ${email.mailOptions.to}`, error, details);
},
onSent(email, details) {
console.log(`Email "${email.mailOptions.subject}" successfully sent to ${email.mailOptions.to}`, details);
},
});
export { mailQueue };
Only one MailTime Server instance required to send email. In the other parts of an app (like UI units or in sub-apps) use mail-time in the Client mode to add emails to queue
// mail-queue.js
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
const mailQueue = new MailTime({
type: 'client',
queue: new RedisQueue({
client: await createClient({ url: 'redis://url' }).connect()
}),
});
export { mailQueue };
Import created mailQueue where needed and call .sendMail() method. See NodeMailer's message configuration documentation for details
import { mailQueue } from './mail-queue.js';
await mailQueue.sendMail({
to: 'user@gmail.com',
subject: 'You\'ve got an email!',
text: 'Plain text message',
html: '<h1>HTML</h1><p>Styled message</p>'
});
MailTime uses separate storage for Queue management and Scheduler. In the example below MongoDB is used for both
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { transports } from './transports.js';
const db = (await MongoClient.connect('mongodb://url')).db('database');
const mailQueue = new MailTime({
queue: new MongoQueue({
db: db,
}),
josk: {
adapter: {
type: 'mongo',
db: db,
}
},
transports,
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"Awesome App" <${transport.options.from}>`;
}
});
MailTime uses separate storage for Queue management and Scheduler. In the example below MongoDB is used for queue and Redis is used for scheduler
import { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { createClient } from 'redis';
import { transports } from './transports.js';
const mailQueue = new MailTime({
queue: new MongoQueue({
db: (await MongoClient.connect('mongodb://url')).db('database'),
}),
josk: {
adapter: {
type: 'redis',
client: await createClient({ url: 'redis://url' }).connect(),
}
},
transports,
from(transport) {
return `"Awesome App" <${transport.options.from}>`;
}
});
MailTime uses separate storage for Queue management and Scheduler. In the example below Redis is used for queue and MongoDB is used for scheduler
import { MailTime, RedisQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { createClient } from 'redis';
import { transports } from './transports.js';
const mailQueue = new MailTime({
queue: new RedisQueue({
client: await createClient({ url: 'redis://url' }).connect(),
}),
josk: {
adapter: {
type: 'mongo',
db: (await MongoClient.connect('mongodb://url')).db('database'),
}
},
transports,
from(transport) {
return `"Awesome App" <${transport.options.from}>`;
}
});
MailTime uses separate storage for Queue management and Scheduler. In the example below Redis is used for both
import { MailTime, RedisQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
import { createClient } from 'redis';
import { transports } from './transports.js';
const redisClient = await createClient({ url: 'redis://url' }).connect();
const mailQueue = new MailTime({
queue: new RedisQueue({
client: redisClient,
}),
josk: {
adapter: {
type: 'redis',
client: redisClient,
}
},
transports,
from(transport) {
return `"Awesome App" <${transport.options.from}>`;
}
});
MailTime instances usage exampleCreate two MailTime instances with different settings. One for urgent (e.g. "transactional" emails), and another one for other types of emails (e.g. "marketing" emails)
import { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
import { transports } from './transports.js';
const redisClient = await createClient({ url: 'redis://url' }).connect();
// CREATE mailQueue FOR NON-URGENT EMAILS WHICH IS OKAY TO CONCATENATE
const mailQueue = new MailTime({
queue: new RedisQueue({
client: redisClient,
}),
transports,
strategy: 'backup',
failsToNext: 1,
concatEmails: true,
josk: {
adapter: {
type: 'redis',
client: redisClient
},
zombieTime: 120000
}
});
// CREATE mailInstantQueue FOR TRANSACTIONAL EMAILS AND ALERTS
const mailInstantQueue = new MailTime({
queue: new RedisQueue({
client: redisClient,
prefix: 'instant'
}),
transports,
prefix: 'instant',
retryDelay: 2000,
strategy: 'backup',
failsToNext: 1,
concatEmails: false,
josk: {
adapter: {
type: 'redis',
client: redisClient
},
zombieTime: 20000
}
});
await mailQueue.sendMail({
to: 'user@gmail.com',
subject: 'You\'ve got an email!',
text: 'Plain text message',
html: '<h1>HTML</h1><p>Styled message</p>'
});
await mailInstantQueue.sendMail({
to: 'user@gmail.com',
subject: 'Sign in request',
text: 'Your OTP login code: xxxx:',
html: '<h1>Code:</h1><code>XXXX</code>'
});
All options passed to the .sendMail() method are available inside text, html, and global templates
const templates = {
global: '<html xmlns="http://www.w3.org/1999/xhtml"><head><title>{{subject}}</title></head><body>{{{html}}}<footer>Message sent to @{{username}} user ({{to}})</footer></body></html>',
signInCode: {
text: 'Hello @{{username}}! Here\'s your login code: {{code}}',
html: `<h1>Sign-in request</h1><p>Hello @{{username}}! <p>Copy your login code below:</p> <pre><code>{{code}}</code></pre>`
}
};
const mailQueue = new MailTime({
queue: new RedisQueue({ /* ... */ }),
template: templates.global
});
await mailQueue.sendMail({
to: 'user@gmail.com',
subject: 'Sign-in request',
username: 'johndoe',
code: 'XXXXX-YY',
text: templates.signInCode.text,
html: templates.signInCode.html
});
All available constructor options and .sendMail() method API overview
new MailTime(opts) constructoropts {object} - Configuration objectopts.type {string} - [Optional] client or server, default - serveropts.queue {RedisQueue|MongoQueue|CustomQueue} - Queue storage driver instanceopts.transports {[object]} - [Required for "server"] An array of nodemailer's transports, returned from nodemailer.createTransport({}). Required for {type: 'server'}opts.josk {object} - [Required for "server"] JoSk package optionsopts.josk.adapter {object|RedisAdapter|MongoAdapter|CustomAdapter} - Config object or Adapter instanceopts.josk.adapter.type {string} - One of mongo or redis; Pass josk.adapter.type to avoid burden of creating Adapter instance manuallyopts.josk.adapter.client {RedisClient} - RedisClient instanceopts.josk.adapter.db {Db} - Mongo's Db instanceopts.josk[option] {mix} - Any other options passed to JoSk instanceopts.from {function} - [Optional] A function which returns string of from field, format: "MyApp" <user@example.com>opts.strategy {string} - [Optional] backup or balancer, default - backup. If set to backup, first transport will be used unless failed to send failsToNext times. If set to balancer - transports will be used equally in round robin chainopts.failsToNext {number} - [Optional] After how many failed "send attempts" switch to the next transport, applied only for backup strategy, default - 4opts.prefix {string} - [Optional] Use unique prefixes to create multiple MailTime instances within the same applicationopts.retries {number} - [Optional] How many times resend failed emails, default - 60opts.retryDelay {number} - [Optional] Interval in milliseconds between send re-tries, default - 60000opts.keepHistory {boolean} - [Optional] By default sent emails not stored in the database. Set { keepHistory: true } to keep queue task as it is in the database, default - falseopts.concatEmails {boolean} - [Optional] Concatenate email by to field (e.g. to the same addressee), default - falseopts.concatSubject {string} - [Optional] Email subject used in concatenated email, default - Multiple notificationsopts.concatDelimiter {string} - [Optional] HTML or plain string delimiter used between concatenated email, default - <hr>opts.concatDelay {number} - [Optional] Time in milliseconds while emails are waiting to be concatenated, default - 60000opts.revolvingInterval {number} - [Optional] Interval in milliseconds in between queue checks, default - 256opts.template {string} - [Optional] Mustache-like template, default - {{{html}}}, all options passed to sendMail is available in Template, like to, subject, text, html or any other custom option. Use {{opt}} for string placeholders and {{{opt}}} for html placeholdersopts.onError(error, email, details) {function} - [Optional] called when email has failed to get sent and exhausted all send attempts (opts.retries), called with 3 arguments:
error {Error|object} - Error objectemail {object} - email's objectdetails {object} - not always present, details from SMTP protocolopts.onSent(email, details) {function} - [Optional] called when email was successfully handed over to receiving/recipient's SMTP server, called with 2 arguments:
email {object} - email's objectdetails {object} - not always present, details from SMTP server/protocolimport { MailTime, MongoQueue, RedisQueue } from 'mail-time';
import nodemailer from 'nodemailer';
import { createClient } from 'redis';
const redisClient = await createClient({ url: 'redis://url' }).connect();
const mailQueue = new MailTime({
type: 'server',
strategy: 'backup',
prefix: 'appMailQueue',
transports: [nodemailer.createTransport({/* ... */})],
failsToNext: 4,
retries: 60,
retryDelay: 60000,
keepHistory: false,
concatEmails: false,
concatDelay: 60000,
concatDelimiter: '<hr>',
concatSubject: 'Multiple notifications',
revolvingInterval: 256,
template: '{{{html}}}',
queue: new RedisQueue({
client: redisClient,
prefix: 'appMailQueue',
}),
josk: {
adapter: {
type: 'redis',
client: redisClient,
}
},
from(transport) {
// To pass spam-filters `from` field should be correctly set
// for each transport, check `transport` object for more options
return `"App Name" <${transport.options.from}>`;
},
onError(error, email, details) {
console.log(`Email "${email.mailOptions.subject}" wasn't sent to ${email.mailOptions.to}`, error, details);
},
onSent(email, details) {
console.log(`Email "${email.mailOptions.subject}" successfully sent to ${email.mailOptions.to}`, details);
},
});
await mailQueue.sendMail({
to: 'johndoe@example.com',
subject: 'Email subject',
text: 'You have got email!',
html: '<p>You have got email!</p>',
});
new RedisQueue(opts) constructorCreate Redis Queue instance. Use for opts.queue when creating MailTime instance
opts {object} - Configuration objectopts.client {RedisClient} - Required, Redis'es RedisClient instance, like one returned from await redis.createClient().connect() methodopts.prefix {string} - Optional prefix for scope isolation; use when creating multiple MailTime instances within the single applicationimport { MailTime, RedisQueue } from 'mail-time';
import { createClient } from 'redis';
new RedisQueue({
client: await createClient({ url: 'redis://url' }).connect(),
prefix: 'appMailQueue',
});
new MongoQueue(opts) constructorCreate MongoDB Queue instance. Use for opts.queue when creating MailTime instance
opts {object} - Configuration objectopts.db {Db} - Required, Mongo's Db instance, like one returned from MongoClient#db()opts.prefix {string} - Optional prefix for scope isolation; use when creating multiple MailTime instances within the single applicationimport { MailTime, MongoQueue } from 'mail-time';
import { MongoClient } from 'mongodb';
new MongoQueue({
db: (await MongoClient.connect('mongodb://url')).db('database'),
prefix: 'appMailQueue',
});
sendMail(opts)Add email to the queue. Returns Promise<string> unique email's uuid
opts {object} - Configuration objectopts.sendAt {number} - When email should be sent, default - Date.now()opts.template {string} - Email specific template, this will override default template passed to MailTime constructoropts.concatSubject {string} - Email specific concatenation subject, this will override default concatenation subject passed to MailTime constructoropts[key] {mix} - Other custom and NodeMailer specific options, like text, html and to, learn more here. Note: if attachments are used via path — file must exists on all micro-services serverscancelMail(uuid)Remove email from queue. Returns Promise<boolean> — true if cancelled or false if not found, was sent, or was cancelled previously. Throws Error
uuid {string|promise} — email's uuid returned from .sendEmail() methodimport { mailQueue } from './mail-queue.js';
const uuid = await mailQueue.sendMail({
to: 'johndoe@example.com',
subject: 'Email subject',
text: 'You have got email!',
html: '<p>You have got email!</p>',
});
await mailQueue.cancelMail(uuid);
static MailTime.TemplateSimple and bulletproof HTML template, see its source. Usage example:
import { MailTime, MongoQueue, RedisQueue } from 'mail-time';
// Make it default
const mailQueue = new MailTime({
/* .. */
template: MailTime.Template
});
// For single letter
mailQueue.sendMail({
/* .. */
template: MailTime.Template
});
Pass custom template via template property to .sendMail() method
mailQueue.sendMail({
to: 'user@gmail.com',
userName: 'Mike',
subject: 'Sign up confirmation',
text: 'Hello {{userName}}, \r\n Thank you for registration \r\n Your login: {{to}}',
html: '<div style="text-align: center"><h1>Hello {{userName}}</h1><p><ul><li>Thank you for registration</li><li>Your login: {{to}}</li></ul></p></div>',
template: '<body>{{{html}}}</body>'
});
# Before running tests make sure NODE_ENV === development
# Install NPM dependencies
npm install --save-dev
# DEFAULT RUN
REDIS_URL="redis://127.0.0.1:6379" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test
# OPTIONALLY RUN WITH CUSTOM DOMAIN
EMAIL_DOMAIN="your-domain.com" REDIS_URL="redis://127.0.0.1:6379" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test
# IF SOME TESTS ARE FAILING: ENABLE DEBUG
DEBUG="true" REDIS_URL="redis://127.0.0.1:6379" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test
# Be patient, tests are taking around 8 mins
FAQs
📮 Email queue extending NodeMailer with multi SMTP transports and horizontally scaled apps support
The npm package mail-time receives a total of 198 weekly downloads. As such, mail-time popularity was classified as not popular.
We found that mail-time demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.