cleveland

Cleveland 🍃

Cleveland is a simple tool for local network discovery, with the ability to distribute commands. This makes it possible to send a task to a single server instance, and have it be executed on every server instance running Cleveland within the local network, without the need to have a list of the exact hostnames of all server nodes.

The immediate application of this is in the form of notifying servers to check for available updates of the software they run, whenever an update is deployed. It also enables the sharing of data and configuration values between servers.

This also makes it easy to create custom Slack Slash Commands for devops procedures, e.g. calling certain scripts on all servers in a VPC without keeping a list or relying on the APIs of your cloud provider. Take a look at our Slack example.

Usage

Installation

Cleveland is meant to be used as part of an existing or new Node app. In your app's main directory, install using NPM:

npm install --save cleveland

Initialisation

Starting a basic instance of the Cleveland server is simple:

const Cleveland = require('./src/index');

// Start Cleveland instance on port 9570
// (this port should only be open to local network devices, never for the entire internet)
var instance = new Cleveland();

Doing this creates a small web server with some simple API endpoints, and a process to scan the local IP space for other Cleveland instances. As soon as one is found, Cleveland attempts to connect and exchanges secret keys. Those secret keys give it permission to send commands to the other instance.

Sending commands

Now you probably want to send some commands to be distributed. The most common way to do that is to build a public web server (on a different port than the Cleveland instance, of course). You could use Express for that, by adding something like this to the code above:

const Express = require('express'),
    app = Express();

app.get('/deploy', (req, res) => {

    // Broadcast command
    var command = instance.createCommand('shell', {
        command: 'bash /var/deploy.sh'
    });

    // Add event listeners
    command.on('log', (event) => {
        // Catch logs returned by Cleveland instances that are executing the command
        console.log(event.command.id + ': ' + event.message);
    });
    command.once('complete', (event) => {
        // Fired when the ack_done count >= serverCount and only once
        console.log(event.command.id + ': DONE');
    });
});

app.listen(3000, () => {
    console.log('Listening for deploy requests on http://0.0.0.0:3000/deploy');
});

Adding handlers

The above example uses the default handler shell (see source here: default-handlers.js). You can easily add your own handlers:

// Add a 'ping' handler, which just outputs 'pong' to the console.
instance.handlers.ping = (command) => {
    
    console.log('pong');

    // Broadcast log
    command.log('Pinged has ponged successfully');

    // Mark as done
    command.done();

    // Or mark as failed (with Error/Exception as argument);
    // command.fail(new Error('Could not pong ping'));
};

Discovery logic

To discover other Cleveland instances, Cleveland determines the current machine's local IP and uses the first 3 fields as a basis, to then iterate over all IPs in the same space, scanning the ports 9570-9580 for each one. If my IP would be 192.168.178.15, this is what Cleveland would scan:

192.168.178.1:9570
192.168.178.1:9571
192.168.178.1:9572
192.168.178.1:...
192.168.178.2:9570
192.168.178.2:...
etc.

After completing this series of scans (up to IP x.x.x.255:9580), it would start again at x.x.x.1:9570.

Distribution logic

Every server, upon receiving a broadcasted command, does the following:

1. Check if it has already processed the command before (by checking if the command ID is known). If it has, ignore. Otherwise:
2. Broadcast the command to all known other Cleveland nodes, to ensure complete distribution even if some of the nodes are still in the process of scanning and aren't connected to all nodes.
3. Execute the command locally.
4. Acknowledge successful or failed execution of the command, by broadcasting the command _ack_done.

Note: The complete event on the original command is called once the source Cleveland server has received at least as many acknowledgements of completion as there are known nodes. This means the complete event might fire before it has executed on all nodes, as it might not yet have finished discovering all the nodes in the local network.

To do

• Add global server-level events, like connecting to server, disconnecting to server, receiving a command.
• Add more default handlers. Any suggestions for which ones we need?