ucb

ucb

An upper confidence bounds algorithm for multi-armed bandit problems

This implementation is based on Bandit Algorithms for Website Optimization and related empirical research in "Algorithms for the multi-armed bandit problem".

Specification

This module conforms to the BanditLab/2.0 specification.

Quick start

First, install this module in your project:

npm install ucb --save

Then, use the algorithm:

1. Create an optimizer with 3 arms and epsilon 0.25:

   var Algorithm = require('ucb');
   
   var algorithm = new Algorithm({
     arms: 3,
     epsilon: 0.25
   });
   

2. Select an arm (for exploration or exploitation, according to the algorithm):

   algorithm.select().then(function (arm) {
     // do something based on the chosen arm
   });
   

3. Report the reward earned from a chosen arm:

   algorithm.reward(arm, value);
   

API

Algorithm(config)

Creates a new optimization algorithm.

Arguments

• config (Object): algorithm instance parameters

The config object supports two parameters:

• arms: (Number:Integer, Optional), default=2, the number of arms over which the optimization will operate

Alternatively, the state object returned from Algorithm#serialize can be passed as config.

Returns

An instance of the ucb optimization algorithm.

Example

var Algorithm = require('ucb');
var algorithm = new Algorithm();

assert.equal(algorithm.arms, 2);

Or, with a passed config:

var Algorithm = require('ucb');
var algorithm = new Algorithm({arms: 4});

assert.equal(algorithm.arms, 4);

Algorithm#select()

Choose an arm to play, according to the specified bandit algorithm.

Arguments

None

Returns

A promise that resolves to a Number corresponding to the associated arm index.

Example

var Algorithm = require('ucb');
var algorithm = new Algorithm();

algorithm.select().then(function (arm) { console.log(arm); });

0

Algorithm#reward(arm, reward)

Inform the algorithm about the payoff from a given arm.

Arguments

• arm (Integer): the arm index (provided from algorithm.select())
• reward (Number): the observed reward value (which can be 0, to indicate no reward)

Returns

A promise that resolves to an updated instance of the algorithm.

Example

var Algorithm = require('ucb');
var algorithm = new Algorithm();

algorithm.reward(0, 1).then(function (algorithmUpdated) { console.log(algorithmUpdated) });

<Algorithm>{
  arms: 2,
  counts: [ 1, 0 ],
  values: [ 1, 0 ]
}

Algorithm#serialize()

Obtain a plain object representing the internal state of the algorithm.

Arguments

None

Returns

A promise that resolves to an Object representing parameters required to reconstruct algorithm state.

Example

var Algorithm = require('ucb');
var algorithm = new Algorithm();

algorithm.serialize().then(function (state) { console.log(state); });

{
  arms: 2,
  counts: [0, 0],
  values: [0, 0]
}

Tests

To run the unit test suite:

npm test

Or, to run the test suite and view test coverage:

npm run coverage

Contribute

PRs are welcome! For bugs, please include a failing test which passes when your PR is applied. Travis CI provides on-demand testing for commits and pull requests.

Caveat emptor

This implementation often encounter extended floating point numbers. Arm selection is therefore subject to JavaScript's floating point precision limitations. For general information about floating point issues see the floating point guide.

While these factors generally do not impede common application, I would consider the implementation suspect in an academic setting.