ucb

ucb

A upper confidence bounds algorithm for multi-armed bandit problems

This implementation is based on Bandit Algorithms for Website Optimization and related empirical research in "Algorithms for the multi-armed bandit problem".

Specification

This module conforms to the BanditLab/1.0 specification.

Quick start

First, install this module in your project:

npm install ucb --save

Then, use the algorithm:

1. Create an optimizer with 3 arms:

   var Algorithm = require('ucb');
   
   var algorithm = new Algorithm({
     arms: 3
   });
   

2. Select an arm (for exploration or exploitation, according to the algorithm):

   algorithm.select().then(function (arm) {
     ...
   });
   

3. Report the reward earned from a chosen arm:

   algorithm.reward(armId, value).then(function (n) {
     ...
   });
   

API

Algorithm([config])

Create a new optimization algorithm.

Arguments

• config (Object, Optional): algorithm instance parameters

The config object supports three parameters:

• arms: (Number:Integer, Optional), default=2, the number of arms over which the optimization will operate

Returns

An instance of the ucb optimization algorithm.

Example

> var Algorithm = require('ucb');
> var algorithm = new Algorithm();
> assert.equal(algorithm.arms, 2);

Algorithm#select()

Choose an arm to play, according to the specified bandit algorithm.

Arguments

None

Returns

A promise that resolves to a Number corresponding to the associated arm index.

Example

> var Algorithm = require('ucb');
> var algorithm = new Algorithm();
> algorithm.select().then(function (arm) { console.log(arm); });

0

Algorithm#reward(arm, reward)

Inform the algorithm about the payoff from a given arm.

Arguments

• arm (Integer): the arm index (provided from algorithm.select())
• reward (Number): the observed reward value (which can be 0, to indicate no reward)

Returns

A promise that resolves to a Number representing the count of observed rounds.

Example

> var Algorithm = require('ucb');
> var algorithm = new Algorithm();
> algorithm.reward(0, 1).then(function (n) { console.log(n); });

1

Algorithm#serialize()

Obtain a plain object representing the internal state of the algorithm.

Arguments

None

Returns

A promise that resolves to an Object representing parameters required to reconstruct algorithm state.

Example

> var Algorithm = require('ucb');
> var algorithm = new Algorithm();
> algorithm.serialize().then(function (state) { console.log(state); });

{
  arms: 2,
  gamma: 0.0000001,
  counts: [0, 0],
  values: [0, 0]
}

Algorithm#load(state)

Restore an instance of an algorithm to a previously serialized state. This method overrides any options parameters passed at instantiation.

Arguments

• state (Object): a serialized algorithm state (provided from algorithm.serialize())

Returns

A promise that resolves to a Number representing the count of observed rounds.

Example

> var state = {arms: 2, gamma: 0.0000001, counts: [1, 2], values: [1, 0.5]};
> var Algorithm = require('ucb');
> var algorithm = new Algorithm();
> algorithm.load(state).then(function (n) { console.log(n); });

3

Tests

To run the unit test suite:

npm test

Or, to run the test suite and view test coverage:

npm run coverage

Contribute

PRs are welcome! For bugs, please include a failing test which passes when your PR is applied. Travis CI provides on-demand testing for commits and pull requests.

Caveat emptor

Currently, this implementation relies on the native Math.random() which uses a seeded "random" number generator. In addition, the underlying calculations often encounter extended floating point numbers. Arm selection is therefore subject to JavaScript's floating point precision limitations. For general information about floating point issues see the floating point guide.

While these factors generally do not impede common application, I would consider the implementation suspect in an academic setting.