go-engine-tester

Introduction

This program can compare the strength of two Go (围棋, 囲碁, 바둑) engines that are GTP-compatible by running a batch of games between them. It can also compare the strength under different options of the same engine.

Requirements

• Node.js 8 or higher installed.

Usage

Add a file config.json, then run:

npx go-engine-tester config.json

config.json example:

{
    "first": {
        "command": "./leelaz",
        "args": ["-w", "network-173.gz", "--gtp", "--noponder"],
        "initialGtpCommands": [
            {"name": "komi", "args": ["7.5"]},
            {"name": "time_settings", "args": ["0", "0", "1"]},
            {"name": "boardsize", "args": ["19"]}
        ]
    },
    "second": {
        "command": "./leelaz",
        "args": ["-w", "network-157.gz", "--gtp", "--noponder"],
        "initialGtpCommands": [
            {"name": "komi", "args": ["7.5"]},
            {"name": "time_settings", "args": ["0", "0", "1"]},
            {"name": "boardsize", "args": ["19"]}
        ]
    },
    "times": 30
}

This will play 30 games between the two engines, alternating black and white for each game, and finally print the result. Note that the path in command is relative to the working directory not the config file. Also note that on Windows, use \\ not \ for path separator, or just use /.

For details of GTP commands such as time_settings, click here.

There's an optional spawnOptions property, so that it could pass environment variables to the engine processes. For example:

{
    "first": {
        "command": "./leelaz",
        "args": ["-w", "network-173.gz", "--gtp", "--noponder"],
        "initialGtpCommands": [
            {"name": "komi", "args": ["7.5"]},
            {"name": "time_settings", "args": ["0", "0", "1"]},
            {"name": "boardsize", "args": ["19"]}
        ],
        "spawnOptions": {
            "env": {
                "DYLD_LIBRARY_PATH": "/Users/your-name/boost_1_67_0/lib"
            }
        }
    },
    "second": {
        "command": "./leelaz",
        "args": ["-w", "network-157.gz", "--gtp", "--noponder"],
        "initialGtpCommands": [
            {"name": "komi", "args": ["7.5"]},
            {"name": "time_settings", "args": ["0", "0", "1"]},
            {"name": "boardsize", "args": ["19"]}
        ],
        "spawnOptions": {
            "env": {
                "DYLD_LIBRARY_PATH": "/Users/your-name/boost_1_67_0/lib"
            }
        }
    },
    "times": 30
}

For details of spawnOptions, click here.

FAQ

Q: Why does it restart the process for each game? Process initialization can be slow!

A: This ensures that each game is treated equally. A lot of programs are so smart that they can cache the previous calculation to speed up calculations for the next game. We must avoid that, otherwise the result may be incorrect.