ctrl-benchmark

Continual Transfer Learning Benchmark

The CTrL benchmark aims to help research by studying the transfer behaviour of different models in the Lifelong Learning context.

For a quick presentation of the benchmark, please check out this blog post.

For more details, the paper is available on arxiv.

Creating a Stream:

The TaskGenerator class is at the center of the implementation of the CTrL Benchmark. It gives access to a high-level API allowing to seamlessly generate a wide variety of streams with a loose coupling between the different components such as the underlying dataset(s), the strategy to generate the tasks (split, incremental, mixture of datasets, ...) and the processing to apply to each task.

The 3 main components of a Task Generator are:

• A pool of concepts to select from to generate the tasks. It can be a few classes, a full dataset or even a mixture of datasets.
• A pool of transformation that can be modified or combined to apply specific processing to the data for each task.
• A Strategy, describing how to combine the conecpts and trasnformation over time to generate an actual stream.

Each of these components can be created by hand or using our automatic TaskGenerator creation tool using yaml configuration files.

For examples simply executing

import ctrl
task_gen = ctrl.get_stream('s_minus')

will return the corresponding task generator that be used either directy to generate tasks on the fly:

t1 = task_gen.add_task()
t2 = task_gen.add_task()
t3 = task_gen.add_task()
...

or as an iterator:

for t in task_gen:
    print(t)

Available streams:

In the current version, only the streams of the CTrL benchmark are directly available, they can be obtained by passing the following name arguments in ctrl.get_stream:

• S⁺: "s_plus"
• S^-: "s_minus"
• Sⁱⁿ: "s_in"
• S^out: "s_out"
• S^pl: "s_pl"
• S^long: "s_long"

More documentation and details on the internal components will be progressively added.

See the CONTRIBUTING file for how to help out.

LICENSE

See the LICENSE file.