muteria

MUTERIA (MUlti-Tools and criTERIA framework for automated software testing)

https://github.com/muteria/muteria

Muteria is a Software Analysis/Testing framework that integrate multiple tools. Muteria support tools from various programming languages which are supported by implementing a driver.

Report Sample.

Muteria can be used through:

• Its API
• Commnad Line Interface (CLI)

Suported Systems

Muteria is written in Python and thus can run on Windows, Linux or macOS.

Installation

muteria requires Python 3.

• Install Muteria by running:

pip install muteria

• View the usage help:

muteria --help

Usage

Muteria requires to have the underlying tools installed on the system.

Usage example C (in Docker container)

A docker image, with preinstalled tools, can be used to run muteria on a sample C language program. The installed tools are: GNU GCov, KLEE, Shadow, Mart, SEMu.

• Pull the docker image:

docker pull thierrytct/cm

• run the docker image in a container:

docker run -it --rm thierrytct/cm bash

• Download the example program:

git clone https://github.com/muteria/example_c.git 

• Change into the example program directory:

cd example_c

• run using the configuration file in ctrl/conf.py

muteria --config ctrl/conf.py --lang c run

Usage example Python

Example of measuring coverage for a python program using coverage.py.

• Install coverage.py:

pip install coverage

• Download the example program:

git clone https://github.com/muteria/example_python.git 

• Change into the example program directory:

cd example_python

• run using the configuration file in ctrl/conf.py

muteria --config ctrl/conf.py --lang python run

Drivers development

Testcase Tools Drivers

Details on writing drivers for testcase (test generation) tools can be found here.

Test Criteria Tools Drivers

Details on writing drivers for test criteria tools can be found here.

Current Limitation

• Interface difference between multiple versions of the same tool

Development

Things to fix

• Ensure tools plugins do not leave the repo srcs or exes in an infected state. They must cleanup the changes made. Ensure that by cleaning for them at top level (when the driver's methods are called in the meta tools)

• Handle reporting test error in stats (first within meta test execution)

• Enable having no criterion set

• Complete and test the web UI.

• Choice of test level (unit, system, ...)

• Complete the documentation

• Augment the test suite

Features

• parallelism

• Implement option to run phases of the analysis in parallel.
• Implement option to run tests in parallel

• Web Interface

• Front-end UI
• server (back-end)
• Write custom configuration function on-line
• Write manual tests on-line

• Command Line Interface

• basic interaction to launch execution

• Library API interface

• API in docstring
• make the documented API available

• Reporting

• Basic metric display
• cross project reporting (for experiment and evolution)
• plotting

• Checkpointing and Logging

• checkpoint and logging implemented
• logging optional
• checkpointing optional

• Multiple projects

• obtain data from previous versions execution in current (regression)
• merge data across multiple project (for experiments)

• Test Adequacy Criteria(TAC)

• Support adding new TACs
• Support enabling/disabling TAC during execution

• Tools

• support plugging in test generation tools
• support plugging in TAC instrumenting tools
• support plugging in TAC test objectives selection/prioritization.
• support plugging in test case selection prioritization techniques.
• support adding test formats (for test execution)
• support adding build systems.
• support custom build script

• Programming languages

• support adding programming languages
• support multi-languages tools