bcqa

BCQA (Benchmarking Complex QA)

BCQA is a benchmark for a wide range of complex Qa tasks. It also aims to provide a easy to use framework for evaluating retrieval and reasoning approaches for answering complex multi-hop questions.

Setup

• Clone the repo
  
• Create a conda environment conda create -n bcqa
  
• pip install -e .
  

Running Evaluation

The evaluation scripts for retreival and LLMs are in the evaluation folder

For instance to run dpr retreival for Wikimultihopqa run
python3 evaluation/wikimultihop/run_dpr_inference.py

Before running the above script make sure you have configured the correct paths for the data and corpus files in evaluation/config.ini

Example: wikimultihopqa = /home/bcqa/BCQA/2wikimultihopQA
wikimultihopqa-corpus = /home/bcqa/BCQA/wiki_musique_corpus.json

Coding Practices

Auto-formatting code

• Install black: pip install black or conda install black
• In your IDE: Enable formatting on save.
• Install isort: pip install isort or conda install isort
• In your IDE: Enable sorting import on save.

In VS Code, you can do this using the following config:

{
    "editor.formatOnSave": true,
    "editor.codeActionsOnSave": {
        "source.organizeImports": true
    }
}

Type hints

Use type hints for everything! No exceptions.

Docstrings

Write a docstring for every function (except the main function). We use the Google format. In VS Code, you can use autoDocstring.

Example

def sum(a: float, b: float) -> float:
    """Compute the sum of a and b.

    Args:
        a (float): First number.
        b (float): Second number.
    
    Returns:
        float: The sum of a and b.
    """

    return a + b