An Apache 2.0 NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
⚠️ NOTICE: The AllenNLP library is now in maintenance mode. That means we are no longer adding new features or upgrading dependencies. We will still respond to questions and address bugs as they arise up until December 16th, 2022. If you have any concerns or are interested in maintaining AllenNLP going forward, please open an issue on this repository.
AllenNLP has been a big success, but as the field is advancing quickly it's time to focus on new initiatives. We're working hard to make AI2 Tango the best way to organize research codebases. If you are an active user of AllenNLP, here are some suggested alternatives:
modules and nn packages, check out delmaksym/allennlp-light. It's even compatible with AI2 Tango!If you're interested in using AllenNLP for model development, we recommend you check out the AllenNLP Guide for a thorough introduction to the library, followed by our more advanced guides on GitHub Discussions.
When you're ready to start your project, we've created a couple of template repositories that you can use as a starting place:
allennlp train and config files to specify experiments, use this
template. We recommend this approach.In addition, there are external tutorials:
And others on the AI2 AllenNLP blog.
AllenNLP supports loading "plugins" dynamically. A plugin is just a Python package that
provides custom registered classes or additional allennlp subcommands.
There is ecosystem of open source plugins, some of which are maintained by the AllenNLP team here at AI2, and some of which are maintained by the broader community.
| Plugin | Maintainer | CLI | Description |
| allennlp-models | AI2 | No | A collection of state-of-the-art models |
| allennlp-semparse | AI2 | No | A framework for building semantic parsers |
| allennlp-server | AI2 | Yes | A simple demo server for serving models |
| allennlp-optuna | Makoto Hiramatsu | Yes | Optuna integration for hyperparameter optimization |
AllenNLP will automatically find any official AI2-maintained plugins that you have installed,
but for AllenNLP to find personal or third-party plugins you've installed,
you also have to create either a local plugins file named .allennlp_plugins
in the directory where you run the allennlp command, or a global plugins file at ~/.allennlp/plugins.
The file should list the plugin modules that you want to be loaded, one per line.
To test that your plugins can be found and imported by AllenNLP, you can run the allennlp test-install command.
Each discovered plugin will be logged to the terminal.
For more information about plugins, see the plugins API docs. And for information on how to create a custom subcommand to distribute as a plugin, see the subcommand API docs.
| allennlp | An open-source NLP research library, built on PyTorch |
| allennlp.commands | Functionality for the CLI |
| allennlp.common | Utility modules that are used across the library |
| allennlp.data | A data processing module for loading datasets and encoding strings as integers for representation in matrices |
| allennlp.fairness | A module for bias mitigation and fairness algorithms and metrics |
| allennlp.modules | A collection of PyTorch modules for use with text |
| allennlp.nn | Tensor utility functions, such as initializers and activation functions |
| allennlp.training | Functionality for training models |
AllenNLP requires Python 3.6.1 or later and PyTorch.
We support AllenNLP on Mac and Linux environments. We presently do not support Windows but are open to contributions.
The simplest way to install AllenNLP is using conda (you can choose a different python version):
conda install -c conda-forge python=3.8 allennlp
To install optional packages, such as checklist, use
conda install -c conda-forge allennlp-checklist
or simply install allennlp-all directly. The plugins mentioned above are similarly installable, e.g.
conda install -c conda-forge allennlp-models allennlp-semparse allennlp-server allennlp-optuna
It's recommended that you install the PyTorch ecosystem before installing AllenNLP by following the instructions on pytorch.org.
After that, just run pip install allennlp.
⚠️ If you're using Python 3.7 or greater, you should ensure that you don't have the PyPI version of
dataclassesinstalled after running the above command, as this could cause issues on certain platforms. You can quickly check this by runningpip freeze | grep dataclasses. If you see something likedataclasses=0.6in the output, then just runpip uninstall -y dataclasses.
If you need pointers on setting up an appropriate Python environment or would like to install AllenNLP using a different method, see below.
Conda can be used set up a virtual environment with the version of Python required for AllenNLP. If you already have a Python 3 environment you want to use, you can skip to the 'installing via pip' section.
Create a Conda environment with Python 3.8 (3.7 or 3.9 would work as well):
conda create -n allennlp_env python=3.8
Activate the Conda environment. You will need to activate the Conda environment in each terminal in which you want to use AllenNLP:
conda activate allennlp_env
Installing the library and dependencies is simple using pip.
pip install allennlp
To install the optional dependencies, such as checklist, run
pip install allennlp[checklist]
Or you can just install all optional dependencies with pip install allennlp[all].
Looking for bleeding edge features? You can install nightly releases directly from pypi
AllenNLP installs a script when you install the python package, so you can run allennlp commands just by typing allennlp into a terminal. For example, you can now test your installation with allennlp test-install.
You may also want to install allennlp-models, which contains the NLP constructs to train and run our officially
supported models, many of which are hosted at https://demo.allennlp.org.
pip install allennlp-models
Docker provides a virtual machine with everything set up to run AllenNLP-- whether you will leverage a GPU or just run on a CPU. Docker provides more isolation and consistency, and also makes it easy to distribute your environment to a compute cluster.
AllenNLP provides official Docker images with the library and all of its dependencies installed.
Once you have installed Docker, you should also install the NVIDIA Container Toolkit if you have GPUs available.
Then run the following command to get an environment that will run on GPU:
mkdir -p $HOME/.allennlp/
docker run --rm --gpus all -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest
You can test the Docker environment with
docker run --rm --gpus all -v $HOME/.allennlp:/root/.allennlp allennlp/allennlp:latest test-install
If you don't have GPUs available, just omit the --gpus all flag.
For various reasons you may need to create your own AllenNLP Docker image, such as if you need a different version
of PyTorch. To do so, just run make docker-image from the root of your local clone of AllenNLP.
By default this builds an image with the tag allennlp/allennlp, but you can change this to anything you want
by setting the DOCKER_IMAGE_NAME flag when you call make. For example,
make docker-image DOCKER_IMAGE_NAME=my-allennlp.
If you want to use a different version of Python or PyTorch, set the flags DOCKER_PYTHON_VERSION and DOCKER_TORCH_VERSION to something like
3.9 and 1.9.0-cuda10.2, respectively. These flags together determine the base image that is used. You can see the list of valid
combinations in this GitHub Container Registry: github.com/allenai/docker-images/pkgs/container/pytorch.
After building the image you should be able to see it listed by running docker images allennlp.
REPOSITORY TAG IMAGE ID CREATED SIZE
allennlp/allennlp latest b66aee6cb593 5 minutes ago 2.38GB
You can also install AllenNLP by cloning our git repository:
git clone https://github.com/allenai/allennlp.git
Create a Python 3.7 or 3.8 virtual environment, and install AllenNLP in editable mode by running:
pip install -U pip setuptools wheel
pip install --editable .[dev,all]
This will make allennlp available on your system but it will use the sources from the local clone
you made of the source repository.
You can test your installation with allennlp test-install.
See https://github.com/allenai/allennlp-models
for instructions on installing allennlp-models from source.
Once you've installed AllenNLP, you can run the command-line interface
with the allennlp command (whether you installed from pip or from source).
allennlp has various subcommands such as train, evaluate, and predict.
To see the full usage information, run allennlp --help.
You can test your installation by running allennlp test-install.
Everyone is welcome to file issues with either feature requests, bug reports, or general questions. As a small team with our own internal goals, we may ask for contributions if a prompt fix doesn't fit into our roadmap. To keep things tidy we will often close issues we think are answered, but don't hesitate to follow up if further discussion is needed.
The AllenNLP team at AI2 (@allenai) welcomes contributions from the community.
If you're a first time contributor, we recommend you start by reading our CONTRIBUTING.md guide.
Then have a look at our issues with the tag Good First Issue.
If you would like to contribute a larger feature, we recommend first creating an issue with a proposed design for discussion. This will prevent you from spending significant time on an implementation which has a technical limitation someone could have pointed out early on. Small contributions can be made directly in a pull request.
Pull requests (PRs) must have one approving review and no requested changes before they are merged. As AllenNLP is primarily driven by AI2 we reserve the right to reject or revert contributions that we don't think are good additions.
If you use AllenNLP in your research, please cite AllenNLP: A Deep Semantic Natural Language Processing Platform.
@inproceedings{Gardner2017AllenNLP,
title={AllenNLP: A Deep Semantic Natural Language Processing Platform},
author={Matt Gardner and Joel Grus and Mark Neumann and Oyvind Tafjord
and Pradeep Dasigi and Nelson F. Liu and Matthew Peters and
Michael Schmitz and Luke S. Zettlemoyer},
year={2017},
Eprint = {arXiv:1803.07640},
}
AllenNLP is an open-source project backed by the Allen Institute for Artificial Intelligence (AI2). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering. To learn more about who specifically contributed to this codebase, see our contributors page.
FAQs
An open-source NLP research library, built on PyTorch.
We found that allennlp demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket’s threat research team has detected six malicious npm packages typosquatting popular libraries to insert SSH backdoors.
Security News
MITRE's 2024 CWE Top 25 highlights critical software vulnerabilities like XSS, SQL Injection, and CSRF, reflecting shifts due to a refined ranking methodology.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.