pyrodigal

🔥 Pyrodigal

Cython bindings and Python interface to Prodigal, an ORF finder for genomes and metagenomes. Now with SIMD!

🗺️ Overview

Pyrodigal is a Python module that provides bindings to Prodigal using Cython. It directly interacts with the Prodigal internals, which has the following advantages:

• single dependency: Pyrodigal is distributed as a Python package, so you can add it as a dependency to your project, and stop worrying about the Prodigal binary being present on the end-user machine.
• no intermediate files: Everything happens in memory, in a Python object you fully control, so you don't have to invoke the Prodigal CLI using a sub-process and temporary files. Sequences can be passed directly as strings or bytes, which avoids the overhead of formatting your input to FASTA for Prodigal.
• better memory usage: Pyrodigal uses more compact data structures compared to the original Prodigal implementation, allowing to save memory to store the same information. A heuristic is used to estimate the number of nodes to allocate based on the sequence GC% in order to minimize reallocations.
• better performance: Pyrodigal uses SIMD instructions to compute which dynamic programming nodes can be ignored when scoring connections. This can save from a third to half the runtime depending on the sequence. The Benchmarks page of the documentation contains comprehensive comparisons. See the JOSS paper for details about how this is achieved.
• same results: Pyrodigal is tested to make sure it produces exactly the same results as Prodigal v2.6.3+31b300a. This was verified extensively by Julian Hahnfeld and can be checked with his comparison repository.

📋 Features

The library now features everything from the original Prodigal CLI:

• run mode selection: Choose between single mode, using a training sequence to count nucleotide hexamers, or metagenomic mode, using pre-trained data from different organisms (prodigal -p).
• region masking: Prevent genes from being predicted across regions containing unknown nucleotides (prodigal -m).
• closed ends: Genes will be identified as running over edges if they are larger than a certain size, but this can be disabled (prodigal -c).
• training configuration: During the training process, a custom translation table can be given (prodigal -g), and the Shine-Dalgarno motif search can be forcefully bypassed (prodigal -n)
• output files: Output files can be written in a format mostly compatible with the Prodigal binary, including the protein translations in FASTA format (prodigal -a), the gene sequences in FASTA format (prodigal -d), or the potential gene scores in tabular format (prodigal -s).
• training data persistence: Getting training data from a sequence and using it for other sequences is supported; in addition, a training data file can be saved and loaded transparently (prodigal -t).

In addition, the new features are available:

• custom gene size threshold: While Prodigal uses a minimum gene size of 90 nucleotides (60 if on edge), Pyrodigal allows to customize this threshold, allowing for smaller ORFs to be identified if needed.
• custom metagenomic models: Since v3.0.0, you can use your own metagenomic models to run Pyrodigal in meta-mode. Check for instance pyrodigal-gv, which provides additional models for giant viruses and gut phages.

🐏 Memory

Pyrodigal makes several changes compared to the original Prodigal binary regarding memory management:

• Sequences are stored as raw bytes instead of compressed bitmaps. This means that the sequence itself takes 3/8th more space, but since the memory used for storing the sequence is often negligible compared to the memory used to store dynamic programming nodes, this is an acceptable trade-off for better performance when extracting said nodes.
• Node fields use smaller data types to fit into 128 bytes, compared to the 176 bytes of the original Prodigal data structure.
• Node arrays are pre-allocated based on the sequence GC% to extrapolate the probability to find a start or stop codon.
• Genes are stored in a more compact data structure than in Prodigal (which reserves a buffer to store string data), saving around 1KiB per gene.

🧶 Thread-safety

pyrodigal.GeneFinder instances are thread-safe. In addition, the find_genes method is re-entrant. This means you can train an GeneFinder instance once, and then use a pool to process sequences in parallel:

import multiprocessing.pool
import pyrodigal

gene_finder = pyrodigal.GeneFinder()
gene_finder.train(training_sequence)

with multiprocessing.pool.ThreadPool() as pool:
    predictions = pool.map(orf_finder.find_genes, sequences)

🔧 Installing

This project is supported on Python 3.7 and later.

Pyrodigal can be installed directly from PyPI, which hosts some pre-built wheels for the x86-64 architecture (Linux/MacOS/Windows) and the Aarch64 architecture (Linux/MacOS), as well as the code required to compile from source with Cython:

$ pip install pyrodigal

Otherwise, Pyrodigal is also available as a Bioconda package:

$ conda install -c bioconda pyrodigal

Check the install page of the documentation for other ways to install Pyrodigal on your machine.

💡 Example

Let's load a sequence from a GenBank file, use an GeneFinder to find all the genes it contains, and print the proteins in two-line FASTA format.

🔬 Biopython

To use the GeneFinder in single mode (corresponding to prodigal -p single, the default operation mode of Prodigal), you must explicitly call the train method with the sequence you want to use for training before trying to find genes, or you will get a RuntimeError:

import Bio.SeqIO
import pyrodigal

record = Bio.SeqIO.read("sequence.gbk", "genbank")

orf_finder = pyrodigal.GeneFinder()
orf_finder.train(bytes(record.seq))
genes = orf_finder.find_genes(bytes(record.seq))

However, in meta mode (corresponding to prodigal -p meta), you can find genes directly:

import Bio.SeqIO
import pyrodigal

record = Bio.SeqIO.read("sequence.gbk", "genbank")

orf_finder = pyrodigal.GeneFinder(meta=True)
for i, pred in enumerate(orf_finder.find_genes(bytes(record.seq))):
    print(f">{record.id}_{i+1}")
    print(pred.translate())

On older versions of Biopython (before 1.79) you will need to use record.seq.encode() instead of bytes(record.seq).

🧪 Scikit-bio

import skbio.io
import pyrodigal

seq = next(skbio.io.read("sequence.gbk", "genbank"))

orf_finder = pyrodigal.GeneFinder(meta=True)
for i, pred in enumerate(orf_finder.find_genes(seq.values.view('B'))):
    print(f">{record.id}_{i+1}")
    print(pred.translate())

We need to use the view method to get the sequence viewable by Cython as an array of unsigned char.

🔖 Citation

Pyrodigal is scientific software, with a published paper in the Journal of Open-Source Software. Please cite both Pyrodigal and Prodigal if you are using it in an academic work, for instance as:

Pyrodigal (Larralde, 2022), a Python library binding to Prodigal (Hyatt et al., 2010).

Detailed references are available on the Publications page of the online documentation.

💭 Feedback

⚠️ Issue Tracker

Found a bug ? Have an enhancement request ? Head over to the GitHub issue tracker if you need to report or ask something. If you are filing in on a bug, please include as much information as you can about the issue, and try to recreate the same bug in a simple, easily reproducible situation.

🏗️ Contributing

Contributions are more than welcome! See CONTRIBUTING.md for more details.

📋 Changelog

This project adheres to Semantic Versioning and provides a changelog in the Keep a Changelog format.

⚖️ License

This library is provided under the GNU General Public License v3.0. The Prodigal code was written by Doug Hyatt and is distributed under the terms of the GPLv3 as well. See vendor/Prodigal/LICENSE for more information.

This project is in no way not affiliated, sponsored, or otherwise endorsed by the original Prodigal authors. It was developed by Martin Larralde during his PhD project at the European Molecular Biology Laboratory in the Zeller team.