Dedoc is an open universal system for converting documents to a unified output format. It extracts a document’s logical structure and content: tables, text formatting and metadata. The document’s content is represented as a tree storing headings and lists of any level. Dedoc can be integrated in a document contents and structure analysis system as a separate module.
Workflow description is given here
Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and unstructured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats. Document structure extraction is fully automatic regardless of input data type. Metadata and text formatting are also extracted automatically.
In 2022, the system won a grant to support the development of promising AI projects from the Innovation Assistance Foundation (Фонд содействия инновациям).
The system processes different document formats. The main formats are listed below:
| Format group | Description |
|---|---|
| Office formats | DOCX, XLSX, PPTX and formats that can be converted to them. Handling of these formats is held by analysis of format inner representation and using specialized libraries (python-docx, BeautifulSoup) |
| HTML, EML, MHTML | HTML documents are parsed using tags analysis, HTML handler is used for handling documents of other formats in this group |
| TXT | Only raw textual content is analyzed |
| Archives | Attachments of the archive are analyzed |
| PDF, document images | Copyable PDF documents (with a textual layer) can be handled using pdfminer-six library or tabby software. Non-copyable PDF documents or images are handled using Tesseract-OCR, machine learning methods (including neural network methods) and image processing methods |
Dedoc can only process scanned black and white documents, such as technical specifications, regulations, articles, etc.
In particular, dedoc recognizes tabular information only from tables with explicit boundaries. Here are examples of documents that can be processed by an dedoc's image handler:
The system also automatically detects and corrects the orientation of scanned documents
This project may be useful as a first step of an automatic document analysis pipeline (e.g. before the NLP part). Dedoc is in demand for information analytic systems, information leak monitoring systems, as well as for natural language processing systems. The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents.
Relevant documentation of dedoc is available here
Have questions or want to discuss Dedoc? Join our Telegram chat and connect with the community and the developers.
Join our Telegram channel to get notifications about the most recent updates.
This project has a REST api and you can run it in Docker container.
Also, dedoc can be installed as a library via pip.
There are two ways to install and run dedoc as a web application or a library that are described below.
You should have git and docker installed for running dedoc by this method.
This method is more flexible because it doesn't depend on the operating system and other user's limitations,
still, the docker application should be installed and configured properly.
If you don't need to change the application configuration, you may use the built docker image as well.
docker pull dedocproject/dedoc
docker run -p 1231:1231 --rm dedocproject/dedoc python3 /dedoc_root/dedoc/main.py
Go to dockerhub to get more information about available dedoc images.
If you need to change some application settings, you may update config.py according to your needs and re-build the image.
You can build and run image:
git clone https://github.com/ispras/dedoc
dedoc directorycd dedoc
docker compose up --build
test="true" docker compose up --build
If you need to change some application settings, you may update config.py according to your needs and re-build the image.
If you don't want to use docker for running the application, it's possible to run dedoc locally.
However, it isn't suitable for any operating system (Ubuntu 20+ is recommended) and
there may be not enough machine resources for its work.
You should have python (python3.8, python3.9 are recommended) and pip installed.
Installation instructions via pip are available here.
You can also install and run dedoc locally from sources using this instruction.
FAQs
Extract content and logical tree structure from textual documents
We found that dedoc demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
An advanced npm supply chain attack is leveraging Ethereum smart contracts for decentralized, persistent malware control, evading traditional defenses.
Security News
Research
Attackers are impersonating Sindre Sorhus on npm with a fake 'chalk-node' package containing a malicious backdoor to compromise developers' projects.
Security News
The npm package for the LottieFiles Player web component was hit with a supply chain attack after a software engineer's npmjs credentials were compromised.