PDF-Extract-Kit: High-Quality PDF Extraction Toolkit🔥🔥🔥
Easier to use: Just grab MinerU Desktop. No coding, no login, just a simple interface and smooth interactions. Enjoy it without any fuss!🚀🚀🚀
doclayout_yolo(2501) model, improving layout recognition accuracy.unimernet(2501) model, improving formula recognition accuracy.lang configuration to auto during document parsing will automatically select the appropriate OCR language model, improving the accuracy of scanned document parsing.doclayout_yolo model, which speeds up processing by more than 10 times compared to the original solution while maintaining similar parsing effects, and can be freely switched with layoutlmv3 via the configuration file.unimernet 0.2.1, improving formula parsing accuracy while significantly reducing memory usage.PDF-Extract-Kit 1.0, you need to re-download the model. Please refer to How to Download Models for detailed steps.MinerU is a tool that converts PDFs into machine-readable formats (e.g., markdown, JSON), allowing for easy extraction into any format. MinerU was born during the pre-training process of InternLM. We focus on solving symbol conversion issues in scientific literature and hope to contribute to technological development in the era of large models. Compared to well-known commercial products, MinerU is still young. If you encounter any issues or if the results are not as expected, please submit an issue on issue and attach the relevant PDF.
https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c
If you encounter any installation issues, please first consult the FAQ.
If the parsing results are not as expected, refer to the Known Issues.
There are three different ways to experience MinerU:
[!WARNING] Pre-installation Notice—Hardware and Software Environment Support
To ensure the stability and reliability of the project, we only optimize and test for specific hardware and software environments during development. This ensures that users deploying and running the project on recommended system configurations will get the best performance with the fewest compatibility issues.
By focusing resources on the mainline environment, our team can more efficiently resolve potential bugs and develop new features.
In non-mainline environments, due to the diversity of hardware and software configurations, as well as third-party dependency compatibility issues, we cannot guarantee 100% project availability. Therefore, for users who wish to use this project in non-recommended environments, we suggest carefully reading the documentation and FAQ first. Most issues already have corresponding solutions in the FAQ. We also encourage community feedback to help us gradually expand support.
| Operating System | |||
| Linux after 2019 | Windows 10 / 11 | macOS 11+ | |
| CPU | x86_64 / arm64 | x86_64(unsupported ARM Windows) | x86_64 / arm64 |
| Memory Requirements | 16GB or more, recommended 32GB+ | ||
| Storage Requirements | 20GB or more, with a preference for SSD | ||
| Python Version | 3.10(Please make sure to create a Python 3.10 virtual environment using conda) | ||
| Nvidia Driver Version | latest (Proprietary Driver) | latest | None |
| CUDA Environment | Automatic installation [12.1 (pytorch) + 11.8 (paddle)] | 11.8 (manual installation) + cuDNN v8.7.0 (manual installation) | None |
| CANN Environment(NPU support) | 8.0+(Ascend 910b) | None | None |
| GPU Hardware Support List | GPU VRAM 8GB or more | 2080~2080Ti / 3060Ti~3090Ti / 4060~4090 8G VRAM can enable all acceleration features | None |
Synced with dev branch updates:
conda create -n mineru python=3.10
conda activate mineru
pip install -U "magic-pdf[full]" --extra-index-url https://wheels.myhloli.com
Refer to How to Download Model Files for detailed instructions.
After completing the 2. Download model weight files step, the script will automatically generate a magic-pdf.json file in the user directory and configure the default model path.
You can find the magic-pdf.json file in your 【user directory】.
[!TIP] The user directory for Windows is "C:\Users\username", for Linux it is "/home/username", and for macOS it is "/Users/username".
You can modify certain configurations in this file to enable or disable features, such as table recognition:
[!NOTE] If the following items are not present in the JSON, please manually add the required items and remove the comment content (standard JSON does not support comments).
{
// other config
"layout-config": {
"model": "doclayout_yolo" // Please change to "layoutlmv3" when using layoutlmv3.
},
"formula-config": {
"mfd_model": "yolo_v8_mfd",
"mfr_model": "unimernet_small",
"enable": true // The formula recognition feature is enabled by default. If you need to disable it, please change the value here to "false".
},
"table-config": {
"model": "rapid_table", // Default to using "rapid_table", can be switched to "tablemaster" or "struct_eqtable".
"sub_model": "slanet_plus", // When the model is "rapid_table", you can choose a sub_model. The options are "slanet_plus" and "unitable"
"enable": true, // The table recognition feature is enabled by default. If you need to disable it, please change the value here to "false".
"max_time": 400
}
}
If your device supports CUDA and meets the GPU requirements of the mainline environment, you can use GPU acceleration. Please select the appropriate guide based on your system:
[!IMPORTANT] Docker requires a GPU with at least 8GB of VRAM, and all acceleration features are enabled by default.
Before running this Docker, you can use the following command to check if your device supports CUDA acceleration on Docker.
docker run --rm --gpus=all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
wget https://github.com/opendatalab/MinerU/raw/master/docker/global/Dockerfile -O Dockerfile
docker build -t mineru:latest .
docker run -it --name mineru --gpus=all mineru:latest /bin/bash -c "echo 'source /opt/mineru_venv/bin/activate' >> ~/.bashrc && exec bash"
magic-pdf --help
If your device has NPU acceleration hardware, you can follow the tutorial below to use NPU acceleration:
If your device uses Apple silicon chips, you can enable MPS acceleration for certain supported tasks (such as layout detection and formula detection).
You can enable MPS acceleration by setting the device-mode parameter to mps in the magic-pdf.json configuration file.
{
// other config
"device-mode": "mps"
}
[!TIP] Since the formula recognition task cannot utilize MPS acceleration, you can disable the formula recognition feature in tasks where it is not needed to achieve optimal performance.
You can disable the formula recognition feature by setting the
enableparameter in theformula-configsection tofalse.
[!TIP] For more information about the output files, please refer to the Output File Description.
Derived projects include secondary development projects based on MinerU by project developers and community developers,
such as application interfaces based on Gradio, RAG based on llama, web demos similar to the official website, lightweight multi-GPU load balancing client/server ends, etc.
These projects may offer more features and a better user experience.
For specific deployment methods, please refer to the Derived Project README
TODO
index and list in the main textThis project currently uses PyMuPDF to achieve advanced functionality. However, since it adheres to the AGPL license, it may impose restrictions on certain usage scenarios. In future iterations, we plan to explore and replace it with a more permissive PDF processing library to enhance user-friendliness and flexibility.
@misc{wang2024mineruopensourcesolutionprecise,
title={MinerU: An Open-Source Solution for Precise Document Content Extraction},
author={Bin Wang and Chao Xu and Xiaomeng Zhao and Linke Ouyang and Fan Wu and Zhiyuan Zhao and Rui Xu and Kaiwen Liu and Yuan Qu and Fukai Shang and Bo Zhang and Liqun Wei and Zhihao Sui and Wei Li and Botian Shi and Yu Qiao and Dahua Lin and Conghui He},
year={2024},
eprint={2409.18839},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2409.18839},
}
@article{he2024opendatalab,
title={Opendatalab: Empowering general artificial intelligence with open datasets},
author={He, Conghui and Li, Wei and Jin, Zhenjiang and Xu, Chao and Wang, Bin and Lin, Dahua},
journal={arXiv preprint arXiv:2407.13773},
year={2024}
}
Magic-Doc Fast speed ppt/pptx/doc/docx/pdf extraction tool
Magic-HTML Mixed web page extraction tool
FAQs
A practical tool for converting PDF to Markdown
We found that magic-pdf demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Bybit's $1.46B hack by North Korea's Lazarus Group pushes 2025 crypto losses to $1.6B in just two months, already surpassing all of 2024's $1.49B total.
Security News
OpenSSF has published OSPS Baseline, an initiative designed to establish a minimum set of security-related best practices for open source software projects.
Security News
Michigan TypeScript founder Dimitri Mitropoulos implements WebAssembly runtime in TypeScript types, enabling Doom to run after processing 177 terabytes of type definitions.