insights-extractor

Extractlib

This is a Python package that provides a set of tools and utilities for processing and analyzing PDF documents. It includes functionality for extracting text and tables from PDFs, cleaning and preprocessing text data, and analyzing content for keywords and patterns. The package also provides a number of configuration options for customizing the behavior of the tools and utilities, making it flexible and easy to use in a variety of different contexts. Whether you need to extract data from PDF documents for data analysis, or analyze PDF content for specific keywords or patterns, this package provides the tools you need to get the job done quickly and efficiently.

Dependency Overview

Python Dependency Install

This project leverages packages

nltk == 3.8.1 
spacy == 3.5.2
PyMuPDF == 1.21.1 
camelot-py == 0.11.0 
opencv-python == 4.7.0.72 
ghostscript == 0.7

manually install supporting binaries

spacy dependencies

python -m spacy download en_core_web_sm

camelot dependencies

• https://camelot-py.readthedocs.io/en/master/user/install-deps.html#install-deps

Ubuntu

$ apt install ghostscript python3-tk

MacOS

$ brew install ghostscript tcl-tk

windows dependency installations

• Install ghostscript: https://ghostscript.com/releases/gsdnld.html
• Install Tinker: https://platform.activestate.com/activestate/activetcl-8.6/auto-fork?_ga=2.93217438.2024444162.1679060315-1994225326.1678735799

Configuration

• configuration file should be placed in the root of the project and named 'extractlib.config.json'

Example 'extractlib.config.json' File

this file should be located in the root of your project

{
  "std_out_logging": true,
  "supported_file_types": [".pdf"],
  "invalid_content_regexs": ["X{2,}"],
  "stop_words": [
    "na",
    "dependent",
    "address",
    "plans",
    "network",
    "nonnetwork",
    "additional",
    "covered"
  ],
  "keywords": {
    "dental": 5,
    "vision": 5,    
    "life": 5,
    "disability": 5
  },
  "keyword_synonyms": {
    "dental": ["orthodontic", "Endo", "Perio", "Oral"],
    "vision": ["eye", "vision", "lens", "lenses", "contact", "contacts"],
    "life": [
      "accident",
      "critical",
      "illness",
      "accidental",
      "dismember",
      "AD&D"
    ],
    "disability": []
  },
  "word_min_length": 3
}

access config variables

from extractlib.settings import config

print(json.dump(config.config_raw, indent=4))

Example implementation

from extractlib.document.process import process_document
import json

def main(file: str):
    result = process_document(file,  exclude_pages=[2,3], use_multithreading=False, split_pages_output_dir='./output', delete_split_pages=False)
    # Save the HTML content to a temporary file
    with open('temp.json', 'w') as f:
        json.dump(result, f, indent=4)


if __name__ == '__main__':

    # get working directory
    import os
    target_dir = os.path.dirname(os.path.abspath(__file__))
    main(f'{target_dir}/_testdata/PDF.pdf')