@the-grid/nikita

Nikita: Content extraction from documents

TODO

• Fix hardcoded temporary directory, clean up after upload
• Test some PDF files
• Add a wrapper graph which takes an object in, enriches then sends out again
• Integrate with AMQP, add as worker in thegrid-apis

Later

• Test how much faster Tika Java API is at XHTML + image extraction over cli tools
• Avoid temporary files for images+html output if/when passing to NoFlo

Setup

Configuration is passed as environment variables:

AMAZON_API_ID: Amazon S3 API identifier
AMAZON_API_TOKEN: Amazon S3 API token/secret
AMAZON_API_REGION: Amazon S3 region, ex: 'us-west-2'
AMAZON_API_BUCKET: Amazon S3 bucket for uploaded files, ex: 'thegrid-user-content'

Design

Separate Heroku worker, integrated into TheGrid APIs.

Inputs:

• URL to s3 backed document (Word,PDF)

Outputs:

• Extracted HTML with img src referring to S3 backend

Notes:

• Tika provides full XHTML document, where as Embed.ly gives only (and we expect)