PDF file parser that converts PDF binaries to JSON and text, powered by porting a fork of PDF.JS to Node.js
The pdf2json npm package is a utility that allows you to convert PDF files into JSON format. This can be particularly useful for extracting and processing data from PDF documents programmatically.
Convert PDF to JSON
This feature allows you to convert a PDF file into a JSON object. The code sample demonstrates how to load a PDF file, parse it, and save the resulting JSON data to a file.
const fs = require('fs');
const PDFParser = require('pdf2json');
let pdfParser = new PDFParser();
pdfParser.on('pdfParser_dataError', errData => console.error(errData.parserError));
pdfParser.on('pdfParser_dataReady', pdfData => {
fs.writeFile('./output.json', JSON.stringify(pdfData), (err) => {
if (err) throw err;
console.log('PDF data has been converted to JSON and saved to output.json');
});
});
pdfParser.loadPDF('./sample.pdf');Extract Text from PDF
This feature allows you to extract text content from a PDF file. The code sample demonstrates how to parse a PDF file and extract the text content, then save it to a text file.
const fs = require('fs');
const PDFParser = require('pdf2json');
let pdfParser = new PDFParser();
pdfParser.on('pdfParser_dataError', errData => console.error(errData.parserError));
pdfParser.on('pdfParser_dataReady', pdfData => {
const text = pdfData.formImage.Pages.map(page => page.Texts.map(text => decodeURIComponent(text.R[0].T)).join(' ')).join('\n');
fs.writeFile('./output.txt', text, (err) => {
if (err) throw err;
console.log('Text has been extracted from PDF and saved to output.txt');
});
});
pdfParser.loadPDF('./sample.pdf');The pdf-parse package is another utility for parsing PDF files and extracting text content. It is simpler to use compared to pdf2json and provides a straightforward API for extracting text and metadata from PDF files.
The pdf-lib package is a more comprehensive library for working with PDF files. It allows you to create, modify, and extract content from PDF documents. While it offers more features than pdf2json, it may be more complex to use for simple PDF to JSON conversion tasks.
The pdfjs-dist package is a distribution of the PDF.js library, which is a popular library for rendering PDF documents in web applications. It also provides functionality for extracting text and other content from PDF files. It is more focused on rendering PDFs in the browser but can be used for extraction purposes as well.
pdf2json is a node.js module converts binary PDF to JSON and text. Built with pdf.js, it extracts text content and interactive form elements for server-side processing and command-line use.
npm i pdf2json
Or, install it globally:
npm i pdf2json -g
To update with latest version:
npm update pdf2json -g
To Run in RESTful Web Service or as command line Utility
After install, run command line:
npm test
pretest step builds bundles and source maps for both ES Module and CommonJS, output to ./dist directory. The Jest test suit is defined in ./test/_test_.cjs with commonJS, test run will also cover parse-r and parse-fd with ES Modules via command line.
The default Jest test suits are essential tests for all PRs. But it only covers a portion of all testing PDFs, for more broader coverage, run:
npm run test:forms
It'll scan and parse 260 PDF AcroForm files under ./test/pdf, runs with -s -t -c -m command line options, generates primary output JSON, additional text content JSON, form fields JSON and merged text file for each PDF. It usually takes ~20s in my MacBook Pro to complete, check ./test/target/ for outputs.
update on 4/27/2024: parsing 260 PDFs by npm run test:forms on M2 Mac takes 7~8s
To run Jest test suits with commonJS bundle only
npm run test:jest
After install, run command line:
npm run test:misc
It'll scan and parse all PDF files under ./test/pdf/misc, also runs with -s -t -c -m command line options, generates primary output JSON, additional text content JSON, form fields JSON and merged text JSON file for 15 PDF fields, 12 are expected to success while the other three's exceptions are expected to catch with stack trace for:
pdf/misc/i200_test.pdfpdf/misc/i43_encrypted.pdfpdf/misc/i243_problem_file_anon.pdfAfter install, run command line:
npm run parse-r
It scans 165 PDF files under ./test/pdf/fd/form/, parses with Stream API, then generates output to ./test/target/fd/form/.
More test scripts with different command line options can be found at package.json.
For CI/CD, you probably would like to disable unnecessary logs for unit testing.
The code has two types of logs:
To disable the first type, you could mock the console.log and console.warn APIs, but to disable the second one, you can either set the env variable PDF2JSON_DISABLE_LOGS to "1", passes -s (silect) in command line, or pass in VERBOSITY_LEVEL to be 0 when invoking PDFParser.loadPDF (ex. src/cli/p2jcli.js).
import fs from "fs";
import PDFParser from "pdf2json";
const pdfParser = new PDFParser();
pdfParser.on("pdfParser_dataError", (errData) =>
console.error(errData.parserError)
);
pdfParser.on("pdfParser_dataReady", (pdfData) => {
fs.writeFile(
"./pdf2json/test/F1040EZ.json",
JSON.stringify(pdfData),
(data) => console.log(data)
);
});
pdfParser.loadPDF("./pdf2json/test/pdf/fd/form/F1040EZ.pdf");
Or, call directly with buffer:
fs.readFile(pdfFilePath, (err, pdfBuffer) => {
if (!err) {
pdfParser.parseBuffer(pdfBuffer);
}
});
Or, use more granular page level parsing events (v2.0.0)
pdfParser.on("readable", (meta) => console.log("PDF Metadata", meta));
pdfParser.on("data", (page) =>
console.log(page ? "One page paged" : "All pages parsed", page)
);
pdfParser.on("error", (err) => console.error("Parser Error", err));
import fs from "fs";
import PDFParser from "pdf2json";
const pdfParser = new PDFParser(this, 1);
pdfParser.on("pdfParser_dataError", (errData) =>
console.error(errData.parserError)
);
pdfParser.on("pdfParser_dataReady", (pdfData) => {
fs.writeFile(
"./pdf2json/test/F1040EZ.content.txt",
pdfParser.getRawTextContent(),
() => {
console.log("Done.");
}
);
});
pdfParser.loadPDF("./pdf2json/test/pdf/fd/form/F1040EZ.pdf");
import fs from "fs";
import PDFParser from "pdf2json";
const pdfParser = new PDFParser();
pdfParser.on("pdfParser_dataError", (errData) =>
console.error(errData.parserError)
);
pdfParser.on("pdfParser_dataReady", (pdfData) => {
fs.writeFile(
"./pdf2json/test/F1040EZ.fields.json",
JSON.stringify(pdfParser.getAllFieldsTypes()),
() => {
console.log("Done.");
}
);
});
pdfParser.loadPDF("./pdf2json/test/pdf/fd/form/F1040EZ.pdf");
Alternatively, you can pipe input and output streams: (requires v1.1.4)
import fs from "fs";
import PDFParser from "pdf2json";
const inputStream = fs.createReadStream(
"./pdf2json/test/pdf/fd/form/F1040EZ.pdf",
{ bufferSize: 64 * 1024 }
);
const outputStream = fs.createWriteStream(
"./pdf2json/test/target/fd/form/F1040EZ.json"
);
inputStream
.pipe(new PDFParser())
.pipe(new StringifyStream())
.pipe(outputStream);
With v2.0.0, last line above changes to
inputStream
.pipe(this.pdfParser.createParserStream())
.pipe(new StringifyStream())
.pipe(outputStream);
For additional output streams support:
//private methods
#generateMergedTextBlocksStream() {
return new Promise( (resolve, reject) => {
const outputStream = ParserStream.createOutputStream(this.outputPath.replace(".json", ".merged.json"), resolve, reject);
this.pdfParser.getMergedTextBlocksStream().pipe(new StringifyStream()).pipe(outputStream);
});
}
#generateRawTextContentStream() {
return new Promise( (resolve, reject) => {
const outputStream = ParserStream.createOutputStream(this.outputPath.replace(".json", ".content.txt"), resolve, reject);
this.pdfParser.getRawTextContentStream().pipe(outputStream);
});
}
#generateFieldsTypesStream() {
return new Promise( (resolve, reject) => {
const outputStream = ParserStream.createOutputStream(this.outputPath.replace(".json", ".fields.json"), resolve, reject);
this.pdfParser.getAllFieldsTypesStream().pipe(new StringifyStream()).pipe(outputStream);
});
}
#processAdditionalStreams() {
const outputTasks = [];
if (PROCESS_FIELDS_CONTENT) {//needs to generate fields.json file
outputTasks.push(this.#generateFieldsTypesStream());
}
if (PROCESS_RAW_TEXT_CONTENT) {//needs to generate content.txt file
outputTasks.push(this.#generateRawTextContentStream());
}
if (PROCESS_MERGE_BROKEN_TEXT_BLOCKS) {//needs to generate json file with merged broken text blocks
outputTasks.push(this.#generateMergedTextBlocksStream());
}
return Promise.allSettled(outputTasks);
}
Note, if primary JSON parsing has exceptions, none of additional stream will be processed. See p2jcmd.js for more details.
events:
alternative events: (v2.0.0)
start to parse PDF file from specified file path asynchronously:
function loadPDF(pdfFilePath);
If failed, event "pdfParser_dataError" will be raised with error object: {"parserError": errObj}; If success, event "pdfParser_dataReady" will be raised with output data object: {"formImage": parseOutput}, which can be saved as json file (in command line) or serialized to json when running in web service. note: "formImage" is removed from v2.0.0, see breaking changes for details.
function getRawTextContent();
returns text in string.
function getAllFieldsTypes();
returns an array of field objects.
Current parsed data has four main sub objects to describe the PDF document.
'Transcoder': pdf2json version number
'Agency': the main text identifier for the PDF document. If Id.AgencyId present, it'll be same, otherwise it'll be set as document title; (deprecated since v2.0.0, see notes below)
'Id': the XML meta data that embedded in PDF document (deprecated since v2.0.0, see notes below)
./test/pdf/fd/form/F1040.pdf, full metadata is:Meta: {
PDFFormatVersion: '1.7',
IsAcroFormPresent: true,
IsXFAPresent: false,
Author: 'SE:W:CAR:MP',
Subject: 'U.S. Individual Income Tax Return',
Creator: 'Adobe Acrobat Pro 10.1.8',
Producer: 'Adobe Acrobat Pro 10.1.8',
CreationDate: "D:20131203133943-08'00'",
ModDate: "D:20140131180702-08'00'",
Metadata: {
'xmp:modifydate': '2014-01-31T18:07:02-08:00',
'xmp:createdate': '2013-12-03T13:39:43-08:00',
'xmp:metadatadate': '2014-01-31T18:07:02-08:00',
'xmp:creatortool': 'Adobe Acrobat Pro 10.1.8',
'dc:format': 'application/pdf',
'dc:description': 'U.S. Individual Income Tax Return',
'dc:creator': 'SE:W:CAR:MP',
'xmpmm:documentid': 'uuid:4d81e082-7ef2-4df7-b07b-8190e5d3eadf',
'xmpmm:instanceid': 'uuid:7ea96d1c-3d2f-284a-a469-f0f284a093de',
'pdf:producer': 'Adobe Acrobat Pro 10.1.8',
'adhocwf:state': '1',
'adhocwf:version': '1.1'
}
}
Each page object within 'Pages' array describes page elements and attributes with 5 main fields:
v0.4.5 added support when fields attributes information is defined in external xml file. pdf2json will always try load field attributes xml file based on file name convention (pdfFileName.pdf's field XML file must be named pdfFileName_fieldInfo.xml in the same directory). If found, fields info will be injected.
Same reason to having "HLines" and "VLines" array in 'Page' object, color and style dictionary will help to reduce the size of payload when transporting the parsing object over the wire. This dictionary data contract design will allow the output just reference a dictionary key , rather than the actual full definition of color or font style. It does require the client of the payload to have the same dictionary definition to make sense out of it when render the parser output on to screen.
const kColors = [
"#000000", // 0
"#ffffff", // 1
"#4c4c4c", // 2
"#808080", // 3
"#999999", // 4
"#c0c0c0", // 5
"#cccccc", // 6
"#e5e5e5", // 7
"#f2f2f2", // 8
"#008000", // 9
"#00ff00", // 10
"#bfffa0", // 11
"#ffd629", // 12
"#ff99cc", // 13
"#004080", // 14
"#9fc0e1", // 15
"#5580ff", // 16
"#a9c9fa", // 17
"#ff0080", // 18
"#800080", // 19
"#ffbfff", // 20
"#e45b21", // 21
"#ffbfaa", // 22
"#008080", // 23
"#ff0000", // 24
"#fdc59f", // 25
"#808000", // 26
"#bfbf00", // 27
"#824100", // 28
"#007256", // 29
"#008000", // 30
"#000080", // Last + 1
"#008080", // Last + 2
"#800080", // Last + 3
"#ff0000", // Last + 4
"#0000ff", // Last + 5
"#008000", // Last + 6
"#000000", // Last + 7
];
const kFontFaces = [
"QuickType,Arial,Helvetica,sans-serif", // 00 - QuickType - sans-serif variable font
"QuickType Condensed,Arial Narrow,Arial,Helvetica,sans-serif", // 01 - QuickType Condensed - thin sans-serif variable font
"QuickTypePi", // 02 - QuickType Pi
"QuickType Mono,Courier New,Courier,monospace", // 03 - QuickType Mono - san-serif fixed font
"OCR-A,Courier New,Courier,monospace", // 04 - OCR-A - OCR readable san-serif fixed font
"OCR B MT,Courier New,Courier,monospace", // 05 - OCR-B MT - OCR readable san-serif fixed font
];
const kFontStyles = [
// Face Size Bold Italic StyleID(Comment)
// ----- ---- ---- ----- -----------------
[0, 6, 0, 0], //00
[0, 8, 0, 0], //01
[0, 10, 0, 0], //02
[0, 12, 0, 0], //03
[0, 14, 0, 0], //04
[0, 18, 0, 0], //05
[0, 6, 1, 0], //06
[0, 8, 1, 0], //07
[0, 10, 1, 0], //08
[0, 12, 1, 0], //09
[0, 14, 1, 0], //10
[0, 18, 1, 0], //11
[0, 6, 0, 1], //12
[0, 8, 0, 1], //13
[0, 10, 0, 1], //14
[0, 12, 0, 1], //15
[0, 14, 0, 1], //16
[0, 18, 0, 1], //17
[0, 6, 1, 1], //18
[0, 8, 1, 1], //19
[0, 10, 1, 1], //20
[0, 12, 1, 1], //21
[0, 14, 1, 1], //22
[0, 18, 1, 1], //23
[1, 6, 0, 0], //24
[1, 8, 0, 0], //25
[1, 10, 0, 0], //26
[1, 12, 0, 0], //27
[1, 14, 0, 0], //28
[1, 18, 0, 0], //29
[1, 6, 1, 0], //30
[1, 8, 1, 0], //31
[1, 10, 1, 0], //32
[1, 12, 1, 0], //33
[1, 14, 1, 0], //34
[1, 18, 1, 0], //35
[1, 6, 0, 1], //36
[1, 8, 0, 1], //37
[1, 10, 0, 1], //38
[1, 12, 0, 1], //39
[1, 14, 0, 1], //40
[1, 18, 0, 1], //41
[2, 8, 0, 0], //42
[2, 10, 0, 0], //43
[2, 12, 0, 0], //44
[2, 14, 0, 0], //45
[2, 12, 0, 0], //46
[3, 8, 0, 0], //47
[3, 10, 0, 0], //48
[3, 12, 0, 0], //49
[4, 12, 0, 0], //50
[0, 9, 0, 0], //51
[0, 9, 1, 0], //52
[0, 9, 0, 1], //53
[0, 9, 1, 1], //54
[1, 9, 0, 0], //55
[1, 9, 1, 0], //56
[1, 9, 1, 1], //57
[4, 10, 0, 0], //58
[5, 10, 0, 0], //59
[5, 12, 0, 0], //60
];
v2.0.0: to access these dictionary programactically, do either
import { kColors, kFontFaces, kFontStyles } from "./lib/pdfconst.js"; // <-- pre 3.1.0
import { kColors, kFontFaces, kFontStyles } from "pdf2json"; // <-- since 3.1.0
or via public static getters of PDFParser:
console.dir(PDFParser.colorDict);
console.dir(PDFParser.fontFaceDict);
console.dir(PDFParser.fontStyleDict);
v0.1.5 added interactive forms element parsing, including text input, radio button, check box, link button and drop down list.
Interactive forms can be created and edited in Acrobat Pro for AcroForm, or in LiveCycle Designer ES for XFA forms. Current implementation for buttons only supports "link button": when clicked, it'll launch a URL specified in button properties. Examples can be found at f1040ezt.pdf file under test/data folder.
All interactive form elements parsing output will be part of corresponding 'Page' object where they belong to, radio buttons and check boxes are in 'Boxsets' array while all other elements objects are part of 'Fields' array.
Each object with in 'Boxset' can be either checkbox or radio button, the only difference is that radio button object will have more than one element in 'boxes' array, it indicates it's a radio button group. The following sample output illustrate one checkbox ( Id: F8888 ) and one radio button group ( Id: ACC ) in the 'Boxsets' array:
Boxsets: [
{//first element, check box
boxes: [ //only one box object object in this array
{
x: 47,
y: 40,
w: 3,
h: 1,
style: 48,
TI: 39,
AM: 4,
id: {
Id: "F8888",
},
T: {
Name: "box"
}
}
],
id: {
Id: "A446",
}
},//end of first element
{//second element, radio button group
boxes: [// has two box elements in boxes array
{
x: 54,
y: 41,
w: 3,
h: 1,
style: 48,
TI: 43,
AM: 132,
id: {
Id: "ACCC",
},
T: {
Name: "box"
}
},
{
x: 67,
y: 41,
w: 3,
h: 1,
style: 48,
TI: 44,
AM: 132,
id: {
Id: "ACCS",
EN: 0
},
T: {
Name: "box"
}
}
],
id: {
Id: "ACC",
EN: 0
}
}//end of second element
] //end of Boxsets array
'Fields' array contains parsed object for text input (Name: 'alpha'), drop down list (Name: 'apha', but has 'PL' object which contains label array in 'PL.D' and value array in 'PL.V'), link button (Name: 'link', linked URL is in 'FL.form.Id' field). Some examples:
Text input box example:
{
style: 48,
T: {
Name: "alpha",
TypeInfo: { }
},
id: {
Id: "p1_t40",
EN: 0
},
TU: "alternative text", //for accessibility, added only when available from PDF stream. (v0.3.6).
TI: 0,
x: 6.19,
y: 5.15,
w: 30.94,
h: 0.85,
V: "field value" //only available when the text input box has value
},
Note: v0.7.0 extends TU (Alternative Text) to all interactive fields to better support accessibility.
Drop down list box example:
{
x: 60,
y: 11,
w: 4,
h: 1,
style: 48,
TI: 13,
AM: 388,
mxL: 2,
id: {
Id: "ST",
EN: 0
},
T: {
Name: "alpha",
TypeInfo: {
}
},
PL: {
V: [
"",
"AL",
"AK"
],
D: [
"%28no%20entry%29",
"Alabama",
"Alaska"
]
}
}
Link button example:
{
style: 48,
T: {
Name: "link"
},
FL: {form: {Id:"http://www.github.com"},
id: {
Id: "quad8",
EN: 0
},
TI: 0,
x: 52.35,
y: 28.35,
w: 8.88,
h: 0.85
}
v0.2.2 added support for "field attribute mask", it'd be common for all fields, form author can set it in Acrobat Pro's Form Editing mode: if a field is ReadOnly, it's AM field will be set as 0x00000400, otherwise AM will be set as 0.
Another supported field attributes is "required": when form author mark a field is "required" in Acrobat, the parsing result for 'AM' will be set as 0x00000010.
"Read-Only" filed attribute mask example:
{
style: 48,
T: {
Name: "alpha",
TypeInfo: { }
},
id: {
Id: "p1_t40",
EN: 0
},
TI: 0,
AM: 1024, //If (AM & 0x00000400) set, it indicates this is a read-only filed
x: 6.19,
y: 5.15,
w: 30.94,
h: 0.85
}
v2.X.X added support for the signature form element (Name: 'signature'). If the field has been signed, the 'Sig' property will be present, and will contain any of the following signature details if available:
Signature example:
{
style: 48,
T: {
Name: "signature",
TypeInfo: {}
},
id: {
Id: "SignatureFormField_1",
EN: 0
},
TI: 0,
AM: 16,
x: 5.506,
y: 31.394,
w: 14.367,
h: 4.241,
Sig: {
Name: "Signer Name",
M: "2022-03-15T19:17:34-04:00"
}
}
v0.1.8 added text input field formatter types detection for
v0.3.9 added "arbitrary mask" (in "special" format category) support, the input field format type is "mask" and the mask string is added as "MV", its value can be found at Format => Special => Arbitrary Mask in Acrobat; Some examples of "mask" format including:
Additionally, the "arbitrary mask" length is extended from 1 characters to 64 characters. And when the mask has only one character, it has the following meanings:
v0.4.1 added more date format detection, these formats are set in Acrobat's field's Properties => Format => Date => Custom:
Types above are detected only when the widget field type is "Tx" and the additional-actions dictionary 'AA' is set. Like what you see, not all pre-defined formatters and special formatters are supported, if you need more support, you can extend the 'processFieldAttribute' function in core.js file.
For the supported types, the result data is set to the field item's T object. Example of a 'number' field in final json output:
{
style: 48,
T: {
Name: "number",
TypeInfo: { }
},
id: {
Id: "FAGI",
EN: 0
},
TI: 0,
x: 68.35,
y: 22.43,
w: 21.77,
h: 1.08
},
Another example of 'date' field:
{
style: 48,
T: {
Name: "date",
TypeInfo: { }
},
id: {
Id: "Your Birth Date",
EN: 0
},
TI: 0,
x: 33.43,
y: 20.78,
w: 5.99,
h: 0.89
},
v0.1.11 added text style information in addition to style dictionary. As we discussed earlier, the idea of style dictionary is to make the parsing result payload to be compact, but I found out the limited dictionary entries for font (face, size) and style (bold, italic) can not cover majority of text contents in PDFs, because of some styles are matched with closest dictionary entry, the client rendering will have mis-aligned, gapped or overlapped text. To solve this problem, pdf2json v0.1.11 extends the dictionary approach, all previous dictionary entries stay the same, but parsing result will not try to match to a closest style entry, instead, all exact text style will be returned in a TS filed.
When the actual text style doesn't match any pre-defined style dictionary entry, the text style ID (S filed) will be set as -1. The actual text style will be set in a new field (TS) with or without a matched style dictionary entry ID. This means, if your client renderer works with pdf2json v0.1.11 and later, style dictionary ID can be ignored. Otherwise, previous client renderer can still work with style dictionary ID.
The new TS filed is an Array with format as:
For example, the following is a text block data in the parsing result:
{
x: 7.11,
y: 2.47,
w: 1.6,
clr: 0,
A: "left",
R: [
{
T: "Modesty%20PDF%20Parser%20NodeJS",
S: -1,
TS: [0, 15, 1, 0]
}
]
},
The text is "Modesty PDF Parser NodeJS", text style dictionary entry ID is -1 (S field, meaning no match), and its Font Face ID is 0 (TS[0], "QuickType,Arial,Helvetica,sans-serif"), Font Size is 15px (TS[1]), Font weight is bold (TS[2]) and font style is normal (TS[3]).
Note: (v0.3.7) When a color is not in style dictionary, "clr" value will be set to -1. Item's (fills and text) original color in hex string format will be added to "oc" field. In other word, "oc" only exists if and only if "clr" is -1;
V0.1.13 added text rotation value (degree) in the R array's object, if and only if the text rotation angle is not 0. For example, if text is not rotated, the parsed output would be the same as above. When the rotation angle is 90 degree, the R array object would be extended with "RA" field:
{
x: 7.11,
y: 2.47,
w: 1.6,
clr: 0,
A: "left",
R: [
{
T: "Modesty%20PDF%20Parser%20NodeJS",
S: -1,
TS: [0, 15, 1, 0],
RA: 90
}
]
},
pdf.js is designed and implemented to run within browsers that have HTML5 support, it has some dependencies that's only available from browser's JavaScript runtime, including:
In order to run pdf.js in Node.js, we have to address those dependencies and also extend/modify the fork of pdf.js. Here below are some works implemented in this pdf2json module to enable pdf.js running with Node.js:
After the changes and extensions listed above, this pdf2json node.js module will work either in a server environment ( I have a RESTful web service built with resitify and pdf2json, it's been running on an Amazon EC2 instance) or as a standalone command line tool (something similar to the Vows unit tests).
More porting notes can be found at Porting and Extending PDFJS to NodeJS.
This pdf2json module's output does not 100% maps from PDF definitions, some of them is because of time limitation I currently have, some others result from the 'dictionary' concept for the output. Given these known issues or unsupported features in current implementation, it allows me to contribute back to the open source community with the most important features implemented while leaving some improvement space for the future. All un-supported features listed below can be resolved technically some way or other, if your use case really requires them:
v0.1.15 added the capability to run pdf2json as command line tool. It enables the use case that when running the parser as a web service is not absolutely necessary while transcoding local pdf files to json format is desired. Because in some use cases, the PDF files are relatively stable with less updates, even though parsing it in a web service, the parsing result will remain the same json payload. In this case, it's better to run pdf2json as a command line tool to pre-process those pdf files, and deploy the parsing result json files onto web server, client side form renderer can work in the same way as before while eliminating server side process to achieve higher scalability.
This command line utility is added as an extension, it doesn't break previous functionalities of running with a web service context. In my real project, I have a web service written in restify.js to run pdf2json with a RESTful web service interface, I also have the needs to pre-process some local static pdfs through the command line tool without changing the actual pdf2json module code.
To use the command line utility to transcode a folder or a file:
node pdf2json.js -f [input directory or pdf file]
When -f is a PDF file, it'll be converted to json file with the same name and saved in the same directory. If -f is a directory, it'll scan all ".pdf" files within the specified directory to transcode them one by one.
Optionally, you can specify the output directory: -o:
node pdf2json.js -f [input directory or pdf file] -o [output directory]
The output directory must exist, otherwise, it'll exit with an error.
Additionally, you can also use -v or --version to show version number or to display more help info with -h.
v0.2.1 added the ability to run pdf2json directly from the command line without specifying "node" and the path of pdf2json. To run this self-executable in command line, first install pdf2json globally:
npm install pdf2json -g
Then run it in command line:
pdf2json -f [input directory or pdf file]
or
pdf2json -f [input directory or pdf file] -o [output directory]
v0.5.4 added "-s" or "--silent" command line argument to suppress informative logging output. When using pdf2json as a command line tool, the default verbosity is 5 (INFOS). While when running as a web service, default verbosity is 9 (ERRORS). Examples to suppress logging info from command line:
pdf2json -f [input directory or pdf file] -o [output directory] -s
or
pdf2json -f [input directory or pdf file] -o [output directory] --silent
Examples to turn on logging info in web service:
var pdfParser = new PFParser();
...
pdfParser.loadPDF(pdfFilePath, 5);
v0.5.7 added the capability to skip input PDF files if filename begins with any one of "!@#$%^&*()+=[]\';,/{}|":<>?~`.-_ ", usually these files are created by PDF authoring tools as backup files.
v0.6.2 added "-t" command line argument to generate fields json file in addition to parsed json. The fields json file will contain one Array which contains fieldInfo object for each field, and each fieldInfo object will have 4 fields:
Example of fields.json content:
[
{"id":"ADDRCH","type":"alpha","calc":false,"value":"user input data"},
{"id":"FSRB","type":"radio","calc":false,"value":"Single"},
{"id":"APPROVED","type":"alpha","calc":true,"value":"Approved Form"}
...
]
The fields.json output can be used to validate fields IDs with other data source, and/or to extract data value from user submitted PDFs.
v0.6.8 added "-c" or "--content" command line argument to extract raw text content from PDF. It'll be a separated output file named as (pdf_file_name).content.txt. If all you need is the textual content of the PDF, "-c" essentially converts PDF to text, of cause, all formatting and styling will be lost.
It takes less than 1 minutes for pdf2json to parse 261 PDFs under test/pdf directory. Usually, it takes about 40 seconds or so to parses all of them. Besides the primary JSON for each PDF, it also generates text content JSON and form fields JSON file (by -c and -t parameters) for further testing.
The 265 PDFs are all fill-able tax forms from government agencies for tax year 2013, including 165 federal forms, 23 efile instructions and 9 other state tax forms.
Shell script is current driver for unit test. To parse one agency's PDFs, run the command line:
cd test
sh p2f.one.sh [2_character_agency_name]
For example, to parse and generate all 165 federal forms together with text content and forms fields:
cd test
sh p2f.one.sh fd
To parse and generate all VA forms together with text content and forms fields:
cd test
sh p2f.one.sh va
Additionally, to parse all 261 PDFs from commandline:
cd test
sh p2f.forms.sh
Or, from npm scripts:
npm test
Some testing PDFs are provided by bug reporters, like the "unsupported encryption" (#43), "read property num from undefined" (#26), and "excessive line breaks in text content" (#28), their PDFs are all stored in test/pdf/misc directory. To run tests against these community contributed PDFs, run commandline:
npm run-script test-misc
If you have an early version of pdf2json, please remove your local node_modules directory and re-run npm install to upgrade to pdf2json@1.0.x.
v1.x.x upgraded dependency packages, removed some unnecessary dependencies, started to assumes ES6 / ES2015 with node ~v4.x. More PDFs are added for unit testing.
Note: pdf2json has been in production for over 3 years, it's pretty reliable and solid when parsing hundreds (sometimes tens of thousands) of PDF forms every day, thanks to everybody's help.
Starting v1.0.3, I'm trying to address a long over due annoying problem on broken text blocks. It's the biggest problem that hinders the efficiency of PDF content creation in our projects. Although the root cause lies in the original PDF streams, since the client doesn't render JSON character by character, it's a problem often appears in final rendered web content. We had to work around it by manually merge those text blocks. With the solution in v1.0.x, the need for manual text block merging is greatly reduced.
The solution is to put to a post-parsing process stage to identify and auto-merge those adjacent blocks. It's not ideal, but works in most of my tests with those 261 PDFs underneath test directory.
The auto merge solution still needs some fine tuning, I keep it as an experimental feature for now, it's off by default, can be turned on by "-m" switch in command line.
In order to support this auto merging capability, text block objects have an additional "sw" (space width of the font) property together with x, y, clr and R. If you have a more effective usage of this new property for merging text blocks, please drop me a line.
Breaking Changes:
v1.1.4 unified event data structure: only when you handle these top level events, no change if you use commandline
v1.0.8 fixed issue 27, it converts x coordinate with the same ratio as y, which is 24 (96/4), rather than 8.7 (96/11), please adjust client renderer accordingly when position all elements' x coordinate.
v2.0.0 output data field, Agency and Id are replaced with Meta, JSON of the PDF's full metadata. (See above for details). Each page object also added Width property besides Height.
v3.0.0 converted commonJS to ES Modules, plus dependency update and other minor bug fixes. Please update your project configuration file to enable ES Module before upgrade, ex., in tsconfig.json, set "compilerOptions":{"module":"ESNext"}
Meta as root property, unnecessary "formImage" is removed from v2.0.0, also Width is move from root to each page object under Pages.npm run parse-r, plus new events are added to PDF.js, including readable, data, end, error. These new Readable Stream like events can be optional replacement for customed events (pdfjs_parseDataReady, and pdfjs_parseDataError). It offers more granular data chunk flow control, like readable with Meta, data sequence for each PDF page result, instead of pdfjs_parseDataReady combines all pages in one shot. See ./lib/parserstream.js for more detailsclr index value is valid, then oc (original color) field is removed.PDFParser is no longer the default export, it's a named export that requires changes to import statement.npm ci and npm test$ nodejs --version
v0.10.22
sudo rm -f /usr/sbin/node
sudo ln -s /usr/bin/nodejs /usr/sbin/node
$ which node
/usr/sbin/node
$ node --version
v4.5.0
$ npm install -g pdf2json
npm http GET https://registry.npmjs.org/pdf2json
npm http 304 https://registry.npmjs.org/pdf2json
/usr/bin/pdf2json -> /usr/lib/node_modules/pdf2json/bin/pdf2json
pdf2json@0.6.1 /usr/lib/node_modules/pdf2json
$ which pdf2json
/usr/bin/pdf2json
$ pdf2json --version
0.6.2
More info can be found at Restful Web Service for pdf2json.
Participating in this project, you are expected to honor open code of conduct.
Licensed under the Apache License Version 2.0.
I'm currently running this project in my spare time. Thanks all for your stars and supports.
FAQs
PDF file parser that converts PDF binaries to JSON and text, powered by porting a fork of PDF.JS to Node.js
The npm package pdf2json receives a total of 112,186 weekly downloads. As such, pdf2json popularity was classified as popular.
We found that pdf2json demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Socket and Seal Security collaborate to fix a critical npm overrides bug, resolving a three-year security issue in the JavaScript ecosystem's most popular package manager.
Security News
TypeScript is porting its compiler to Go, delivering 10x faster builds, lower memory usage, and improved editor performance for a smoother developer experience.
Research
Security News
The Socket Research Team has discovered six new malicious npm packages linked to North Korea’s Lazarus Group, designed to steal credentials and deploy backdoors.