Security News
Introducing the Socket Python SDK
The initial version of the Socket Python SDK is now on PyPI, enabling developers to more easily interact with the Socket REST API in Python projects.
grobid_tei_xml
: Python parser and transforms for GROBID-flavor TEI-XMLThis is a simple python library for parsing the TEI-XML structured documents returned by GROBID, a machine learning tool for extracting text and bibliographic metadata from research article PDFs.
TEI-XML is a standard format, and there exist other libraries to parse entire documents and work with annotated text. This library is focused specifically on extracting "header" metadata from document (eg, title, authors, journal name, volume, issue), content in flattened text form (full abstract and body text as single strings, for things like search indexing), and structured citation metadata.
grobid_tei_xml
works with Python 3, using only the standard library. It does
not talk to the GROBID HTTP API or read files off disk on it's own, but see
examples below. The library is packaged on pypi.org.
Install using pip
, usually within a virtualenv
:
pip install grobid_tei_xml
The main entry points are the functions process_document_xml(xml_text)
and
process_citation_xml(xml_text)
(or process_citation_list_xml(xml_text)
for
multiple citations), which return python dataclass objects. The helper method
.to_dict()
can be useful for, eg, serializing these objects to JSON.
Read an XML file from disk, parse it, and print to stdout as JSON:
import json
import grobid_tei_xml
xml_path = "./tests/files/small.xml"
with open(xml_path, 'r') as xml_file:
doc = grobid_tei_xml.parse_document_xml(xml_file.read())
print(json.dumps(doc.to_dict(), indent=2))
Use requests
to download a PDF from the web, submit to GROBID (via HTTP API),
parse the TEI-XML response with grobid_tei_xml
, and print some metadata
fields:
import requests
import grobid_tei_xml
pdf_resp = requests.get("https://arxiv.org/pdf/1802.01168v3")
pdf_resp.raise_for_status()
grobid_resp = requests.post(
"https://cloud.science-miner.com/grobid/api/processFulltextDocument",
files={
'input': pdf_resp.content,
'consolidate_Citations': 0,
'includeRawCitations': 1,
},
timeout=60.0,
)
grobid_resp.raise_for_status()
doc = grobid_tei_xml.parse_document_xml(grobid_resp.text)
print("title: " + doc.header.title)
print("authors: " + ", ".join([a.full_name for a in doc.header.authors]))
print("doi: " + str(doc.header.doi))
print("citation count: " + str(len(doc.citations)))
print("abstract: " + doc.abstract)
Use requests
to submit a "raw" citation string to GROBID for extraction,
parse the response with grobid_tei_xml
, and print the structured output to
stdout:
import requests
import grobid_tei_xml
raw_citation = "Kvenvolden K.A. and Field M.E. 1981. Thermogenic hydrocarbons in unconsolidated sediment of Eel River Basin, offshore northern California. AAPG Bulletin 65:1642-1646"
grobid_resp = requests.post(
"https://cloud.science-miner.com/grobid/api/processCitation",
data={
'citations': raw_citation,
'consolidateCitations': 0,
'includeRawCitations': 1,
},
timeout=10.0,
)
grobid_resp.raise_for_status()
citation = grobid_tei_xml.parse_citation_xml(grobid_resp.text)
print(citation)
grobid_client_python
:
Python client and CLI tool for making requests to GROBID via HTTP API. Returns
TEI-XML; could be used with this library (grobid_tei_xml
) for parsing into
python object or, eg, JSON.
s2orc-doc2json: Python library from AI2 which includes a similar Python library for extracting both bibliographic metadata and (structured) full text from GROBID TEI-XML. Has nice features like resolving references to bibliography entry.
delb: more flexible/powerful interface to TEI-XML documents. would be a better tool for working with structured text (body, abstract, etc)
"Parsing TEI XML documents with Python" (2019): blog post about basic parsing of GROBID TEI-XML files into Pandas DataFrames
This library is available under the permissive MIT License. See LICENSE.txt
for a copy.
FAQs
parser and transforms for GROBID-flavor TEI-XML
We found that grobid-tei-xml demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The initial version of the Socket Python SDK is now on PyPI, enabling developers to more easily interact with the Socket REST API in Python projects.
Security News
Floating dependency ranges in npm can introduce instability and security risks into your project by allowing unverified or incompatible versions to be installed automatically, leading to unpredictable behavior and potential conflicts.
Security News
A new Rust RFC proposes "Trusted Publishing" for Crates.io, introducing short-lived access tokens via OIDC to improve security and reduce risks associated with long-lived API tokens.