dbpedia-entity-lookup

dbpedia-entity-lookup

1. Overview
2. Installation
3. Use
4. API
5. Development

Overview

Finds entities (people, places, organizations and titles) in dbpedia. Meant to be used with cwrc-public-entity-dialogs where it runs in the browser.

Although it will not work in node.js as-is, it does use the Fetch API for http requests, and so could likely therefore use a browser/node.js compatible fetch implementation like: isomorphic-fetch.

SPARQL

dbpedia supports sparql, but SPARQL has limited support for full text search. The expectation with SPARQL mostly seems to be that you know exactly what you are matching on So, a query that exactly details the label works fine:

SELECT DISTINCT ?s WHERE { ?s ?label "The Rolling Stones"@en . ?s ?p ?o }

We'd like, however, to match with full text search, so we can match on partial strings, variant spellings, etc.
Just in the simple case above, for example, someone searching for The Rolling Stones would have to fully specify 'The Rolling Stones' and not just 'Rolling Stones'. If they left out 'The' then their query won't return the result.

There is a SPARQL CONTAINS operator that can be used within a FILTER, and that matches substrings, which is better, and CONTAINS does work with dbpedia, but the (admittedly limited) testing we did found it very slow.

There is at least one alternative to CONTAINS - REGEX - but as described here: https://www.cray.com/blog/dont-use-hammer-screw-nail-alternatives-regex-sparql/ REGEX has even worse performance than CONTAINS.

Dbpedia does, however, provide a search service:

https://github.com/dbpedia/lookup

a hosted version of which can be accessed at:

http://lookup.dbpedia.org/api/search/KeywordSearch?QueryClass=place&MaxResults=5&QueryString=berlin

(Note that we set an accept header of application/json so we get back json and not the default xml.)

The hosted Dbpedia lookup does not, however, have an HTTPS endpoint. And so, we proxy our calls to the dbpedia lookup through own server:

https://lookup.services.cwrc.ca/dbpedia

to thereby allow the CWRC-Writer to make HTTPS calls to the dbpedia lookup.
We can’t make plain HTTP calls from the CWRC-Writer because the CWRC-Writer may only be loaded over HTTPS, and any page loaded with HTTPS is not allowed (by many browsers) to make HTTP AJAX calls.

We also rewrite the uri that is returned in the dbpedia results so that it uses another cwrc proxy:

https://dbpedia.lookup.services.cwrc.ca

which proxies calls to

http://dbpedia.org

Installation

npm i dbpedia-entity-lookup -S

Use

const dbpediaLookup = require('dbpedia-entity-lookup');

API

findPerson(query)

findPlace(query)

findOrganization(query)

findTitle(query)

where the 'query' argument is an object:

{
    entity:  The name of the thing the user wants to find.
    options: TBD 
}

and all find* methods return promises that resolve to an object like the following:

{
   Object
   
   description: "Paris is the capital and largest city of France. It is situated on the river Seine, in northern France, at the heart of the Île-de-Franc…"
   
   id: "http://dbpedia.org/resource/Paris"
   
   name: "Paris"
   
   nameType: "place"
   
   originalQueryString: "paris"
   
   repository: "dbpedia"
   
   uri: "http://dbpedia.org/resource/Paris"
   
   uriForDisplay: "https://dbpedia.lookup.services.cwrc.ca/resource/Paris"
    
}

There are a further four methods that are mainly made available to facilitate testing (to make it easier to mock calls to the dbpedia service):

getPersonLookupURI(query)

getPlaceLookupURI(query)

getOrganizationLookupURI(query)

getTitleLookupURI(query)

where the 'query' argument is the entity name to find and the methods return the dbpedia URL that in turn returns results for the query.

Development

CWRC-Writer-Dev-Docs describes general development practices for CWRC-Writer GitHub repositories, including this one.

Testing

The code in this repository is intended to run in the browser, and so we use browser-run to run browserified tape tests directly in the browser.

We decorate tape with tape-promise to allow testing with promises and async methods.

Mocking

We use fetch-mock to mock http calls (which we make using the Fetch API rather than XMLHttpRequest).

We use sinon fake timers to test our timeouts, without having to wait for the timeouts.

Code Coverage

We generate code coverage by instrumenting our code with istanbul before browser-run runs the tests, then extract the coverage (which istanbul writes to the global object, i.e., the window in the browser), format it with istanbul, and finally report (Travis actually does this for us) to codecov.io

Transpilation

We use babelify and babel-plugin-istanbul to compile our code, tests, and code coverage with babel

Continuous Integration

We use Travis.

Note that to allow our tests to run in Electron on Travis, the following has been added to .travis.yml:

addons:
  apt:
    packages:
      - xvfb
install:
  - export DISPLAY=':99.0'
  - Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
  - npm install

Release

We follow SemVer, which Semantic Release makes easy.
Semantic Release also writes our commit messages, sets the version number, publishes to NPM, and finally generates a changelog and a release (including a git tag) on GitHub.