autocapture

AutoCapture

This project contains Datachecker's AutoCapture tool, that captures images of identity documents (ID/Passport/Driver license). The tool only takes a capture once a document is detected and it passes the quality control.

The tool will be run in the browser and is therefore written in JavaScript.

Trigger mechanism

The tool performs the following checks:

• Is the environment not too dark (under exposure)?
• Is there a document?
• Is the detected document not too far?
• Is the detected document not too close?
• Is the image sharp?

Prerequisites

Please visit Datachecker API documentation.

• Datachecker OAuth Token
• Datachecker SDK Token

OAuth Token

Datachecker uses OAuth authorization. In order to request the SDK token you will need to provide a valid OAuth token in the header.

Example header:

header = {'Authorization': `Bearer ${response.accessToken}`}

This OAuth token can be retrieved with the Datachecker OAuth Token API. The scope "productapi.sdk.read" needs to be present to make use of the SDK token. If this scope is missing you will not be able to retrieve an SDK token.

Example OAuth:

fetch(<BASE_ENDPOINT>+"/oauth/token", {
    method: 'POST',
    body: JSON.stringify({
        "clientId": <CLIENTID>,
        "clientSecret": <CLIENTSECRET>,
        "scopes": [
            "productapi.sdk.read",
        ]
    })
})
.then(response => response.json())

Note: Contact Datachecker for client_id and client_secret.

SDK Token

The SDK is locked. In order to use the SDK in production a token is required. The application can only be started with a valid token. This token is a base64 string. The token can be generated by calling the Datachecker SDK Token API.

Example:

fetch(<BASE_ENDPOINT>+"/sdk/token?customer_reference=<CUSTOMER>&services=AUTO_CAPTURE", {
    method: 'GET',
    headers: {
        'Accept': 'application/json',
        'Content-Type': 'application/json',
        'Authorization': `Bearer <ACCESSTOKEN>`
    }
})
.then(response => response.json())

Steps

1. Request OAUTH Token
2. Put OAuth in header
3. SDK configuration (add SDK token)
4. Run SDK

Configuration

To run this tool, you will need initialise with the following variables.

ATTRIBUTE	FORMAT	DEFAULT VALUE	EXAMPLE	NOTES
APPROVAL	bool	false	false	optional Approval screen after capture as an extra quality check.
CONTAINER_ID	string		"AC_mount"	required div id to mount tool on.
CROP_CARD	bool	false	false	optional Enable cropping of card as output.
CROP_FACE	bool	true	true	optional Enable cropping of face as output.
DEBUG	bool	false	false	optional When debug is true more detailed logs will be visible.
GLARE_LIVE_CHECK	bool	true	true	optional Enable glare detection.
LANGUAGE	string	"nl"	"nl"	required Notifications in specific language.
MODELS_PATH	string	"models/"	"models/"	optional Path referring to models location.
MRZ_SETTINGS	object	see MRZ_SETTINGS	see MRZ_SETTINGS	optional Settings of MRZ scanning.
MRZ	bool	false	false	optional Enable MRZ scanning.
ROI_MODE	string	"landscape-landscape"	portrait-landscape	optional Frame orientation options: "portrait-landscape", "landscape-landscape"
ROOT	string	""	"../"	optional Root location.
TOKEN	string		see SDK Token	required Datachecker SDK token.
onComplete	javascript function		function(data) {console.log(data)}	required Callback function on complete.
onError	javascript function	function(error) {console.log(error)}	function(error) {console.log(error)}	required Callback function on error.
onImage	javascript function	function(data) {console.log(data)}	function(data) {console.log(data)}	optional Callback function on image.
onUserExit	javascript function	function(error) {console.log(error)}	function(error) {window.history.back()}	required Callback function on user exit.

Handling callbacks

Within the application, you can take advantage of four callback functions to enhance the user experience and manage the flow of your process.

Note: When integrating the application into Native Apps using web views, it's essential to adapt and utilize these callback functions according to the conventions and requirements of the native platforms (e.g., iOS, Android). Native app development environments may have specific ways of handling JavaScript callbacks, and you should ensure seamless communication between the web view and the native code.

Example Web (JS):

let AC = new AutoCapture();
AC.init({
    CONTAINER_ID: 'AC_mount',
    LANGUAGE: 'en',
    TOKEN: "<SDK_TOKEN>",
    onComplete: function(data) {
        console.log(data);
    },
    onImage: function(data) {
        console.log(data);
    },
    onError: function(error) {
        console.log(error)
    },
    onUserExit: function(error) {
        console.log(error);
        window.history.back();
    }
});

ATTRIBUTE	FORMAT	DEFAULT VALUE	EXAMPLE	NOTES
onComplete	javascript function		function(data) {console.log(data)}	required Callback that fires when all interactive tasks in the workflow have been completed.
onError	javascript function	function(error) {console.log(error)}	function(error) {console.log(error)}	required Callback that fires when an error occurs.
onImage	javascript function	function(data) {console.log(data)}	function(data) {console.log(data)}	optional Callback that fires when frame succesfully passes quality controls. This callback can be used when you want to process or analyze live frames. (See External-MRZ)
onUserExit	javascript function	function(error) {console.log(error)}	function(error) {window.history.back()}	required Callback that fires when the user exits the flow without completing it.

onComplete

This callback function will be called once all the tasks within the workflow succesfully have been completed. This callback function is required. The data parameter within the function represents the output of the completed process. You can customize this function to handle and display the data as needed.

Example Web (JS):

Within the example below we are logging the output (data) to console.

let AC = new AutoCapture();
AC.init({
    ...,
    onComplete: function(data) {
        console.log(data);
    }
});

onImage

This callback function will be called when a frame succesfully passes quality controls. This callback can be used when you want to process or analyze live frames (See External-MRZ). This callback function is optional. The data parameter within the function represents the frame that succesfully passed the quality controls. It's format is a base64 image string where the Data URI ("data:image/png;base64,") has been taken off.

Example Web (JS):

Within the example below we are logging the output (data) to console.

let AC = new AutoCapture();
AC.init({
    ...,
    onImage: function(data) {
        // data = "iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADg....."
        console.log(data)
    }
});

onError

This callback can be used to alert users when something goes wrong during the process. This callback function is required. The error parameter within the function contains information about the specific error encountered, allowing you to log or display error messages for debugging or user guidance. The errors that are thrown are either known or unknown. The known errors can be found within the Languages dictionary. On the other hand, the unknown errors will be thrown as is.

Example Web (JS):

Within the example below we are logging the output (error) to console.

let AC = new AutoCapture();
AC.init({
    ...,
    onError: function(error) {
        console.log(error)
    }
});

onUserExit

This callback can be used to implement actions like returning users to the previous page or prompting them for confirmation before exiting to ensure they don't lose any unsaved data or work. This callback function is required. The error parameter within the function contains information about the specific error encountered, allowing you to log or display error messages for debugging or user guidance. The error that is thrown is "exit".

Example Web (JS):

Within the example below we are logging the output (error) to console. Finally, we move back one page in the session history with window.history.back().

let AC = new AutoCapture();
AC.init({
    ...,
    onUserExit: function(error) {
        console.log(error);
        window.history.back()
    }
});

Usage/Examples

The tool first needs to be initialised to load all the models. Once its initialised, it will be started.

let AC = new AutoCapture();
AC.init({
    CONTAINER_ID: ...,
    LANGUAGE: ...,
    TOKEN: ...,
    onComplete: ...,
    onError: ...,
    onUserExit: ...
});

To stop the camera and delete the container with its contents the stop function can be called. This function will automatically be called within onComplete, onError and onUserExit thus do not have to be called within your own custom versions of these functions.

...
AC.stop();

Example below:

let AC = new AutoCapture();
AC.init({
    CONTAINER_ID: 'AC_mount',
    LANGUAGE: 'nl',
    TOKEN: "<SDK_TOKEN>",
    onComplete: function(data) {
        console.log(data);
    },
    onError: function(error) {
        console.log(error)
    },
    onUserExit: function(error) {
        console.log(error);
        window.history.back()
    }
});

Requirements

CSS stylesheet.

<link href="css/autocapture.css" rel="stylesheet" type="text/css" />

The meta tags below are required to view the tool properly.

<meta name="viewport" content="user-scalable=no, width=device-width, viewport-fit=cover, initial-scale=1, minimum-scale=1" />
<meta name="apple-mobile-web-app-capable" content="yes" />

Load script.

<script src="js/autocapture.obf.js" type="text/javascript"></script>

Demo

File present under html/examples/index.html

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>AutoCapture</title>
<link href="../css/autocapture.css" rel="stylesheet" type="text/css" />
<meta name="viewport" content="user-scalable=no, width=device-width, initial-scale=1, minimum-scale=1" />
<meta name="apple-mobile-web-app-capable" content="yes" />
</head>

<body>
    <div id="AC_mount"></div>
</body>

<script src="../js/autocapture.obf.js" type="text/javascript"></script>

<script>
    let AC = new Autocapture();
    AC.init({
        CONTAINER_ID: 'AC_mount',
        LANGUAGE: 'nl',
        ROOT: "../",
        TOKEN: "<SDK_TOKEN>",
        onComplete: function (data) {
            console.log(data)
        },
        onError: function(error) {
            console.log(error)
        },
        onUserExit: function(error) {
            console.log(error);
            window.history.back()
        }
    });
</script>

</html>

Languages

There are two ways in which notifications can be loaded: from file, from object (json).

File

The languages can be found in js/language/. The current support languages are en and nl. More languages could be created.

The notifications can be loaded in configuration like the following:

let AC = new AutoCapture();
AC.init({
    LANGUAGE: 'en',
    ...

To create support for a new language, a js file needs to be created with specific keys. The keys can be derived from the current language js files (js/language/en.js).

Example:

var LANGUAGE = {
    "start_prompt": "Tap to start",
    "flip": "Flip the document",
    "flip_frontside": "Flip the document to the frontside",
    "flip_backside": "Flip the document to the backside",
    "std_msg_0": "Place your document in the frame",
    "exp_dark": "Environment is too dark",
    "exp_bright": "Environment is too bright",
    "blur": "Hold still...",
    "glare": "Glare detected",
    "size": "Move closer",
    "focus": "Focus on document.",
    "approval_prompt": "Is the image right?",
    "retry": "Try again",
    "confirm": "Accept",
    "capture_error": "We were unable to capture an image. Camera access is required.",
    "mrz_search": "Searching MRZ...",
    "continue": "Continue",
    "rotate_phone": "Please rotate your phone upright",
}

Object (json)

Notifications can also be loaded as a json object like the following:

let AC = new AutoCapture();
AC.init({
    LANGUAGE: JSON.stringify(
        {
            "start_prompt": "Tap to start",
            "flip": "Flip the document",
            "flip_frontside": "Flip the document to the frontside",
            "flip_backside": "Flip the document to the backside",
            "std_msg_0": "Place your document in the frame",
            "exp_dark": "Environment is too dark",
            "exp_bright": "Environment is too bright",
            "blur": "Hold still...",
            "glare": "Glare detected",
            "size": "Move closer",
            "focus": "Focus on document.",
            "approval_prompt": "Is the image right?",
            "retry": "Try again",
            "confirm": "Accept",
            "capture_error": "We were unable to capture an image. Camera access is required.",
            "mrz_search": "Searching MRZ...",
            "continue": "Continue",
            "rotate_phone": "Please rotate your phone upright",
        }
    ),
    ...

Models

The tool uses a collection of neural networks. Make sure that you host the full directory so the models can be accessed. The models path can be configured. (see Configuration) The models are located under models/.

Face

There is a face detector within the SDK. This detector will try to find the portrait picture on the document. If it is found, this face will be present in the output. Else, the output will not have any mention of face. When more faces are found only one face is being returned. The face with the highest score will be returned.

Example:

{
    "image": ["...base64_img"],
    "face": {
        "score": 0.9,
        "data": "...base64_img"
    },
}

MRZ

The application offers Machine Readable Zone (MRZ) scanning functionality, which allows it to extract information from documents containing MRZ data. This is achieved through Optical Character Recognition (OCR) techniques. The application includes an internal OCR engine specifically designed for MRZ scanning. Additionally, users have the option to integrate an external OCR engine (see External MRZ) for MRZ scanning if the internal OCR engine consumes excessive RAM.

Supported Document Types

The MRZ scanning tool supports various types of documents, including:

• TD1: Travel Document Type 1
• TD2: Travel Document Type 2
• TD3: Travel Document Type 3
• Dutch eDL: Dutch electronic driver's license

MRZ Scanning Process

The MRZ scanning process involves the following steps:

1. Quality Checks: MRZ scanning initiates only after passing all necessary quality checks to ensure the document's readability.

2. Crop and Perspective Correction: The tool identifies the location of the MRZ on the document and performs cropping and perspective correction to isolate the MRZ area. This cropped section is then used as an image for the scanning process.

   • Note: When an external OCR is used, the image that will be used is the cropped card instead of the isolated MRZ area. (see External MRZ)

3. Retries: If the initial scan does not yield the desired results, the tool offers the option to retry scanning. In that case, the tool will try to use the next frame. Only frames that pass all the quality checks will be used. The number of retries can be configured using the following setting: (see Configuration).

   MRZ_RETRIES: 5

   To enable infinite retries, use:

   MRZ_RETRIES: -1

Enabling MRZ Scanning

To enable MRZ scanning use MRZ: true(see Configuration).

MRZ Configuration (MRZ_SETTINGS)

ATTRIBUTE	FORMAT	DEFAULT VALUE	EXAMPLE	NOTES
FLIP_EXCEPTION	Array<String>	[]	["P"]	optional If FLIP is true skip these misc values. Examples of possible misc values are: ["P", "I", "D"].
FLIP_INCLUDE	Array<Object>	[{ "misc": "all", "country": "all", "nationality": "all"}]	[{ "misc": "P", "country": "NLD", "nationality": "NLD", }]	optional If FLIP is true only flip if these values are included in the mrz output.
FLIP	bool	true	true	optional Prompt user to flip card.
MIN_VALID_SCORE	int	50	50	optional Minimum valid score of MRZ. The valid score is a range between 0-100 indicating how valid the found MRZ is.
MRZ_RETRIES	int	5	5	optional Amount of retries for MRZ scanning before continuing. Use -1 for infinite retries.
OCR	bool	true	true	optional Wether to use the internal OCR for MRZ reading. An external OCR can also be used see MRZ.

Example:

let AC = new Autocapture();
AC.init({
    CONTAINER_ID: 'AC_mount',
    MRZ: true,
    MRZ_SETTINGS: {
        MRZ_RETRIES: -1,
        FLIP: true,
        FLIP_EXCEPTION: [],
        FLIP_INCLUDE: [
            {
                "misc": "I",
                "country": "all",
                "nationality": "all",
            },
            {
                "misc": "D",
                "country": "all",
                "nationality": "all",
            },
            {
                "misc": "P",
                "country": "NLD",
                "nationality": "NLD",
            },
        ],
        MIN_VALID_SCORE: 90,
        OCR: true
    },
    TOKEN: "<SDK_TOKEN>",
    onComplete: function (data) {
        console.log(data)
    },
    onError: function(error) {
        console.log(error)
    },
    onUserExit: function (error) {
        console.log(error)
        window.history.back()
    }
})

Example output:

{   
    "angle": "...",
    "type": "PASSPORT",
    "subtype": "<",
    "country": "NLD",
    "lname": "DE",
    "lname2": "BRUIJN",
    "spacing": "",
    "fname": "WILLEKE",
    "mname1": "LISELOTTE",
    "name_complement": "",
    "number": "SPECI2014",
    "check_digit_document_number": "2",
    "nationality": "NLD",
    "date_of_birth": "1965-03-10",
    "check_digit_date_of_birth": "1",
    "sex": "F",
    "expiration_date": "2024-03-09",
    "check_digit_expiration_date": "6",
    "complement": "999999990<<<<<84",
    "mrz_type": "td3",
    "raw_mrz": [
        "P<NLDDE<BRUIJN<<WILLEKE<LISELOTTE<<<<<<<<<<<",
        "SPECI20142NLD6503101F2403096999999990<<<<<84"
    ],
    "check_digit_composite": "4",
    "personal_number": "999999990",
    "check_digit_personal_number": "8",
    "valid_number": true,
    "valid_date_of_birth": true,
    "valid_expiration_date": true,
    "valid_personal_number": true,
    "valid_composite": true,
    "valid_misc": true,
    "valid_score": 100,
    "misc": "P",
    "names": "WILLEKE LISELOTTE",
    "surname": "DE BRUIJN"
}

External MRZ

It is possible to retrieve the MRZ externally and then send it to the SDK using the following approach:

Using the onImage Callback

1. Utilize the onImage callback to trigger an external process with the current frames.

2. If this external process successfully retrieves the MRZ information, it can then be sent to the SDK using the parse_mrz function.

   The parse_mrz function expects the MRZ lines as an argument, provided as a single string with line breaks ("\n") separating the lines.

Here's an example of how this can be implemented in JavaScript:

// Initialize the Autocapture SDK
let AC = new Autocapture();
AC.init({
    // ... other configurations
    onImage: function (data) {
        // External OCR process retrieves MRZ information
        let MRZ_text = EXTERNAL_OCR_FUNCTION(data); // Example MRZ text: "P<NLDDE<BRUIJN<<WILLEKE<LISELOTTE<<<<<<<<<<<\nSPECI20142NLD6503101F2403096999999990<<<<<84"
        
        // Send the retrieved MRZ information to the SDK for parsing
        AC.parse_mrz(MRZ_text);
    },
});

Output

The SDK will output in the following structure:

{
    "image": ["...base64_img"],
    "face": {
        "score": "...",
        "data": "...base64_img"
    },
    "force_capture": ["bool"],
    "meta": [
        {
            "angle": "...",
            "coordinates": [
                ["...", "..."],
                ["...", "..."],
                ["...", "..."],
                ["...", "..."]
            ]
        }
    ],
    "token": "sdk_token"
}

With MRZ:

{
    "image": ["...base64_img"],
    "face": {
        "score": "...",
        "data": "...base64_img"
    },
    "force_capture": [false, false],
    "meta": [
        {
            "angle": "...",
            "coordinates": [
                ["...", "..."],
                ["...", "..."],
                ["...", "..."],
                ["...", "..."]
            ]
        }
    ],
    "mrz": {
        "angle": "...",
        "type": "...",
        "subtype": "...",
        "country": "...",
        "lname": "...",
        "lname2": "...",
        "spacing": "...",
        "fname": "...",
        "mname1": "...",
        "name_complement": "...",
        "number": "...",
        "check_digit_document_number": "...",
        "nationality": "...",
        "date_of_birth": "...",
        "check_digit_date_of_birth": "...",
        "sex": "...",
        "expiration_date": "...",
        "check_digit_expiration_date": "...",
        "complement": "...",
        "mrz_type": "...",
        "raw_mrz": [
            "...",
            "..."
        ],
        "check_digit_composite": "...",
        "personal_number": "...",
        "check_digit_personal_number": "...",
        "valid_number": true,
        "valid_date_of_birth": true,
        "valid_expiration_date": true,
        "valid_personal_number": true,
        "valid_composite": true,
        "valid_misc": true,
        "valid_score": true,
        "misc": "...",
        "names": "...",
        "surname": "..."
    },
    "token": "sdk_token"
}

Example:

{
    "image": ["iVBORw0KGgoAAAANSUhEUgAAAysAAAS..."],
    "face": {
        "score": 0.9,
        "data": "iVBORw0KGgoAAAANSUhEUgAAAysAAAS..."
    },
    "force_capture": [
        false,
        true
    ],
    "meta": [
        {
            "angle": 0,
            "coordinates": [
                [0, 0],
                [0, 100],
                [150, 100],
                [150, 0]
            ]
        }
    ],
    "mrz": {
        "angle": "...",
        "type": "PASSPORT",
        "subtype": "<",
        "country": "NLD",
        "lname": "DE",
        "lname2": "BRUIJN",
        "spacing": "",
        "fname": "WILLEKE",
        "mname1": "LISELOTTE",
        "name_complement": "",
        "number": "SPECI2014",
        "check_digit_document_number": "2",
        "nationality": "NLD",
        "date_of_birth": "1965-03-10",
        "check_digit_date_of_birth": "1",
        "sex": "F",
        "expiration_date": "2024-03-09",
        "check_digit_expiration_date": "6",
        "complement": "999999990<<<<<84",
        "mrz_type": "td3",
        "raw_mrz": [
            "P<NLDDE<BRUIJN<<WILLEKE<LISELOTTE<<<<<<<<<<<",
            "SPECI20142NLD6503101F2403096999999990<<<<<84"
        ],
        "check_digit_composite": "4",
        "personal_number": "999999990",
        "check_digit_personal_number": "8",
        "valid_number": true,
        "valid_date_of_birth": true,
        "valid_expiration_date": true,
        "valid_personal_number": true,
        "valid_composite": true,
        "valid_misc": true,
        "valid_score": 100,
        "misc": "P",
        "names": "WILLEKE LISELOTTE",
        "surname": "DE BRUIJN"
    },
    "token": "sdk_token"
}