goose-parser

goose-parser

This tool moves routine crawling process to the new level. Now it's possible to parse a web page for a few moments. All you need is to specify parsing rules based on css selectors. It's so simple as Goose can do it. This library allows to parse such data types as grids, collections, and simple objects. Parser supports pagination via infinite scroll and pages. It offers next features: pre-parse actions and post-parse transformations.

Installation

npm install goose-parser

This library has dependency on PhantomJS 2.0. Follow instructions provided by the link or build it manually.

Documentation

All css selectors can be set in a sizzle format.

Environments

This is a special atmosphere where Parser has to be executed. The main purpose of the environment is to provide a method for evaluating JS on the page.

PhantomEnvironment

That environment is used for running Parser on node.

var env = new PhantomEnvironment({
    url: 'http://google.com',
});

The main and only required parameter is url. It contains an url address of the site, where Parser will start.

More detailed information about default options you can find here.

BrowserEnvironment

That environment is used for running Parser in the browser.

var env = new BrowserEnvironment();

To created packed js-file with Parser execute following command:

npm run build

Parser

Parser.js is the main component of the package which performs page parsing.

var parser = new Parser({
    environment: env,
    pagination: pagination
});

Fields:

• environment - one of the allowed environments.
• pagination [optional] - pagination definition.

Parse rules

Simple rule

The purpose of this rule - retrieving simple textual node value.

Example:

Parsing rule

{
    name: 'node',
    scope: 'div.simple-node'
}

HTML

<div class='simple-node'>simple-value</div>

Parsing result

{
    node: 'simple-value'
}

Fields:

• name - name of the node which is presented in the result dataSet.
• scope - css selector of the node.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• actions [optional] - see Actions.
• transform [optional] - see Transformations.

Collection rule

The purpose of this rule - retrieving collection of nodes.

Example:

Parsing rule

{
    name: 'row',
    scope: 'div.collection-node',
    collection: [
        {
            name: 'node1',
            scope: 'div.simple-node1'
        },
        {
            name: 'node2',
            scope: 'div.simple-node2'
        },
        {
            name: 'nested',
            scope: 'div.nested-node',
            collection: [
                {
                    name: 'node3',
                    scope: 'div.simple-node3'
                }
            ]
        }
    ]
}

HTML

<div class='collection-node'>
    <div class='simple-node1'>simple-value1</div>
    <div class='simple-node2'>simple-value2</div>
    <div class='nested-node'>
        <div class='simple-node3'>simple-value3</div>
    </div>
</div>

Parsing result

{
    row: {
        node1: 'simple-value1',
        node2: 'simple-value2',
        nested: {
            node3: 'simple-value3'
        }
    }
}

Fields:

• name - name of the node which is presented in the result dataSet.
• scope - css selector of the node.
• collection - array of any rule types.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• actions [optional] - see Actions.
• transform [optional] - see Transformations.

Grid rule

The purpose of this rule - retrieving collection of collection.

Example:

Parsing rule

{
    scope: 'div.collection-node',
    collection: [[
        {
            name: 'node1',
            scope: 'div.simple-node1'
        },
        {
            name: 'node2',
            scope: 'div.simple-node2'
        }
    ]]
}

HTML

<div>
    <div class='collection-node'>
        <div class='simple-node1'>simple-value1</div>
        <div class='simple-node2'>simple-value2</div>
    </div>
    <div class='collection-node'>
        <div class='simple-node1'>simple-value3</div>
        <div class='simple-node2'>simple-value4</div>
    </div>
</div>

Parsing result

[
    {
        node1: 'simple-value1',
        node2: 'simple-value2'
    },
    {
        node1: 'simple-value3',
        node2: 'simple-value4'
    }
]

Fields:

• scope - css selector of the node.
• collection - array of array of any rule types.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• actions [optional] - see Actions.
• transform [optional] - see Transformations.

Pagination

This is a way to parse collection-based data. See more info in Paginator.js

Scroll pagination

This type of pagination allows to parse collections with infinite scroll.

{
    type: 'scroll',
    interval: 500,
    maxPagesCount: 2
}

Fields:

• type - "scroll" for that type of pagination.
• interval - interval in pixels to scroll.
• maxPagesCount [optional] - max pages to parse.

Page pagination

This type of pagination allows to parse collections with ajax-page pagination.

JS definition

{
    type: 'page',
    scope: '.page',
    pageScope: '.pageContainer',
    maxPagesCount: 2
}

HTML

<div>
    <div class='pageContainer'>
        <div class='collection-node'>
            <div class='simple-node1'>simple-value1</div>
            <div class='simple-node2'>simple-value2</div>
        </div>
        <div class='collection-node'>
            <div class='simple-node1'>simple-value3</div>
            <div class='simple-node2'>simple-value4</div>
        </div>
    </div>
    <div class='pagination'>
        <div class='page'>1</div>
        <div class='page'>2</div>
        <div class='page'>3</div>
    </div>
</div>

Fields:

• type - "page" for that type of pagination.
• scope - css selector for paginator block (page label).
• pageScope - css selector for page scope (container for page-data).
• maxPagesCount [optional] - max pages to parse.

Actions

Allow to execute actions on the page before parse process.

Click

Click by the element on the page.

Example:

{
    type: 'click',
    scope: '.open-button'
    parentScope: 'body',
    once: true
}

Fields:

• type - type of action
• scope - css selector of the node.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• once [optional] - to perform action only once (can be useful on pre-parse moment).

Wait

Wait for the element on the page.

Example:

{
    type: 'wait',
    scope: '.open-button.done'
    timeout: 2 * 60 * 1000,
    parentScope: 'body',
    once: true
}

Fields:

• type - type of action
• scope - css selector of the node.
• timeout [optional] - time to cancel wait in seconds.
• parentScope [optional] - css selector of the parent node, to specify a global scope (outside current).
• once [optional] - to perform action only once (can be useful on pre-parse moment).

Transformations

Allow to transform parsed value to some specific form.

Date

Format date to specific view (using momentjs).

{
    type: 'date',
    locale: 'ru',
    from: 'HH:mm D MMM YYYY',
    to: 'YYYY-MM-DD'
}

Replace

Replace value using Regex.

{
    type: 'replace',
    re: ['\\s', 'g'],
    to: ''
}

Tests

With PhantomEnvironment

To run tests use command:

npm test

With BrowserEnvironment

To run tests build them with command:

npm run build-test

And then run file in the browser.

Debug

All parser components are covered by debug library, which give an ability to debug application in easy way. Set DEBUG variable with name of js file to show debug information.

DEBUG=Parser,Actions app.js

Usage

var env = new PhantomEnvironment({
    url: uri,
    screen: {
        width: 1080,
        height: 200
    }
});

var parser = new Parser({
    environment: env,
    pagination: {
        type: 'scroll',
        interval: 500
    }
});

parser.parse({
    rules: {
        actions: [
            {
                type: 'wait',
                timeout: 2 * 60 * 1000,
                scope: '.container',
                parentScope: 'body',
                once: true
            }
        ],
        scope: 'div.scope-test',
        collection: [[
            actions: [
                {
                    type: 'click',
                    scope: '.open-button'
                },
                {
                    type: 'wait',
                    scope: '.open-button.done'
                }
            ],
            {name: 'column1', scope: 'div.scope-test-column1'},
            {
                name: 'sub-column',
                scope: 'div:last-child',
                collection: [
                    {
                        name: 'column2', 
                        scope: 'div.scope-test-column2'
                    },
                    {
                        name: 'column3', 
                        scope: 'div.scope-test-column3'
                        transform: [
                            {
                                type: 'date',
                                locale: 'ru',
                                from: 'HH:mm D MMM YYYY',
                                to: 'YYYY-MM-DD'
                            }
                        ]
                    },
                    {
                        name: 'column4', 
                        scope: 'div.scope-test-column4',
                        transform: [
                            {
                                type: 'replace',
                                re: ['\\s', 'g'],
                                to: ''
                            }
                        ]
                    }
                ]
            }
        ]]
    }
}).then(function(parsed) {
    
});