@contrast/crawler

Contrast Crawler Project

This project uses application route data retrieved from TeamServer to seed a web crawler implemented using Crawlee and Playwright .

To Run Contrast Crawler

Customize example.yaml with your TeamServer user credentials (not agent credentials), the TeamServer name of the application to test, and the base URL of the application to test.

Install the crawler globally:

$ npm i -g @contrast/crawler

The launch command:

$ npx @contrast/crawler /path/to/contrast.yaml

To Configure Custom Fake Input

In the faker.json file, each object corresponds to a method in the faker.js API

For, example

  {
    "module": "location",
    "name": "streetAddress"
  }

corresponds to: https://fakerjs.dev/api/location.html#streetaddress

This method will be called and used to generate a fake address when an input field named "address" is encountered by the crawler.

Some objects have a field called "predicates". These are used if your application has unique identifiers for input. For example, "surname" might be used to identify a field where a person's last name is expected. The faker.js method for generating last names is called lastName thus, a predicate is used to retrieve the correct method.

Predicates can be added or removed, according to the needs of your application. They can be stored as a string or an array of strings.

Some objects have a field called "input". These can be used if the faker method needs input to produce a specific kind of output. For example, if you need a zip code to be from New York that needs to be passed into the zipcode method as {state: 'New York'} which would have to be added as "input": "{state: 'New York'}" to the zipcode object in the JSON.

Some objects have a field called "fn". These are used for post-processing fake input. For example, the birthdate method outputs an instance of a Date object which could be invalid input for your application. Adding

    "fn": {
      "methods": [
        {
          "name": "toISOString"
        },
        {
          "name": "split",
          "input": "T"
        }
      ],
      "idx": 0
    }

to the birthdate object does the following: converts the Date object to an ISOString then splits the resulting string by "T" and returns the element at index 0 so that it preserves the year-month-day format wihtout including the timezone.