Contrast Crawler Project
This project uses application route data retrieved from TeamServer
to seed a web crawler implemented using Crawlee
and Playwright .
To Run Contrast Crawler
Customize example-contrast.yaml with your TeamServer user credentials
(not agent credentials), the TeamServer name of the application to test,
and the base URL of the application to test.
The launch command:
$ npm start example-contrast.yaml
To debug with Chrome DevTools for Node:
$ npm run debug example-contrast.yaml
and in Chrome load URL chrome://inspect and click on link Open dedicated DevTools for Node
.
To Configure Custom Fake Input
In the faker.json file, each object corresponds to a method in the faker.js API
For, example
{
"module": "location",
"name": "streetAddress"
}
corresponds to: https://fakerjs.dev/api/location.html#streetaddress
This method will be called and used to generate a fake address when an input field named "address" is encountered by the crawler.
Some objects have a field called "predicates". These are used if your application has unique identifiers for input. For example, "surname" might be used to identify a field where a person's last name is expected. The faker.js method for generating last names is called lastName thus, a predicate is used to retrieve the correct method.
Predicates can be added or removed, according to the needs of your application. They can be stored as a string or an array of strings.
Some objects have a field called "input". These can be used if the faker method needs input to produce a specific kind of output. For example, if you need a zip code to be from New York that needs to be passed into the zipcode method as {state: 'New York'}
which would have to be added as "input": "{state: 'New York'}" to the zipcode object in the JSON.
Some objects have a field called "fn". These are used for post-processing fake input. For example, the birthdate method outputs an instance of a Date object which could be invalid input for your application. Adding
"fn": {
"methods": [
{
"name": "toISOString"
},
{
"name": "split",
"input": "T"
}
],
"idx": 0
}
to the birthdate object does the following: converts the Date object to an ISOString then splits the resulting string by "T" and returns the element at index 0 so that it preserves the year-month-day format wihtout including the timezone.