@attestate/crawler

0.5.0

• (breaking) config.path[] for transformer, extractor and loader, the properties output.path and input.path were renamed to output.name and doesn't have to be a path anymore. Instead, they are file names that are automatically resolved from within env.DATA_DIR.
• (breaking) process.env variables defined in the .env file can now also be defined (and overwritten) in the config.mjs file's environment property.
• (breaking) All lifecycle methods now have an updated interface as outlined below:
  • extractor function init({ state, args, execute })
  • extractor function update({ message })
  • tranformer function onLine({ state }) where state.line is the line function. args can be matched too.
  • loader function* order({ state }) where state.line is the line
  • loader function* direct({ state }) where state.line is the line
• There is a new component to a strategy called the "coordinator" that keeps track of state. The config.mjs file (the path property) features a new field called coordinator where a module and an interval can be defined. They're used to re-run the first path once all jobs have been completed, to e.g. keep in synchronization with a network like Ethereum.
• We forked the extraction-worker from the neume-network organization and added a feature to immediately execute a worker message. More details: https://github.com/attestate/extraction-worker/.
• Reference docs for the extraction worker have been added.
• Reference docs for the crawler CLI have been added.
• Note: The @attestate/crawler-call-block-logs module at version 0.3.0 is compatible.