@attestate/crawler - npm Package Compare versions

Comparing version 0.4.0 to 0.5.0

changelog.md

 # Changelog
 
+##
+
+- (breaking) `config.path[]` for transformer, extractor and loader, the
+  properties `output.path` and `input.path` were renamed to `output.name` and
+  doesn't have to be a path anymore. Instead, they are file names that are
+  automatically resolved from within `env.DATA_DIR`.
+- (breaking) `process.env` variables defined in the `.env` file can now also be
+  defined (and overwritten) in the `config.mjs` file's `environment` property.
+- (breaking) All lifecycle methods now have an updated interface as outlined
+  below:
+  - extractor `function init({ state, args, execute })`
+  - extractor `function update({ message })`
+  - tranformer `function onLine({ state })` where `state.line` is the line
+    function. `args` can be matched too.
+  - loader `function* order({ state })` where `state.line` is the line
+  - loader `function* direct({ state })` where `state.line` is the line
+- There is a new component to a strategy called the "coordinator" that keeps
+  track of state. The config.mjs file (the `path` property) features a new
+  field called `coordinator` where a `module` and an `interval` can be defined.
+  They're used to re-run the first path once all jobs have been completed, to
+  e.g. keep in synchronization with a network like Ethereum.
+- We forked the extraction-worker from the neume-network organization and added
+  a feature to immediately execute a worker message. More details:
+  https://github.com/attestate/extraction-worker/.
+- Reference docs for the extraction worker have been added.
+- Reference docs for the crawler CLI have been added.
+- Note: The `@attestate/crawler-call-block-logs` module at version 0.3.0 is
+  compatible.
+
 ## 0.4.0
@@ -4,0 +33,0 @@

package.json

 {
   "name": "@attestate/crawler",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "@attestate/crawler is a tool chain to retrieve on-chain data from Ethereum.",
@@ -33,3 +33,3 @@ "main": "./src/boot.mjs",
   "dependencies": {
-    "@neume-network/extraction-worker": "0.7.1",
+    "@attestate/extraction-worker": "0.7.2",
     "ajv": "8.11.0",
@@ -41,3 +41,4 @@ "ajv-formats": "2.1.1",
     "lmdb": "2.7.9",
-    "yargs": "^17.5.1"
+    "lodash.invert": "4.3.0",
+    "yargs": "17.5.1"
   },
@@ -44,0 +45,0 @@ "devDependencies": {

readme.md

@@ -12,11 +12,11 @@ # @attestate/crawler
   recover errors.
-
-* **Embedded:** Unlike Graph Protocol, Attestate's Crawler does not have a
+- **Embedded:** Unlike Graph Protocol, Attestate's Crawler does not have a
   separate server process. It writes directly to
   [LMDB](http://www.lmdb.tech/doc/), an embedded database that can be read-from
   thread-safely at any time.
-
 - **Batteries-included:** Support for Ethereum, IPFS, Arweave, GraphQL and
   rate-limit pooling for Infura and Alchemy.
 
+## Links
+
 * **[Getting Started](https://attestate.com/crawler/main/getting-started.html)**
@@ -23,0 +23,0 @@ * Documentation: [attestate.com/crawler/main](https://attestate.com/crawler/main)

No alert changes

Improved metrics

Total package byte prevSize

153961

increased by12.78%

Number of package files

56

increased by3.7%

Lines of code

2073

increased by16.2%

Worsened metrics

Dependency count

9

increased by12.5%

Number of low supply chain risk alerts

11

increased by10%

Dependency changes

• + Added@attestate/extraction-worker@0.7.2

• + Addedlodash.invert@4.3.0

• + Added@attestate/extraction-worker@0.7.2(transitive)

• + Addedcliui@7.0.4(transitive)

• + Addedlodash.invert@4.3.0(transitive)

• + Addedyargs@17.5.1(transitive)

• - Removed@neume-network/extraction-worker@0.7.1

• - Removed@neume-network/extraction-worker@0.7.1(transitive)

• - Removedcliui@8.0.1(transitive)

• - Removedyargs@17.7.2(transitive)

• Updatedyargs@17.5.1