Apify Global Store
What's new in 1.1.3
?
- Addition of
addMethod
and useMethod
(experimental) pushPathToDataset
method name shortened to pushPath
- Support for array formatted paths in
setPath
, deletePath
, and pushPath
- Bug fixes for
pushPath
, summon
, and summonAll
- Improve debug logs
- Improve JSDoc
Installation
npm install apify-global-store
About
This is a simple and lightweight API which allows you to easily create and manage multiple global stores local to an actor's run. By using apify-global-store
, you completely remove the need to pass large and complex data through requests. Simplify the process of storing incomplete data that isn't yet ready to be pushed to the dataset with GlobalStore
!
State persistence, actor migrations, and race conditions are automatically handled under the hood, so the global state will not be lost on migrations or graceful aborts.
Table of Contents:
Importing
ES6+
import { GlobalStore } from 'apify-global-store'
ES5-
const { GlobalStore } = require('apify-global-store')
Example
const Apify = require('apify');
const { GlobalStore } = require('apify-global-store');
Apify.main(async () => {
const store = await GlobalStore.init();
console.log(store.state);
store.set((prev) => {
return {
...prev,
message: 'hello'
}
})
console.log(store.state);
store.set((prev) => {
return {
...prev,
message2: 'world'
}
})
console.log(store.state);
await store.pushPath('message')
});
General Usage
await GlobalStore.init()
(initializeOptions?: InitializeOptions => Promise<GlobalStore>
Argument | Type | Required | Description |
---|
initializeOptions | InitializeOptions | false | Configure the store's name, its initial state, and where it is located. |
interface InitializeOptions {
name?: string;
initialState?: Record<string, unknown>;
cloud?: boolean;
debug?: boolean;
}
GlobalStore
doesn't have a public constructor. Its init()
method runs necessary asynchronous tasks prior to calling on its private constructor.
Usage:
const store = await GlobalStore.init({
name: 'clothing-store',
initialState: {
websites: [],
errors: [],
},
cloud: true,
debug: true
});
If the store already has a previous state stored within the Key-Value store (in the situation that a migration/abort occurred), then the initialState
option will have no effect.
You can only pass letters and "-" within the string, otherwise an error will be thrown (to follow naming conventions). The provided store name will always be forced into uppercase.
Note: A name for a store can only be used once. That includes the default name of GLOBAL-STORE
. If the cloud
option is set to true, then the global store will be stored in the cloud on your Apify account under "Storages" within a named Key-Value store called CLOUD-GLOBAL-STORES
.
store.state
Usage:
const { myValue } = store.state;
A getter method that returns the current state object of the store.
store.info
const { sizeInBytes, lastModified } = store.info;
An object containing information about the contents of the store. Example:
{
sizeInBytes: 17,
lastModified: '2022-03-18T15:39:45.041Z',
globalStoreVersion: '1.0.9',
type: 'LOCAL'
}
Mutating State
store.set()
(setStateParam: SetStateFunctionCallBack) => void
Argument | Type | Required | Description |
---|
setStateParam | SetStateFunctionCallBack | true | A callback function in which the current state is passed and can be modified. Return value must be an object which will be set as the new state. |
A synchronous function which sets the current state. The previous state is passed into the callback function as a parameter. It is recommended to refer to the previous state through this parameter instead of using store.state
within the callback.
Usage:
store.set((prev) => {
return {
...prev,
newValue: 'hello world'
}
});
Note: It is always best practice to spread out the previous state at the top level of the returned object, then to add/override any values below.
store.setPath()
(path: string | string[], value: unknown) => void
Argument | Type | Required | Description |
---|
path | string | true | A string version of the path within the state you'd like to set/replace. |
value | unknown | true | The value you'd like to add for the specified path. |
Usage:
store.setPath(`products.${productId}.reviews`, reviewsData);
If the key you are using has dots in it (eg. "https://google.com"), you should use bracket notation instead of dot notation.
const link = 'https://google.com';
const data = { hello: 'world' }
store.setPath(`websites.${link}`, data);
store.setPath(`websites[${link}]`, data);
store.setPath(['websites', link], data);
store.deletePath()
(path: string | string[]) => void
Argument | Type | Required | Description |
---|
path | string | true | A string version of the path within the state you'd like to delete. |
Usage:
store.setPath(`products.${productId}.reviews`);
Note: This method works similar to store.pushPath()
, except it does not push the data to the dataset prior to deleting it from the state.
store.addReducer()
(reducerFn: ReducerFunction) => void
Argument | Type | Required | Description |
---|
reducerFn | ReducerFunction | true | A custom reducer function which takes (state, action) as its arguments. |
When you find yourself writing the same code over and over again using the store.set()
method, or you are dealing with modifying deeply nested values, it is best practice to migrate to using a reducer function.
Usage:
const reducer = (state, action) => {
switch (action.type) {
default:
return state;
case 'ADD-PRODUCT-REVIEW':
return {
...state,
products: {
...state.products,
[action.productId]: { ...state[action.productId], reviews: [...state[action.productId.reviews], ...action.payload] },
},
};
}
};
store.addReducer(reducer);
Note: Only one reducer can be added to a store instance. Trying to add a second one will result in an error.
store.setWithReducer()
(action: ReducerAction) => void
Argument | Type | Required | Description |
---|
action | ReducerAction | true | Use your reducer function to modify the state. |
Usage:
const action = { type: 'ADD-PRODUCT-REVIEW', productId: 1234, payload: { title: 'review 1', author: 'super cool product!', stars: 4 } };
store.setWithReducer(action);
Data Management
await store.pushPath()
(path: string, dataset?: Dataset => Promise<void>
Argument | Type | Required | Description |
---|
path | string | true | A string version of the path within the state you'd like to push to the dataset. |
dataset | Dataset | false | The dataset to push to. If not provided a dataset, the default one will be used. |
Push some data from the store into the specified dataset (or into the default one), then automatically delete it from the store after it's been pushed.
Usage:
await store.pushPath(`products.${productId}.reviews.${reviewId}`);
Note: When using this method to push to a dataset, the path is deleted within the state. If you don't want the data to be deleted from the global store after being pushed to the dataset, use regular Apify.pushData()
instead.
store.dump()
() => void
Completely empty the entire contents of the store.
await store.backup()
() => Promise<void>
Back the store up to the cloud. The backup will be stored in the "Storages" section of your Apify account under a Key-Value store named CLOUD-GLOBAL-STORES
. If you'd prefer to exclusively keep the store on the cloud, set the cloud
option to true
in InitializeOptions
.
Note: If you're using cloud: true
in InitializeOptions
, you don't need to use this method. Just use store.forceSave()
instead
await store.forceSave()
() => Promise<void>
The store's data is saved to the Key-Value Store every single time the "persistState" event is fired. It can also be forced to be saved instantly with this method.
Static Methods
GlobalStore.summon()
(storeName: string) => GlobalStore
Argument | Type | Required | Description |
---|
storeName | string | false | The name of the store to summon. |
A static method which returns the instance of the store attached to the name provided. A useful and simple aternative to passing a store instance around a parameter.
You are not required to provide the uppercase version of the store name in the storeName
parameter. It is automatically done for you.
If you do not provide a store name, then the default store GLOBAL-STORE
will be summoned.
Usage:
const store = await GlobalStore.init();
const summoned = GlobalStore.summon('GLOBAL-STORE');
console.log(summoned.state);
GlobalStore.summonAll()
() => StoreInstances
Similar to summon
, but returns the entire storeInstances
object, which is a map of all created instances of GlobalStore.
GlobalStore.dumpAll()
() => void
Dump all instances of GlobalStore at once.
Experimental Methods
GlobalStore.addMethod()
(addMethodOptions: AddMethodOptions) => void
Argument | Type | Required | Description |
---|
addMethodOptions | AddMethodOptions | true | An object containing the name and method to add. Within this method, the return value of GlobalStore.summonAll() is already passed in as the first parameter. |
Add a custom method to the entire GlobalStore class, which is accessible through GlobalStore.useMethod()
. This method can be used to modify any store instance.
This is valuable for convenience sake, or when you are making complex modifications to multiple stores.
Usage:
Here is an example of using addMethod to add a complex method to GlobalStore, which modifies the state of three different instances.
import { GlobalStore, CustomMethod } from 'apify-global-store';
import { methodNames, storeNames } from './consts';
const { ADD_PRODUCT_AND_REVIEWS } = methodNames;
const { PRODUCTS, REVIEWS, RUN_DATA } = storeNames;
const method: CustomMethod = async (instances, product, website) => {
const productStore = instances[PRODUCTS];
const reviewsStore = instances[REVIEWS];
const runDataStore = instances[RUN_DATA]
productStore.set((prev) => {
const { reviews, ...productData } = product;
return {
...prev,
[website]: {
products: {
...prev[website].products,
[productData.id]: productData
}
}
}
})
reviewsStore.setPath([website, product.id, 'latestReviews'], product.reviews)
if (reviewsStore.state[website].length >= 100) {
await reviewsStore.pushPath(website);
};
runDataStore.set((prev) => {
return {
...prev,
[website]: {
...prev[website],
runs: prev[website].runs + 1
}
}
})
};
GlobalStore.addMethod({ name: ADD_PRODUCT_AND_REVIEWS, method });
Note: It's not recommended to use reducers inside of these custom methods. That can make things complicated very quickly. addMethod
can serve as an alternative to addReducer
.
GlobalStore.useMethod()
(name: string, ...rest) => void
| Promise<void>
Argument | Type | Required | Description |
---|
name | string | true | The name of the method used when adding it to GlobalStore. |
...rest | any[] | false | Any custom parameters the method needs to run properly |
Usage:
import { methodNames } from './consts';
const { ADD_PRODUCT_AND_REVIEWS } = methodNames;
const product = parseProduct($);
await GlobalStore.useMethod(ADD_PRODUCT_AND_REVIEWS, product, new URL(request.url).hostname);
Best Practices
These are not necessarily gospel, but they can help you modularize and scale your project while also making it more readable.
Store Management
When using more than one instance of GlobalStore, it is best to use custom store names, and to put them into a constant:
consts.ts
:
export enum storeNames { PRODUCTS = 'PRODUCTS', HOTELS = 'HOTELS' };
main.ts
:
import { storeNames } from './consts';
const productStore = await GlobalStore.init({ name: storeNames.PRODUCTS });
const hotelStore = await GlobalStore.init({name: storeNames.HOTELS});
const summoned = GlobalStore.summon(storeNames.HOTELS);
Using a reducer
- It is always best practice to define your action types as constants, then to import them when defining your reducer.
- Always define your reducer outside of the
store.addReducer()
method. Ideally, this should be done in a separate file, then imported.
Example:
consts.ts
:
export enum actions {
GENERAL = 'GENERAL',
ADD_PRODUCT = 'ADD_PRODUCT',
}
reducer.ts
:
import { ReducerFunction } from 'apify-global-store';
import { actions } from './consts';
const reducer: ReducerFunction = (state, action) => {
switch (action.type) {
default:
return state;
case actions.GENERAL:
return {
...state,
...action.payload,
};
case actions.ADD_PRODUCT:
return {
...state,
products: { ...state.products, ...action.payload }
}
}
};
export default reducer;
main.ts
:
import { GlobalStore } from 'apify-global-store';
import { actions } from './consts'
import reducer from './reducer'
const store = await GlobalStore.init();
store.addReducer(reducer);
store.setWithReducer({
type: actions.GENERAL,
payload: { hello: 'world' },
});
Storage
In the Key-Value store, each store will be represented by an object looking like this:
{
"store": {
"hello": "world"
},
"data": {
"sizeInBytes": 17,
"lastModified": "2022-03-18T15:39:45.041Z",
"globalStoreVersion": "1.0.9",
"type": "LOCAL"
}
}
If the cloud
option was not set to true
upon the store's initialization, the store will be local to the actor's run under a key in the default Key-Value store.
If cloud
was set to true
, the global store will be stored in the cloud on your Apify account, under a named Key-Value store called CLOUD-GLOBAL-STORES
.
Please: DO NOT manually modify these objects using Apify.setValue()
. GlobalStore uses the actor's default Key-Value store under the hood.
Available Types
The GlobalStore class.
The options to initialize an instance of GlobalStore.
interface InitializeOptions {
name?: string;
initialState?: Record<string, unknown>;
cloud?: boolean;
debug?: boolean;
}
An object representing the state returned from store.state
.
type StoreState = Record<string, unknown>;
The callback function to be passed into store.set()
type SetStateFunctionCallBack = (previous: StoreState) => StoreState;
The action
object of your reducer function. Must include a type
key.
interface ReducerType {
type: string;
}
type ReducerParam<T> = ReducerType & Record<string, T>;
A function which should be passed into store.addReducer()
type ReducerFunction = <T>(state: StoreState, action: ReducerParam<T>) => StoreState;
An object representing the data which is returned from store.info
interface StoreData {
sizeInBytes: number;
lastModified: string;
globalStoreVersion: string;
type: 'LOCAL' | 'CLOUD';
}
An object which hold all instances of GlobalStore, accessible through GlobalStore.summonAll()
type StoreInstances = Record<DefaultStoreName | string, GlobalStore>;
export type CustomMethod = (storeInstances: StoreInstances, ...rest: any[]) => unknown | Promise<unknown>;
interface AddMethodOptions {
name: string;
method: CustomMethod;
}
Credits
Matt Stephens
Kristýna Lhoťanová
License
Copyright (c) 2022-present Matt Stephens
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.