Security News
ESLint is Now Language-Agnostic: Linting JSON, Markdown, and Beyond
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
@mondomob/gae-js-datastore-backups
Advanced tools
Module to perform scheduled Datastore exports in your app.
Backups can be configured to perform full or partial exports of Datastore into Google Cloud Storage.
As well as regular backups you can configure automatic export and import into BigQuery. This is a full import/export and any existing data in BigQuery will be deleted for the requested kinds.
Datastore exports are long-running operations. This module keeps track of operations it starts in a backup-operations
kind.
After an operation is started it will queue tasks to update the status of the operation until it is complete.
Once complete the backup can be auto-imported into BigQuery.
npm install @mondomob/gae-js-datastore-backups
By default this library uses the task queue named backup-queue
, so create that if it doesn't exist.
Alternatively add configuration for the queue you wish to use.
e.g. add to queue.yaml
(or add equivalent to terraform, etc)
queue:
- name: backup-queue
mode: push
rate: 1/s
retry_parameters:
task_retry_limit: 20
min_backoff_seconds: 120
max_doublings: 5
By default this library will use a bucket called datastore-backup-[PROJECT_ID]
, so create that if it doesn't exist.
This must be a regional bucket in the same region as your datastore instance.
Alternatively you can add configuration for the bucket you wish to use.
Authorise the App Engine Default Service account to perform datastore exports In Google Cloud Console IAM & Admin add the following IAM role:
Datastore -> Cloud Datastore Import Export Admin
e.g. at the root application
app.use("/crons", gaeJsCron, datastoreBackupCronRoutes());
app.use("/tasks", gaeJsTask, datastoreBackupTaskRoutes());
or as part of a sub-application/sub-router
const cronsController = Router();
cronsController.use(datastoreBackupCronRoutes())
cronsController.get("/some-other-cron", asyncHandler(async () => {}))
app.use("/crons", gaeJsCron, cronsController);
const tasksController = Router();
tasksController.use(datastoreBackupTaskRoutes())
tasksController.post("/some-other-task", asyncHandler(async () => {}))
app.use("/tasks", gaeJsTask, tasksController);
Add cron schedules to your cron.yaml
for the backup jobs you wish to run. e.g.
cron:
- description: "Datastore export"
url: /crons/backups/datastore?type=EXPORT&name=FullBackup
schedule: every day 03:00
timezone: Australia/NSW
- description: "Datastore export and load to BigQuery"
url: /crons/backups/datastore?type=EXPORT_TO_BIGQUERY&name=ExportToBigQuery&targetDataset=backup_data&collectionIds=demo-items
schedule: every day 03:00
timezone: Australia/NSW
Default endpoint: GET /crons/backups/datastore
Query params:
Property | Description | Required |
---|---|---|
type | The type of export to perform. Must be one of "EXPORT" or "EXPORT_TO_BIGQUERY" | Y |
name | name for the datastore export | N |
kinds | the kinds to be exported | N for EXPORT, Y for EXPORT_TO_BIGQUERY |
targetDataset | the target dataset for import in BigQuery | Y for EXPORT_TO_BIGQUERY |
No configuration is required if you are happy with the default conventions.
The following options are available under the datastoreBackup
namespace.
Property | Description | Required |
---|---|---|
bucket | the GCS bucket to export to. Default is datastore-backup-[PROJECT_ID] | N |
folderFormat | folder format to use - using Luxon date formatting options. Default yyyy/MM/yyyyMMdd-HHmmss | N |
queue | the queue to send backup tasks to. Default backup-queue | N |
taskPrefix | tasks handler url prefix. Default /tasks | N |
taskService | the app engine service to direct task requests to. Defaults to default service | N |
timeZone | the timezone to use when formatting backup output folder. Defaults to `"Australia/Sydney" | N |
e.g.
{
"projectId": "my-project-dev",
"datastoreBackup": {
"bucket": "my-backup-bucket",
"queue": "admin-queue",
"taskPrefix": "/tasks/admin",
"taskService": "backup-service",
"timeZone": "Australia/Sydney"
}
}
FAQs
Utils for performing datastore backups
We found that @mondomob/gae-js-datastore-backups demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 8 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
ESLint has added JSON and Markdown linting support with new officially-supported plugins, expanding its versatility beyond JavaScript.
Security News
Members Hub is conducting large-scale campaigns to artificially boost Discord server metrics, undermining community trust and platform integrity.
Security News
NIST has failed to meet its self-imposed deadline of clearing the NVD's backlog by the end of the fiscal year. Meanwhile, CVE's awaiting analysis have increased by 33% since June.