
Security News
Deno 2.2 Improves Dependency Management and Expands Node.js Compatibility
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.
datastore-backup
Advanced tools
[Exporting Data from Google Cloud Datastore](https://cloud.google.com/datastore/docs/export-import-entities#rest_ seems easy:
sh``` gcloud datastore export gs://exampleBucket/backupdir
Or in node.js:
js```
const datastore = new Datastore()
datastore.export({outputUrlPrefix: 'gs://exampleBucket/backupdir'});
So why do you need a library for that? Because it is in the details!
For one thing the code above contains an export containing all kinds can not be loaded into BigQuery: Data exported without specifying an entity filter cannot be loaded into BigQuery."
So you need a list of all Kinds in the Datastore. But there are Limits on entity filters: "Each request is limited to 100 entity filter combinations, where each combination of filtered kind and namespace counts as a separate filter towards this limit." (It was 50 e few years before.)
And there is an other limit: "The managed export and import service limits the number of concurrent exports and imports to 50 and allows a maximum of 20 export and import requests per minute for a project."
So if you have a lot of entities you have to do some slicing and dicing.
Command line usage is simple. You have to provide the projectId of the datastore and a bucket name. Both have to be in the same region.
% yarn ts-node src/bin/backup.ts --help
usage: backup.ts [-h] [-v] [-d BACKUPDIR] [-n BACKUPNAME] [-s NAMESPACE] projectId bucket
Backup Datastore.
positional arguments:
projectId Datastore project ID
bucket GCS bucket to store backup
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-d BACKUPDIR, --backupDir BACKUPDIR
prefix/dir within bucket
-n BACKUPNAME, --backupName BACKUPNAME
name of backup (default: autogenerated)
-s NAMESPACE, --namespace NAMESPACE
datastore namespace
Please provide `GOOGLE_APPLICATION_CREDENTIALS` via the Environment!
% export GOOGLE_APPLICATION_CREDENTIALS=~/sampleproj-b0a74af0545e.json
% yarn ts-node src/bin/backup.ts sampleproj samplebucket-tmp
ℹ backup to gs://hudora-tmp/bak/20211221T212651-sampleproj
ℹ Dumping datastore sampleproj
✔ 2 Kinds
2 kinds ℹ Dumping NumberingAncestor, NumberingItem
2 kinds ℹ Dumping to gs://samplebucket-tmp/bak/20211221T212651-sampleproj-0
2 kinds ✔ Dumping finished 27150 records (4.04 MB) in 51s
2 kinds ✔ written gs://samplebucket-tmp/bak/20211221T212651-sampleproj-0/20211221T212651-sampleproj-0.overall_export_metadata
✨ Done in 54.16s.
This generates a single folder structure under gs://samplebucket-tmp/bak/20211221T212651-sampleproj-0
. For projects with more than 100 entities more folders will be generated.
There is one subfolder per kind containing the Entities encoded in LevelDB Format.
import { Datastore } from '@google-cloud/datastore';
import { dumpAllKinds } from '../lib/datastore-backup';
await dumpAllKinds(new Datastore({ projectId: 'sampleproj' }), 'samplebucket-tmp')
FAQs
Programatic Backup of Google Cloud Datastore
The npm package datastore-backup receives a total of 14 weekly downloads. As such, datastore-backup popularity was classified as not popular.
We found that datastore-backup demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Deno 2.2 enhances Node.js compatibility, improves dependency management, adds OpenTelemetry support, and expands linting and task automation for developers.
Security News
React's CRA deprecation announcement sparked community criticism over framework recommendations, leading to quick updates acknowledging build tools like Vite as valid alternatives.
Security News
Ransomware payment rates hit an all-time low in 2024 as law enforcement crackdowns, stronger defenses, and shifting policies make attacks riskier and less profitable.