github.com/ronoaldo/aetools
Package aetools helps writting, testing and analysing Google App Engine applications. The aetools package implements a simple API to export the entity data from Datastore as a JSON stream, as well as load a JSON stream back into the Datastore. This can be used as a simple way to express state into a unit test, to backup a development environment state that can be shared with team members, or to make quick batch changes to data offline, like setting up configuration entities via Remote API. The goal is to provide both an API and a set of executable tools that uses that API, allowing for maximum flexibility. The functions Load, LoadJSON, Dump and DumpJSON operate using JSON data that represents datastore entities. Each entity is mapped to a JSON Object, where each entity property name is an Object atribute, and each property value is the corresponding Object atribute value. The property value is encoded using a JSON primitive, when possible. When the primitives are not sufficient to represent the property value, a JSON Object with the attributes "type" and "value" is used. The "type" attribute is a Datastore type, and value is a json-primitive serialization of that value. For instance, Blobs are encoded as a base64 JSON string, and time.Time values are encoded using the time.RFC3339 layout, also as strings. Datastore Keys are aways encoded as a JSON Array that represents the Key Path, including ancestors, but without the application ID. This is done to allow the entity key to be more readable and to be application independent. Currently, they don't support namespaces. Multiple properties are represented as a JSON Array of values described above. Unindexed properties are aways JSON objects with the "indexed" attribute set to false. This format is intended to make use of the JSON types as much as possible, so an entity can be easily represented as a text file, suitable for read or SCM checkin. The exported data format can also be used as an alternative way to export from Datastore, and then load the results right into other service, such as Google BigQuery or MongoDB. The package aetools/bundle contains a sample webapp to help you manage and stream datastore entities into BigQuery. The bundle uses the aetools/bigquerysync functions to infer an usefull schema from datastore statistics, and sync your entity data into BigQuery. The command aetools/remote_api is a Remote API client that exposes the Load and Dump functions to make backup and restore of development environment state quick and easy. This tool can also help setting up Q.A. or Production apps, but should be used with care.
Readme
import "ronoaldo.gopkg.net/aetools"
The aetools package help you test and analyse Google App Engine Applications
by providing a simple API to export datastore endities as JSON files as well as
load them back into the Datastore.
import "ronoaldo.gopkg.net/aetools/bigquerysync"
The aetools/bigquerysync package provides Datastore to Bigquery synchronization
functions, allowing you to sync your data from Datastore to Bigquery, using the
recomended aproach of a non-conciliated data table and a conciliated table as
described in this document.
The aetools/aeremote command is a simple CLI to interact with the Google Cloud
Datastore, currently via the App Engine Remote API.
The aetools/bundle package contains a ready-to-use Google App Engine webapp
providing handlers to create tables and sync the Datastore directly to Bigquery,
using the aetools and aetools/bigquerysync packages.
The aetools/vmproxy package contains a simple utility to allow on-demand,
low-traffic handlers to dispatch requests to a reusable compute engine
instance.
FAQs
Package aetools helps writting, testing and analysing Google App Engine applications. The aetools package implements a simple API to export the entity data from Datastore as a JSON stream, as well as load a JSON stream back into the Datastore. This can be used as a simple way to express state into a unit test, to backup a development environment state that can be shared with team members, or to make quick batch changes to data offline, like setting up configuration entities via Remote API. The goal is to provide both an API and a set of executable tools that uses that API, allowing for maximum flexibility. The functions Load, LoadJSON, Dump and DumpJSON operate using JSON data that represents datastore entities. Each entity is mapped to a JSON Object, where each entity property name is an Object atribute, and each property value is the corresponding Object atribute value. The property value is encoded using a JSON primitive, when possible. When the primitives are not sufficient to represent the property value, a JSON Object with the attributes "type" and "value" is used. The "type" attribute is a Datastore type, and value is a json-primitive serialization of that value. For instance, Blobs are encoded as a base64 JSON string, and time.Time values are encoded using the time.RFC3339 layout, also as strings. Datastore Keys are aways encoded as a JSON Array that represents the Key Path, including ancestors, but without the application ID. This is done to allow the entity key to be more readable and to be application independent. Currently, they don't support namespaces. Multiple properties are represented as a JSON Array of values described above. Unindexed properties are aways JSON objects with the "indexed" attribute set to false. This format is intended to make use of the JSON types as much as possible, so an entity can be easily represented as a text file, suitable for read or SCM checkin. The exported data format can also be used as an alternative way to export from Datastore, and then load the results right into other service, such as Google BigQuery or MongoDB. The package aetools/bundle contains a sample webapp to help you manage and stream datastore entities into BigQuery. The bundle uses the aetools/bigquerysync functions to infer an usefull schema from datastore statistics, and sync your entity data into BigQuery. The command aetools/remote_api is a Remote API client that exposes the Load and Dump functions to make backup and restore of development environment state quick and easy. This tool can also help setting up Q.A. or Production apps, but should be used with care.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Cyber extortion in the US and Canada hit record levels in 2023, with ransomware attacks surging and median ransom demands skyrocketing, though fewer companies are choosing to pay ransoms.
Security News
Ecma TC39 is meeting this week and has moved key ECMAScript proposals forward, advancing Deferred Import Evaluation, Error.isError(), RegExp Escaping, and Promise.try to the next stages.
Security News
Researchers have demonstrated that teams of LLM agents can exploit zero-day vulnerabilities with a 53% success rate, and the costs of using AI to do so are rapidly becoming more affordable than hiring a human penetration tester.