github.com/mongodb/anser
Package anser provides a document transformation and processing tool to support data migrations. The anser.Application is the primary interface in which migrations are defined and executed. Applications are constructed with a list of MigrationGenerators, and relevant operations. Then the Setup method configures the application, with an anser.Environment, which sets up and collects dependency information. Finally, the Run method executes the migrations in two phases: first by generating migration jobs, and finally by running all migration jobs. The ordering of migrations is derived from the dependency information between generators and the jobs that they generate. When possible jobs are executed in parallel, but the execution of migration operations is a property of the queue object configured in the anser.Environment. The anser package provides a custom amboy/dependency.Manager object, which allows migrations to express dependencies to other migrations. The State() method ensures that all migration IDs specified as edges are satisfied before reporting as "ready" for work. Anser provides the Environment interface, with a global instance accessible via the exported GetEnvironment() function to provide access to runtime configuration state: database connections; amboy.Queue objects, and registries for task implementations. The Environment is an interface: you can build a mock, or use one provided for testing purposes by anser (coming soon). Generators create migration operations and are the first step in an anser Migration. They are supersets of amboy.Job interfaces. The current limitation is that the generated jobs must be stored within the implementation of the generator job, which means they must either all fit in memory *or* be serializable independently (e.g. fit in the 16mb document limit if using a MongoDB backed queue.)
Readme
anser -- Database Migration ToolkitAnser is a toolkit for managing evolving data sets for applications. It focuses on on-line data transformations and providing higher-level tools to support data modeling and access.
For the Evergreen project, anser allows us to treat these routine data migrations, data back fills, and retroactively changing the schema of legacy data as part of application code rather than one-off shell scripts.
In general, anser migrations have a two-phase approach. First a generator runs with some configuration and an input query to collect input documents and creation migration jobs. Then, the output of these generators, are executed in parallel
You can define generators either directly in your own code, or you can use the configuration-file based approach for a more flexible approach.
Concepts
There are three major types of migrations:
- ``simple``: these migrations perform their transformations using
MongoDB's update syntax. Use these migrations for very basic
migrations, particularly when you want to throttle the rate of
migrations and avoid the use of larger difficult-to-index
multi-updates.
- ``manual``: these migrations call a user-defined function on a
``bson.RawDoc`` representation of the document to migrate. Use these
migrations for more complex transformations or those migrations
that you want to write in application code.
- ``stream``: these migrations are similar to manual migrations;
however, they pass a database session *and* an iterator to all
documents impacted by the migration. These jobs offer ultimate
flexibility.
Internally these jobs execute using amboy infrastructure and make it
possible to express dependencies between migrations. Additionally the
`MovingAverageRateLimitedWorkers
<https://godoc.org/github.com/mongodb/amboy/pool#NewMovingAverageRateLimitedWorkers>`_
and `SimpleRateLimitingWorkers
<https://godoc.org/github.com/mongodb/amboy/pool#NewSimpleRateLimitedWorkers>`_
were developed to support anser migrations, as well as the `adaptive
ordering local queue
<https://godoc.org/github.com/mongodb/amboy/queue#NewAdaptiveOrderedLocalQueue>`_
which respects dependency-driven ordering.
Considerations
While it's possible to do any kind of migration with anser, we have found the following properties to be useful to keep in mind when building migrations:
Write your migration implementations so that they are idempotent so that it's possible to run them multiple times with the same effect.
Ensure that generator queries are supported by indexes, otherwise the generator processes will force collection scans.
Rate-Limiting, provided by configuring the underlying amboy infrastructure, focuses on limiting the number of migration (or generator) jobs executed, rather than limiting the jobs based on their impact.
Use batch limits. Generators have limits to control the number of jobs that they will produce. This is particularly useful for tests, but may have adverse effects on job dependency, particularly if logical migrations are split across more than one generator function.
Anser uses Go modules. To download the modules ::
make mod-tidy
Please consult the godoc for most usage. Most of the API is in the top level package <https://godoc.org/github.com/mongodb/anser>; however,
please do also consider the model <https://godoc.org/github.com/mongodb/anser/model>
and bsonutil <https://godoc.org/github.com/mongodb/anser/bsonutil>_ package.
Additionally you can use the interfaces db <https://godoc.org/github.com/mongodb/anser/db>_
package as a wrapper for mgo <https://godoc.org/github.com/mongodb/anser>_ to access
MongoDB which allows you to use mocks <https://godoc.org/github.com/mongodb/anser/mocks>_ as needed for
testing without depending on a running database instance.
Please file feature requests and bug reports in the EVG project <https://jira.mongodb.com/browse/EVG>_ of the MongoDB Jira
instance. This is also the place to file related amboy and grip
requests.
Future anser development will focus on supporting additional migration workflows, supporting additional MongoDB and BSON utilities, and providing tools to support easier data-life-cycle management.
FAQs
Package anser provides a document transformation and processing tool to support data migrations. The anser.Application is the primary interface in which migrations are defined and executed. Applications are constructed with a list of MigrationGenerators, and relevant operations. Then the Setup method configures the application, with an anser.Environment, which sets up and collects dependency information. Finally, the Run method executes the migrations in two phases: first by generating migration jobs, and finally by running all migration jobs. The ordering of migrations is derived from the dependency information between generators and the jobs that they generate. When possible jobs are executed in parallel, but the execution of migration operations is a property of the queue object configured in the anser.Environment. The anser package provides a custom amboy/dependency.Manager object, which allows migrations to express dependencies to other migrations. The State() method ensures that all migration IDs specified as edges are satisfied before reporting as "ready" for work. Anser provides the Environment interface, with a global instance accessible via the exported GetEnvironment() function to provide access to runtime configuration state: database connections; amboy.Queue objects, and registries for task implementations. The Environment is an interface: you can build a mock, or use one provided for testing purposes by anser (coming soon). Generators create migration operations and are the first step in an anser Migration. They are supersets of amboy.Job interfaces. The current limitation is that the generated jobs must be stored within the implementation of the generator job, which means they must either all fit in memory *or* be serializable independently (e.g. fit in the 16mb document limit if using a MongoDB backed queue.)
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
The June TC39 meeting wrapped up this week with eight proposals moving on to the next stage. Here's a quick roundup of the features that the committee approved to advance.
Security News
Cyber extortion in the US and Canada hit record levels in 2023, with ransomware attacks surging and median ransom demands skyrocketing, though fewer companies are choosing to pay ransoms.
Security News
Ecma TC39 is meeting this week and has moved key ECMAScript proposals forward, advancing Deferred Import Evaluation, Error.isError(), RegExp Escaping, and Promise.try to the next stages.