migrate-to-w3up

migrate-to-w3up ⁂

Migrate files to web3.storage.

Supported Migration Sources

• https://old.web3.storage/
  • bring a WEB3_TOKEN environment variable from https://old.web3.storage/tokens/

Please file an issue to suggest a migration source that you would use.

Installation

npm install -g migrate-to-w3up

Usage

⚠️ Warning! ⚠️

• if a migration errors part of the way through
  • you can't undo the partial migration
  • retrying the migration will start from the beginning, but anything already migrated should be super fast because it will just verify that the w3up space already has an Upload with the right cid.
• this can migrate a lot of data rapidly! be sure you really want to add everything from the migration source to the destination.
  • to be extra careful, use filter mode and export your uploads from your source to a file. inspect the file to make sure it only contains uploads you want to migrate. Then pipe that file to migrate-to-w3up filter mode.
• the space you migrate to must have a w3 storage provider attached. This usually means the storage provider will charge the account that added the storage provider to the space for any migrated files.

Easy mode (using the migration wizard 🧙‍♀️)

migrate-to-w3up

You will then see a series of prompts asking you:

• where you want to migrate to, e.g. pick a w3up space
• where you want to migrate from, e.g.
  • old.web3.storage uploads list from WEB3_TOKEN environment variable, if set and user confirms
  • old.web3.storage list from WEB3_TOKEN that user inputs into prompt

Advanced usage

Migrate w/ log

migrate-to-w3up --log /tmp/migrate-to-w3up.$(date +%s).log will run the migration and write migration events to the provided logfile. This includes UploadMigrationSuccess and UploadMigrationFailure events. The latter are logged along with the source upload that could not be migrated, and this allows the log file to serve as a source of uploads to be migrated in a second migration run to retry any failures.

# set this to a space did
MIGRATION_LOG_1="/tmp/migrate-to-w3up.$(date +%s).log"
migrate-to-w3up --log "$MIGRATION_LOG_1" --space "$W3_SPACE"
# wait quite some time

Retry Failures from Log

Because the log contains good records of any failures, you can use it to do a second migration run of any uploads that failed to migrate.

# MIGRATION_LOG_1 should be set from before
MIGRATION_LOG_2="/tmp/migrate-to-w3up.$(date +%s).log"

# retry migrating any uploads from UploadMigrationFailure
migrate-to-w3up log get-uploads-from-failures "$MIGRATION_LOG_1" | migrate-to-w3up --log "$MIGRATION_LOG_2" --space "$W3_SPACE"

# check for failures from second log. rinse and repeat?
migrate-to-w3up log get-uploads-from-failures "$MIGRATION_LOG_2"

Unix filter

You can also use migrate-to-w3up as a unix filter.

Just pipe in ndjson of old.web3.storage Uploads objects.

# gets current w3cli space (requires jq).
# copypasta from `w3 space ls` to pick another one,
# or do `w3 space use <space>` first
W3_SPACE=$(w3 space info --json | jq '.did' -r)

# define a command to get uploads as migration source
alias w32023-export='npx @web3-storage/w3@latest list --json'

migrate-to-w3up --space="$W3_SPACE" \
< <(w32023-export) \
| tee -a /tmp/migrate-to-w3up.$(date +%s).log
# include the previous line only if you want a logfile

# this also works
# jq optional but useful for pretty printing
# https://jqlang.github.io/jq/
w32023-export | migrate-to-w3up --space="$W3_SPACE" | jq

Migrate a single CAR part

Runs a single store/add invocation with the provided CAR link and show the output (this is mostly for w3up debuggers).

# base32 multihash CAR link
migrate-to-w3up $W3_SPACE store/add --link ciqgrph67ihh4imym4pl6d4xlnfhgxycdr4hcm6g6ucnzuxzqsorpsq

# CIDv1 CAR link
migrate-to-w3up $W3_SPACE store/add --link bagbaieranc6p56qopyqzqzy6x4hzow2konpqehdyoez4n5ie3tjptbe5c7fa

Example

$ migrate-to-w3up $W3_SPACE store/add --link ciqgrph67ihh4imym4pl6d4xlnfhgxycdr4hcm6g6ucnzuxzqsorpsq
{
  "ok": {
    "link": {
      "/": "bagbaieranc6p56qopyqzqzy6x4hzow2konpqehdyoez4n5ie3tjptbe5c7fa"
    },
    "with": "did:key:z6MkvSvQTdY87Rkazj3JiTquqQzjCAgWBuoWodPSHY6josAp",
    "status": "done",
    "allocated": 0
  }
}

How it Works

migrate-to-w3up/from-w32023

• exports a migrate function that runs a migration, returning an AsyncIterable of MigratedUpload<W32023Upload> that includes ucanto receipts for every request sent to w3up as part of the migration.

migrate-to-w3up.js

• this is the main bin script that runs when the package is globally installed and someone runs migrate-to-w3up
• it reads cli flags to build options for migrate. If some options aren't provided as flags, and there is an interactive terminal, prompt the terminal for the options required to start a migration, e.g. selecting a source of uploads and destination (e.g. space DID).
• it calls migrate from w32023-to-w3up.js and interprets the results as CLI output
  • it writes ndjson to stdout, and any errors/warnings to stderr.
  • it is recommended to pipe the output of this to a file so it can be explored later to check receipts or explore logs for debugging. But you can also pipe to jq to pretty print the ndjson output for ad-hoc exploration.