ipfs-car

ipfs-car 🚘✨⬢

Convert files to content-addressable archives (.car) and back

Description

ipfs-car is a library and CLI tool to pack & unpack files from Content Addressable aRchives (CAR) file. A thin wrapper over @ipld/car and unix-fs.

Content-addressable archives store data as blocks (a sequence of bytes) each prefixed with the Content ID (CID) derived from the hash of the data; typically in a file with a .car extension.

Use ipfs-car to pack your files into a .car; a portable, verifiable, IPFS compatible archive.

$ ipfs-car pack path/to/files --output my-files.car

or unpack files from a .car, and verify that every block matches it's CID

$ ipfs-car unpack my-files.car --output path/to/write/to

Fetch and locally verify files from a IPFS gateway over http

curl "https://ipfs.io/ipfs/bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu?format=car" | ipfs-car unpack -o images

Install

# install it as a dependency
$ npm i ipfs-car

# OR use the cli without installing via `npx`
$ npx ipfs-car --help

Usage

Pack files into a .car

# write a content addressed archive to stdout.
$ ipfs-car pack path/to/file/or/dir
# note: CAR data streamed to stdout will not have roots set in CAR header!

# specify the car file name.
$ ipfs-car pack path/to/files --output path/to/write/a.car

# by default, ipfs-car will wrap files in an IPFS directory.
# use --no-wrap to avoid this.
$ ipfs-car pack path/to/file --no-wrap --output path/to/write/a.car

Unpack files from a .car

# unpack files to a specific path.
$ ipfs-car unpack path/to/my.car --output /path/to/unpack/files/to

# unpack a specific root.
$ ipfs-car unpack path/to/my.car --root <cid1>

# unpack files from a .car on stdin.
$ cat path/to/my.car | ipfs-car unpack

Show the files and directories in a .car

# show the files and directories.
$ ipfs-car ls path/to/my.car

# show the files and directories, their CIDs and byte sizes.
$ ipfs-car ls path/to/my.car --verbose

Show the root CIDs in a .car

# show the CID roots found in the CAR header.
$ ipfs-car roots path/to/my.car

# show the CID roots found implicitly from the blocks in the file.
$ ipfs-car roots --implicit path/to/my.car

Show the block CIDs in a .car

# show the CIDs for all the blocks.
$ ipfs-car blocks path/to/my.car

Get other information about a CAR

# generate CID for a CAR.
$ ipfs-car hash path/to/my.car

API

To pack files into content-addressable archives, you can use the following:

• createFileEncoderStream a factory function for creating a ReadableStream that encodes a single file into DAG Blocks.
• createDirectoryEncoderStream a factory function for creating a ReadableStream for encoding a directory of files into DAG Blocks.
• CAREncoderStream a TransformStream sub-class that you can write Blocks to and read Uint8Array CAR file data from.

To unpack content-addressable archives to files, you should use @ipld/car and ipfs-unixfs-exporter modules.

Examples

Basic single file pack

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])
const carStream = createFileEncoderStream(file).pipeThrough(new CAREncoderStream())

// carStream.pipeTo(somewhereWritable)

Directory pack to file system in Node.js

import { Writable } from 'stream'
import { createDirectoryEncoderStream, CAREncoderStream } from 'ipfs-car'
import { filesFromPaths } from 'files-from-path'

const files = await filesFromPaths(process.argv.slice(2))

await createDirectoryEncoderStream(files)
  .pipeThrough(new CAREncoderStream())
  .pipeTo(Writable.toWeb(process.stdout))

Usage: node script.js file0 file1 dir0 > my.car.

Obtaining the root CID

The root CID is the final block generated by the file/directory encoder stream. Use a transform stream to record the CID of the last block generated:

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])
let rootCID

await createFileEncoderStream(file)
  .pipeThrough(new TransformStream({
    transform (block, controller) {
      rootCID = block.cid
      controller.enqueue(block)
    }
  }))
  .pipeThrough(new CAREncoderStream())
  .pipeTo(new WritableStream())

console.log(rootCID.toString())

Adding root CIDs to the CAR header

If you need root CIDs in the CAR header, there are two approaches you can use:

1. Buffer all the DAG blocks, then encode with known root:

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])
const blocks = []

// buffer the output
await createFileEncoderStream(file)
  .pipeTo(new WritableStream({ write: b => blocks.push(b) }))

const rootCID = blocks.at(-1).cid
const blockStream = new ReadableStream({
  pull (controller) {
    if (blocks.length) {
      controller.enqueue(blocks.shift())
    } else {
      controller.close()
    }
  }
})

await blockStream
  .pipeThrough(new CAREncoderStream([rootCID])) // pass root to CAR encoder
  .pipeTo(new WritableStream())

1. Write to disk with placeholder CID, then update after DAG is completely generated (Note: Node.js only):

import fs from 'fs'
import { Writable } from 'stream'
import { CarWriter } from '@ipld/car/writer'
import { CID } from 'multiformats/cid'
import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

// Root CID written in CAR file header before it is updated with the real root CID.
const placeholderCID = CID.parse('bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi')

const file = new Blob(['Hello ipfs-car!'])
let rootCID

await createFileEncoderStream(file)
  .pipeThrough(new TransformStream({
    transform (block, controller) {
      rootCID = block.cid
      controller.enqueue(block)
    }
  }))
  .pipeThrough(new CAREncoderStream(placeholderCID))
  .pipeTo(Writable.toWeb(fs.createWriteStream('path/to/my.car')))

// update roots in CAR header
const fd = await fs.promises.open(opts.output, 'r+')
await CarWriter.updateRootsInFile(fd, [rootCID])
await fd.close()

Unpacking files from a CAR

This functionality is not provided by this library, but is easy to do with @ipld/car and ipfs-unixfs-exporter modules:

import { CarIndexedReader } from '@ipld/car/indexed-reader'
import { recursive as exporter } from 'ipfs-unixfs-exporter'

const reader = await CarIndexedReader.fromFile('path/to/my.car')
const roots = await reader.getRoots()

const entries = exporter(roots[0], {
  async get (cid) {
    const block = await reader.get(cid)
    return block.bytes
  }
})

for await (const entry of entries) {
  if (entry.type === 'file' || entry.type === 'raw') {
    console.log('file', entry.path, entry.content)
  } else if (entry.type === 'directory') {
    console.log('directory', entry.path)
  }
}

Contributing

Feel free to join in. All welcome. Open an issue!

License

Dual-licensed under MIT + Apache 2.0

Changelog

1.2.0 (2024-01-22)

Features

• streaming sha256 CAR hash (#162) (f1601a0)