md5dumper

Features of MD5Dumper

MD5Dumper is a tool that:

• Can generate a canonical representation of every file under a directory. (the Fsmap format)
• Can minify Fsmap representations to reduce interchanged data size.
• Can summarize Fsmap to reduce interchanged data size for initial checksum.
• Compare minified Fsmap and local directory Fsmap, and detect missing/corrupted/extra files

Motivation behind MD5Dumper

MD5Dumper aims to automatise file system synchronization between distinct ends. Intended specially for auto-patch servers.

Due to the way auto-patch servers interact with clients, data traffic on each check is really important to optimize. MD5Dumper attempts to minimize the size of data interchanged between those checks.

Anatomy of an Fsmap

An Fsmap is a canonical representation of a directory. It is generated by recursively traversing a directory in a sorted way and creating MD5 hashes of every file's content + file's relative path.

declare type Fsmap = FsmapEntry[];

declare type FsmapEntry = {
  index: number; // Index to ensure traverse order
  path?: string; // Relative file system path
  md5: string; // h(relativePath + content)
};

How should Auto-patcher Servers use it?

• Once development is done, store built artifacts under a directory. (Let's call it ./target for this example)
• Generate fsmap of ./target dir and store it under ./autopatcher/build.fsmap. This file will be used to deserialize on the server side, client won't need this one.
• Minify fsmap of ./target and store it under ./autopatcher/build.min.fsmap
• Summarize fsmap of ./target and store it under ./autopatcher/summary.md5
• Serve those last 2 files for clients to fetch; build.min.fsmap and summary.md5. (They will need them to perform checksum :p)
• Fsmap maps Indices to relative file paths. Serve files and their relative paths by indices using the fsmap as a lookup table! Easy peasy!

How should Auto-patcher Clients use it?

• Generate fsmap of the local directory
• Fetch summary.md5 from the server and compare with local fsmap's summary.
• If they match, then the client is up to date! Else, fetch build.min.fsmap from the server.
• Check for the differences between build.min.fsmap and local fsmap. The diff functionality of MD5Dumper will yield "invalid local files" and "missing file indices".
• Delete invalid local files.
• Request every missing file by their indices. Server shall know which file is represented by the indice! Server will respond you with the file blob and it's relative path!
• The client should be up to date after all the steps! Easy peasy!