🎩 You're Invited:Meet the Socket team at Black Hat in Las Vegas, August 3-6.RSVP →
Sign In

decmpfs

Package Overview
Dependencies
Maintainers
0
Versions
3
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

decmpfs - cargo Package Compare versions

Comparing version
0.1.0
 to
0.1.2
+12
build.rs
//! Reserve Mach-O header slack in the `decmpfs-stub` binary so the `exe` packer
//! can splice a new `LC_SEGMENT_64` (`SMOL/__DECMPFS`) into it in place, without
//! relinking. Only the stub bin, only on macOS, only under the `exe` feature —
//! every other build is untouched.
fn main() {
let exe_feature = std::env::var_os("CARGO_FEATURE_EXE").is_some();
let macos = std::env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("macos");
if exe_feature && macos {
println!("cargo::rustc-link-arg-bin=decmpfs-stub=-Wl,-headerpad,0x1000");
}
}
+29
src/bin/decmpfs-stub.rs
//! The self-replacing stub binary (feature `exe`). A packer injects a
//! zstd-compressed payload executable into a copy of THIS binary; on first run
//! the copy decompresses the payload, writes it back to disk FS-compressed,
//! atomically replaces itself, and execs it. Every later run is the plain
//! materialized executable — this stub is gone.
//!
//! A bare, unpacked `decmpfs-stub` carries no payload, so `self_replace_and_exec`
//! returns `Ok(false)` and there is nothing to run: it exits non-zero with a
//! diagnostic rather than pretend-succeeding.
fn main() {
let argv: Vec<String> = std::env::args().collect();
match decmpfs::exe::self_replace_and_exec(&argv) {
// Unix: exec replaces the process image, so Ok(true) never actually
// returns here. Windows spawns the materialized sibling and returns true.
Ok(true) => {}
Ok(false) => {
eprintln!(
"decmpfs-stub: no packed payload in this binary — pack one in with \
decmpfs::exe::pack_executable_with_stub before running it."
);
std::process::exit(2);
}
Err(e) => {
eprintln!("decmpfs-stub: {e}");
std::process::exit(1);
}
}
}
+597
src/exe/inject.rs
//! Object-format surgery: splice the `SMOL/__DECMPFS` section into a Mach-O
//! stub (signable, then ad-hoc re-signed via the system `codesign`), or append
//! the `[payload][hash][len][MAGIC]` footer to an ELF/PE stub (their loaders
//! don't enforce a signature to `execve`, so no surgery is needed).
//!
//! Ported from napi-rs `crates/napi-compress/src/inject.rs` (the proven Mach-O
//! segment-insertion + slack/LINKEDIT-shift logic), with the crate-based
//! ad-hoc signer replaced by a `codesign -s - -f` shell-out so the crate stays
//! dep-lean.
//!
//! Every structural offset is walked at runtime from the load commands — the
//! stub is rebuilt with header slack (`-headerpad`) that guarantees room for
//! the new segment command, not a fixed offset. The byte layout matches a
//! binject(LIEF)-produced reference: segment `filesize` is the unpadded body
//! length, `vmsize` is page-rounded; section `size` is the body length,
//! `offset`/`fileoff` is the old `__LINKEDIT` fileoff; W^X `initprot` =
//! `maxprot` = `VM_PROT_READ`.
use std::path::Path;
const MH_MAGIC_64: u32 = 0xfeed_facf;
const LC_SEGMENT_64: u32 = 0x19;
const LC_SYMTAB: u32 = 0x02;
const LC_DYSYMTAB: u32 = 0x0b;
const LC_DYLD_INFO: u32 = 0x22;
const LC_DYLD_INFO_ONLY: u32 = 0x8000_0022;
const LC_FUNCTION_STARTS: u32 = 0x26;
const LC_DATA_IN_CODE: u32 = 0x29;
const LC_CODE_SIGNATURE: u32 = 0x1d;
const LC_DYLD_CHAINED_FIXUPS: u32 = 0x8000_0034;
const LC_DYLD_EXPORTS_TRIE: u32 = 0x8000_0033;
const CPU_TYPE_ARM64: u32 = 0x0100_000c;
const MACH_HEADER_64_SIZE: usize = 32;
/// `cmd,cmdsize,segname[16],vmaddr,vmsize,fileoff,filesize,maxprot,initprot,nsects,flags`.
const SEGMENT_COMMAND_64_SIZE: usize = 72;
/// `sectname[16],segname[16],addr,size,offset,align,reloff,nreloc,flags,reserved1..3`.
const SECTION_64_SIZE: usize = 80;
const NEW_LC_SIZE: usize = SEGMENT_COMMAND_64_SIZE + SECTION_64_SIZE; // 152
/// `VM_PROT_READ` only. An injected segment MUST be read-only: RWX (0x07) makes
/// dyld refuse to mmap the bundle on dlopen (EACCES) even with a valid signature.
const VM_PROT_READ: u32 = 0x01;
fn u32_le(bytes: &[u8], off: usize) -> Result<u32, String> {
bytes
.get(off..off + 4)
.and_then(|s| s.try_into().ok())
.map(u32::from_le_bytes)
.ok_or_else(|| format!("truncated u32 at offset {off}"))
}
fn u64_le(bytes: &[u8], off: usize) -> Result<u64, String> {
bytes
.get(off..off + 8)
.and_then(|s| s.try_into().ok())
.map(u64::from_le_bytes)
.ok_or_else(|| format!("truncated u64 at offset {off}"))
}
fn put_u32(bytes: &mut [u8], off: usize, value: u32) -> Result<(), String> {
bytes
.get_mut(off..off + 4)
.ok_or_else(|| format!("truncated u32 write at offset {off}"))?
.copy_from_slice(&value.to_le_bytes());
Ok(())
}
fn put_u64(bytes: &mut [u8], off: usize, value: u64) -> Result<(), String> {
bytes
.get_mut(off..off + 8)
.ok_or_else(|| format!("truncated u64 write at offset {off}"))?
.copy_from_slice(&value.to_le_bytes());
Ok(())
}
fn round_up(value: u64, align: u64) -> u64 {
if align == 0 {
return value;
}
value.div_ceil(align) * align
}
/// One linkedit-pointing field to bump: the absolute byte offset of a `u32`
/// file-offset field within the (post-splice) command stream.
struct OffsetField {
at: usize,
}
/// The structural anchors found by walking the load commands once.
struct Layout {
page_size: u64,
/// Byte offset of the `__LINKEDIT` segment command (splice point).
linkedit_lc_off: usize,
linkedit_fileoff: u64,
linkedit_vmaddr: u64,
/// `LC_CODE_SIGNATURE`, if present: (command byte offset, sig dataoff).
code_sig: Option<CodeSig>,
/// First `__TEXT` section file offset — the slack boundary the new LC must fit
/// under (everything before it is header + load commands).
first_section_offset: u64,
end_of_lc: usize,
/// Linkedit-pointing `u32` file-offset fields, by their byte offset in the
/// ORIGINAL command stream (caller re-bases by +NEW_LC after the splice).
linkedit_pointers: Vec<OffsetField>,
}
struct CodeSig {
lc_off: usize,
dataoff: u64,
}
/// A NUL-padded fixed-width name slot equals `want`.
fn name_eq(slot: &[u8], want: &[u8]) -> bool {
slot.len() >= want.len()
&& &slot[..want.len()] == want
&& slot[want.len()..].iter().all(|&b| b == 0)
}
/// Walk the mach_header_64 + load commands once, recording every anchor the
/// surgery touches. Refuses anything but a single-arch 64-bit LE Mach-O.
fn read_layout(bytes: &[u8]) -> Result<Layout, String> {
if u32_le(bytes, 0)? != MH_MAGIC_64 {
return Err("not a 64-bit little-endian Mach-O (bad magic)".to_string());
}
let cputype = u32_le(bytes, 4)?;
let page_size: u64 = if cputype == CPU_TYPE_ARM64 {
0x4000
} else {
0x1000
};
let ncmds = u32_le(bytes, 16)?;
let mut linkedit_lc_off: Option<usize> = None;
let mut linkedit_fileoff = 0u64;
let mut linkedit_vmaddr = 0u64;
let mut code_sig: Option<CodeSig> = None;
let mut first_section_offset = u64::MAX;
let mut linkedit_pointers: Vec<OffsetField> = Vec::new();
let mut off = MACH_HEADER_64_SIZE;
for _ in 0..ncmds {
let cmd = u32_le(bytes, off)?;
let cmdsize = u32_le(bytes, off + 4)? as usize;
if cmdsize < 8 || off + cmdsize > bytes.len() {
return Err(format!(
"malformed load command at {off} (cmdsize {cmdsize})"
));
}
match cmd {
LC_SEGMENT_64 => {
let segname = bytes
.get(off + 8..off + 24)
.ok_or_else(|| format!("truncated segname at {off}"))?;
let fileoff = u64_le(bytes, off + 40)?;
let nsects = u32_le(bytes, off + 64)?;
if name_eq(segname, b"__LINKEDIT") {
linkedit_lc_off = Some(off);
linkedit_fileoff = fileoff;
linkedit_vmaddr = u64_le(bytes, off + 24)?;
}
// Track the smallest section file offset across every segment — the
// header-slack ceiling (load commands must not overrun the first
// mapped section's bytes).
let mut soff = off + SEGMENT_COMMAND_64_SIZE;
for _ in 0..nsects {
let sect_off = u32_le(bytes, soff + 48)? as u64;
// offset == 0 marks a zero-fill section (__bss/__thread_bss); skip it.
if sect_off != 0 && sect_off < first_section_offset {
first_section_offset = sect_off;
}
soff = soff
.checked_add(SECTION_64_SIZE)
.ok_or_else(|| "section table offset overflow".to_string())?;
}
}
LC_DYLD_INFO | LC_DYLD_INFO_ONLY => {
// rebase_off@8, bind_off@16, weak_bind_off@24, lazy_bind_off@32, export_off@40.
for field in [8, 16, 24, 32, 40] {
linkedit_pointers.push(OffsetField { at: off + field });
}
}
LC_SYMTAB => {
// symoff@8, stroff@16.
linkedit_pointers.push(OffsetField { at: off + 8 });
linkedit_pointers.push(OffsetField { at: off + 16 });
}
LC_DYSYMTAB => {
// tocoff@32, modtaboff@40, extrefsymoff@48, indirectsymoff@56,
// extreloff@64, locreloff@72 — all linkedit-relative file offsets.
for field in [32, 40, 48, 56, 64, 72] {
linkedit_pointers.push(OffsetField { at: off + field });
}
}
LC_FUNCTION_STARTS | LC_DATA_IN_CODE | LC_DYLD_CHAINED_FIXUPS | LC_DYLD_EXPORTS_TRIE => {
// linkedit_data_command: dataoff@8.
linkedit_pointers.push(OffsetField { at: off + 8 });
}
LC_CODE_SIGNATURE => {
// linkedit_data_command: dataoff is a u32 file offset at +8 (NOT a u64).
code_sig = Some(CodeSig {
lc_off: off,
dataoff: u32_le(bytes, off + 8)? as u64,
});
}
_ => {}
}
off = off
.checked_add(cmdsize)
.ok_or_else(|| "load command offset overflow".to_string())?;
}
let linkedit_lc_off =
linkedit_lc_off.ok_or_else(|| "no __LINKEDIT segment to anchor the new section".to_string())?;
if first_section_offset == u64::MAX {
return Err("no mapped section to bound the header slack".to_string());
}
Ok(Layout {
page_size,
linkedit_lc_off,
linkedit_fileoff,
linkedit_vmaddr,
code_sig,
first_section_offset,
end_of_lc: off,
linkedit_pointers,
})
}
/// Build the 152-byte `LC_SEGMENT_64` + one `section_64` for `SMOL/__DECMPFS`.
fn build_segment_lc(
body_len: u64,
delta: u64,
fileoff: u64,
vmaddr: u64,
) -> Result<Vec<u8>, String> {
let mut lc = vec![0u8; NEW_LC_SIZE];
// segment_command_64
put_u32(&mut lc, 0, LC_SEGMENT_64)?;
put_u32(&mut lc, 4, NEW_LC_SIZE as u32)?;
lc.get_mut(8..12)
.ok_or_else(|| "new LC too short for segname".to_string())?
.copy_from_slice(b"SMOL"); // segname (NUL-padded)
put_u64(&mut lc, 24, vmaddr)?; // vmaddr
put_u64(&mut lc, 32, delta)?; // vmsize (page-rounded)
put_u64(&mut lc, 40, fileoff)?; // fileoff
put_u64(&mut lc, 48, body_len)?; // filesize (unpadded body)
put_u32(&mut lc, 56, VM_PROT_READ)?; // maxprot
put_u32(&mut lc, 60, VM_PROT_READ)?; // initprot (W^X)
put_u32(&mut lc, 64, 1)?; // nsects
put_u32(&mut lc, 68, 0)?; // flags
// section_64 at +72
let s = SEGMENT_COMMAND_64_SIZE;
lc.get_mut(s..s + 9)
.ok_or_else(|| "new LC too short for sectname".to_string())?
.copy_from_slice(b"__DECMPFS"); // sectname
lc.get_mut(s + 16..s + 20)
.ok_or_else(|| "new LC too short for section segname".to_string())?
.copy_from_slice(b"SMOL"); // segname
put_u64(&mut lc, s + 32, vmaddr)?; // addr
put_u64(&mut lc, s + 40, body_len)?; // size (unpadded body)
put_u32(&mut lc, s + 48, fileoff as u32)?; // offset
put_u32(&mut lc, s + 52, 2)?; // align 2^2 = 4
// reloff/nreloc/flags/reserved1..3 stay 0
Ok(lc)
}
/// Inject `section_body` into `stub`, dispatching on the stub's object format.
/// Mach-O gets the signable `SMOL/__DECMPFS` segment splice (`section_body` is
/// the caller's [`super::section::build_section_payload`] output); ELF/PE need
/// no surgery — `section_body` is the caller's [`super::section::build_footer`]
/// output and is simply appended at EOF. Returns the modified bytes (Mach-O
/// still UNSIGNED — the caller runs [`resign`]).
#[allow(dead_code)] // wired in the replace stage
pub(crate) fn inject_payload(stub: &[u8], section_body: &[u8]) -> Result<Vec<u8>, String> {
match stub.first().copied() {
Some(0xcf) if stub.get(0..4) == Some(&MH_MAGIC_64.to_le_bytes()) => {
inject_macho(stub, section_body)
}
Some(0x7f) if stub.get(0..4) == Some(b"\x7fELF") => Ok(append_footer(stub, section_body)),
Some(0x4d) if stub.get(0..2) == Some(b"MZ") => Ok(append_footer(stub, section_body)),
_ => Err("unrecognized stub format: not a 64-bit LE Mach-O, ELF, or PE".to_string()),
}
}
/// ELF/PE: their loaders map from program headers / section headers already on
/// disk and never look past EOF for anything the loader cares about, so the
/// footer built by [`super::section::build_footer`] rides as a plain append —
/// no header field moves.
fn append_footer(stub: &[u8], footer: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(stub.len() + footer.len());
out.extend_from_slice(stub);
out.extend_from_slice(footer);
out
}
/// Mach-O: splice a READ-only `SMOL/__DECMPFS` `LC_SEGMENT_64`. Returns the modified
/// bytes (still unsigned — caller re-signs via [`resign`]).
fn inject_macho(stub: &[u8], section_body: &[u8]) -> Result<Vec<u8>, String> {
let layout = read_layout(stub)?;
// 1. Slack guard: the new 152-byte LC must fit between END_OF_LC and the first
// mapped section. The stub is linked with -headerpad,0x1000 to guarantee it.
let slack = (layout.first_section_offset as usize)
.checked_sub(layout.end_of_lc)
.ok_or_else(|| {
"first mapped section precedes the end of load commands (corrupt layout)".to_string()
})?;
if slack < NEW_LC_SIZE {
return Err(format!(
"header slack {slack} < {NEW_LC_SIZE} bytes for the new segment command; \
rebuild the stub with -headerpad,0x1000"
));
}
let body_len = section_body.len() as u64;
let delta = round_up(body_len, layout.page_size);
let new_fileoff = layout.linkedit_fileoff;
let new_vmaddr = layout.linkedit_vmaddr;
let linkedit_start = layout.linkedit_fileoff as usize;
// Exclude the old signature bytes — they trail __LINKEDIT and the signer
// regenerates them. Without a signature, __LINKEDIT runs to EOF.
let linkedit_end = match &layout.code_sig {
Some(sig) => sig.dataoff as usize,
None => stub.len(),
};
if linkedit_end < linkedit_start {
return Err("code signature precedes __LINKEDIT (corrupt layout)".to_string());
}
// Mach-O section fileoff + every LINKEDIT-relative offset are 32-bit. `linkedit_end
// + delta` bounds the widest post-splice offset (the new section's fileoff and each
// `current + delta` bump all fall at or below it), so guard that sum against u32::MAX
// before the `as u32` casts below — release builds have overflow-checks off, so a
// >4 GiB payload would otherwise truncate silently and corrupt the LINKEDIT tables.
if (linkedit_end as u64)
.checked_add(delta)
.is_none_or(|end| end > u64::from(u32::MAX))
{
return Err(format!(
"injected payload too large: file offset {linkedit_end} + {delta} exceeds the u32 Mach-O limit"
));
}
let end_after_new_lc = layout
.end_of_lc
.checked_add(NEW_LC_SIZE)
.ok_or_else(|| "end-of-load-commands offset overflow".to_string())?;
let stub_before_linkedit = stub
.get(0..layout.linkedit_lc_off)
.ok_or_else(|| "__LINKEDIT LC offset out of range".to_string())?;
let stub_lc_tail = stub
.get(layout.linkedit_lc_off..layout.end_of_lc)
.ok_or_else(|| "load-command tail out of range".to_string())?;
let stub_headerpad_tail = stub
.get(end_after_new_lc..linkedit_start)
.ok_or_else(|| "headerpad tail out of range".to_string())?;
let stub_linkedit_body = stub
.get(linkedit_start..linkedit_end)
.ok_or_else(|| "__LINKEDIT body out of range".to_string())?;
// 2. Assemble the new file so NOTHING before __LINKEDIT moves its file offset.
// The new 152-byte LC is written into the header slack: the load-command
// bytes [linkedit_lc_off, END_OF_LC) shift forward by 152, consuming 152 of
// the headerpad gap; the first mapped section and every byte up to
// __LINKEDIT keep their original file offset. __LINKEDIT's body then slides
// down by DELTA only (the page-rounded section content occupies the old
// __LINKEDIT file region).
//
// [0, linkedit_lc_off) header + LCs before __LINKEDIT's LC
// new_lc (152) the SMOL/__DECMPFS segment command
// [linkedit_lc_off, END_OF_LC) __LINKEDIT's LC + the LCs after it
// [END_OF_LC+152, linkedit_start) remaining headerpad + all mapped bytes
// section_body + zero-pad to DELTA the injected section content
// [linkedit_start, linkedit_end) __LINKEDIT body (sans old signature)
let new_lc = build_segment_lc(body_len, delta, new_fileoff, new_vmaddr)?;
let pad_len = (delta - body_len) as usize;
let mut out: Vec<u8> = Vec::with_capacity(stub.len() + delta as usize);
out.extend_from_slice(stub_before_linkedit);
out.extend_from_slice(&new_lc);
out.extend_from_slice(stub_lc_tail);
// Headerpad after the (now-larger) command stream, shrunk by the 152 bytes the
// new LC consumed, so the first mapped section stays at its original offset.
out.extend_from_slice(stub_headerpad_tail);
out.extend_from_slice(section_body);
out.resize(
out
.len()
.checked_add(pad_len)
.ok_or_else(|| "padded section length overflow".to_string())?,
0,
);
out.extend_from_slice(stub_linkedit_body);
// 3. Header: ncmds += 1, sizeofcmds += NEW_LC. (The strip below nets these back
// down by the code-signature command.)
let ncmds = u32_le(&out, 16)?;
put_u32(&mut out, 16, ncmds + 1)?;
let sizeofcmds = u32_le(&out, 20)?;
put_u32(&mut out, 20, sizeofcmds + NEW_LC_SIZE as u32)?;
// 4. Shift __LINKEDIT's own fileoff + vmaddr by DELTA (its LC now sits at
// linkedit_lc_off + NEW_LC, after the new segment command). When the old
// signature was excluded, shrink filesize/vmsize to the bytes that remain on
// disk (up to where the signature began) — else the segment claims more than
// the file holds and the signer's parser rejects it; the signer re-extends.
let le_lc = layout
.linkedit_lc_off
.checked_add(NEW_LC_SIZE)
.ok_or_else(|| "__LINKEDIT LC offset overflow after splice".to_string())?;
let le_fileoff = u64_le(&out, le_lc + 40)?;
put_u64(&mut out, le_lc + 40, le_fileoff + delta)?;
let le_vmaddr = u64_le(&out, le_lc + 24)?;
put_u64(&mut out, le_lc + 24, le_vmaddr + delta)?;
if let Some(sig) = &layout.code_sig {
let remaining = sig
.dataoff
.checked_sub(layout.linkedit_fileoff)
.ok_or_else(|| "code signature precedes __LINKEDIT fileoff".to_string())?;
put_u64(&mut out, le_lc + 48, remaining)?; // filesize
put_u64(&mut out, le_lc + 32, round_up(remaining, layout.page_size))?; // vmsize
}
// 5. Bump every linkedit-pointing file offset by DELTA. A field whose command
// sits at/after __LINKEDIT's LC had its byte position shifted +NEW_LC when the
// new LC was written before __LINKEDIT; a field before it keeps its position.
// Skip zeros (an absent table).
for field in &layout.linkedit_pointers {
let at = if field.at >= layout.linkedit_lc_off {
field
.at
.checked_add(NEW_LC_SIZE)
.ok_or_else(|| "linkedit-pointer offset overflow after splice".to_string())?
} else {
field.at
};
let current = u32_le(&out, at)?;
if current != 0 {
put_u32(&mut out, at, current + delta as u32)?;
}
}
// 6. Strip LC_CODE_SIGNATURE in place (it is the LAST command, so zeroing its
// bytes + decrementing the header counts removes it WITHOUT shifting any file
// offset — a splice here would move __LINKEDIT and re-break step 5). Its
// trailing __LINKEDIT bytes were already excluded in step 2; the signer
// re-adds a correct command into the freed command-stream slack.
if let Some(sig) = &layout.code_sig {
let sig_lc = sig
.lc_off
.checked_add(NEW_LC_SIZE)
.ok_or_else(|| "code-signature LC offset overflow after splice".to_string())?;
let sig_cmdsize = u32_le(&out, sig_lc + 4)? as usize;
let ncmds = u32_le(&out, 16)?;
put_u32(&mut out, 16, ncmds - 1)?;
let sizeofcmds = u32_le(&out, 20)?;
put_u32(&mut out, 20, sizeofcmds - sig_cmdsize as u32)?;
let zero_range = out
.get_mut(sig_lc..sig_lc + sig_cmdsize)
.ok_or_else(|| "code-signature LC out of range after splice".to_string())?;
zero_range.fill(0);
}
Ok(out)
}
/// Ad-hoc re-sign a materialized Mach-O at `path` via the system `codesign`
/// (`codesign -s - -f <path>`). macOS-only; a no-op `Ok(())` elsewhere — ELF/PE
/// loaders enforce no signature, so the appended footer needs none.
#[cfg(not(target_os = "macos"))]
#[allow(dead_code)] // wired in the replace stage
pub(crate) fn resign(_path: &Path) -> Result<(), String> {
Ok(())
}
#[cfg(target_os = "macos")]
#[allow(dead_code)] // wired in the replace stage
pub(crate) fn resign(path: &Path) -> Result<(), String> {
let status = std::process::Command::new("codesign")
.arg("-s")
.arg("-")
.arg("-f")
.arg(path)
.status()
.map_err(|source| format!("spawn codesign for {}: {source}", path.display()))?;
if !status.success() {
return Err(format!("codesign failed for {} ({status})", path.display()));
}
Ok(())
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
use crate::exe::section::{build_footer, find_footer};
#[cfg(target_os = "macos")]
use crate::exe::section::{build_section_payload, find_section, parse_section_payload};
/// A minimal two-segment 64-bit Mach-O: `__TEXT` (one section, anchoring the
/// header-slack boundary) then `__LINKEDIT` (the splice point). Big enough for
/// `read_layout`/`inject_macho` to walk for real, small enough to stay a unit
/// test — unlike [`crate::exe::section::synthetic_object_with_section`] (built
/// for `find_section`'s read path, which never inspects `__LINKEDIT`), this
/// fixture carries the segment `inject_macho` requires to splice at all.
#[cfg(target_os = "macos")]
fn minimal_macho64_with_linkedit(
first_section_offset: u32,
linkedit_fileoff: u64,
linkedit_body: &[u8],
) -> Vec<u8> {
let linkedit_lc_off = MACH_HEADER_64_SIZE + NEW_LC_SIZE; // 32 + 152 = 184
let linkedit_len = linkedit_body.len() as u64;
let mut m = vec![0u8; linkedit_fileoff as usize + linkedit_body.len()];
// mach_header_64.
put_u32(&mut m, 0, MH_MAGIC_64).expect("header magic");
put_u32(&mut m, 16, 2).expect("ncmds"); // __TEXT, __LINKEDIT
put_u32(&mut m, 20, (NEW_LC_SIZE + SEGMENT_COMMAND_64_SIZE) as u32).expect("sizeofcmds");
// __TEXT segment_command_64 + one section_64, at MACH_HEADER_64_SIZE.
let text = MACH_HEADER_64_SIZE;
put_u32(&mut m, text, LC_SEGMENT_64).expect("text cmd");
put_u32(&mut m, text + 4, NEW_LC_SIZE as u32).expect("text cmdsize");
m[text + 8..text + 14].copy_from_slice(b"__TEXT");
put_u32(&mut m, text + 64, 1).expect("text nsects");
let text_sect = text + SEGMENT_COMMAND_64_SIZE;
m[text_sect..text_sect + 6].copy_from_slice(b"__text");
m[text_sect + 16..text_sect + 22].copy_from_slice(b"__TEXT");
put_u32(&mut m, text_sect + 48, first_section_offset).expect("text section offset");
// __LINKEDIT segment_command_64 (no sections).
put_u32(&mut m, linkedit_lc_off, LC_SEGMENT_64).expect("linkedit cmd");
put_u32(&mut m, linkedit_lc_off + 4, SEGMENT_COMMAND_64_SIZE as u32).expect("linkedit cmdsize");
m[linkedit_lc_off + 8..linkedit_lc_off + 18].copy_from_slice(b"__LINKEDIT");
put_u64(
&mut m,
linkedit_lc_off + 24,
0x1_0000_0000 + linkedit_fileoff,
)
.expect("linkedit vmaddr");
put_u64(&mut m, linkedit_lc_off + 40, linkedit_fileoff).expect("linkedit fileoff");
put_u64(&mut m, linkedit_lc_off + 48, linkedit_len).expect("linkedit filesize");
m[linkedit_fileoff as usize..linkedit_fileoff as usize + linkedit_body.len()]
.copy_from_slice(linkedit_body);
m
}
#[cfg(target_os = "macos")]
#[test]
fn macho_injection_round_trips_through_find_section() {
let body = build_section_payload(0x0123_4567_89ab_cdef, b"the-zstd-payload-bytes");
let stub = minimal_macho64_with_linkedit(512, 600, b"LINKEDIT-CONTENT");
let out = inject_payload(&stub, &body).expect("inject");
let raw = find_section(&out).expect("section found");
let got = parse_section_payload(raw).expect("parses");
assert_eq!(got.content_hash, 0x0123_4567_89ab_cdef);
assert_eq!(got.payload, b"the-zstd-payload-bytes");
}
#[test]
fn elf_and_pe_injection_append_the_footer_verbatim() {
// The ELF/PE loader maps from program/section headers already on disk and
// never looks past EOF, so injection is a pure append: the footer bytes
// `find_footer` locates at the tail of the grown image must equal the
// footer `build_footer` produced, byte for byte.
let footer = build_footer(0xfeed_face_dead_beef, b"the-zstd-bytes");
let mut elf_stub = vec![0u8; 64];
elf_stub[0..4].copy_from_slice(b"\x7fELF");
let elf_out = inject_payload(&elf_stub, &footer).expect("inject elf");
assert_eq!(
find_footer(&elf_out).expect("footer found"),
footer.as_slice()
);
let mut pe_stub = vec![0u8; 64];
pe_stub[0..2].copy_from_slice(b"MZ");
let pe_out = inject_payload(&pe_stub, &footer).expect("inject pe");
assert_eq!(
find_footer(&pe_out).expect("footer found"),
footer.as_slice()
);
}
#[test]
fn dispatch_rejects_unknown_format() {
assert!(inject_payload(b"not an object file", b"x").is_err());
}
#[cfg(not(target_os = "macos"))]
#[test]
fn resign_is_a_no_op_off_macos() {
assert!(resign(Path::new("/does/not/exist")).is_ok());
}
}
+352
src/exe/mod.rs
//! Self-replacing executable packing (feature `exe`).
//!
//! [`pack_executable`] turns a real executable `E` into a stub `E'` carrying a
//! zstd-compressed copy of `E` in a signable `SMOL/__DECMPFS` section (macOS) or
//! an EOF footer (ELF/PE). On first run `E'` calls [`self_replace_and_exec`]:
//! it decompresses the payload, writes `E` back to disk **FS-compressed** via
//! [`crate::compress_bytes`], atomically renames over `argv[0]`, re-signs on
//! macOS, and `execve`s the materialized binary. Every later run is native — the
//! stub is gone, replaced by the real (smaller-on-disk) executable.
//!
//! The section/footer wire format is owned by [`section`]; the object surgery by
//! [`inject`]; the runtime swap by [`replace`]. All three are private; the crate
//! surface is the two functions below plus [`PackOutcome`].
use std::path::Path;
use crate::{Error, Gate};
mod inject;
mod replace;
mod section;
/// The result of packing an executable. Only `Err` is a hard failure.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum PackOutcome {
/// Packed: `before` = original size, `after` = the stub's size on disk.
Packed { before: u64, after: u64 },
/// The gate excluded the input (by glob/size) — nothing written.
SkippedGate,
}
/// Host-side packer: read `src`, compress it, inject the payload into a stub, and
/// write the self-replacing executable to `dest`. `gate` filters by glob/size the
/// same way [`crate::compress_bytes`] does; a gate miss returns
/// [`PackOutcome::SkippedGate`] without writing.
///
/// The stub bytes are the CURRENT executable's own image by default (a decmpfs
/// binary that links the `exe` feature IS the stub); pass an explicit stub via
/// [`pack_executable_with_stub`] to cross-pack.
pub fn pack_executable(src: &Path, dest: &Path, gate: &Gate) -> Result<PackOutcome, Error> {
let stub = std::env::current_exe().map_err(|source| Error::Io {
context: "resolve current_exe for the pack stub",
source,
})?;
pack_executable_with_stub(&stub, src, dest, gate)
}
/// [`pack_executable`] with an explicit stub image — the self-replacing runtime
/// binary whose `SMOL/__DECMPFS` section/footer receives the payload.
///
/// `gate` is checked against `dest` (the caller's chosen name, normalized to
/// forward slashes) before anything is read or written — a miss returns
/// [`PackOutcome::SkippedGate`] and touches neither `stub` nor `dest`.
pub fn pack_executable_with_stub(
stub: &Path,
src: &Path,
dest: &Path,
gate: &Gate,
) -> Result<PackOutcome, Error> {
let src_bytes = std::fs::read(src).map_err(|source| Error::Io {
context: "read the source executable to pack",
source,
})?;
let src_len = src_bytes.len() as u64;
let dest_name = dest.to_string_lossy().replace('\\', "/");
if !gate.matches(&dest_name, src_len) {
return Ok(PackOutcome::SkippedGate);
}
let content_hash = section::fnv1a64(&src_bytes);
let zstd_payload =
zstd::stream::encode_all(src_bytes.as_slice(), 19).map_err(|source| Error::Io {
context: "zstd-compress the source executable",
source,
})?;
let stub_bytes = std::fs::read(stub).map_err(|source| Error::Io {
context: "read the stub executable",
source,
})?;
// Mach-O carries the payload in a signable `SMOL/__DECMPFS` section body;
// ELF/PE carry it in an EOF footer. `inject::inject_payload`'s ELF/PE arm is
// a verbatim append (no structural surgery), so the footer must already be
// fully formed here — only the Mach-O arm gets its own wrapper.
let section_body = if is_macho64(&stub_bytes) {
section::build_section_payload(content_hash, &zstd_payload)
} else {
section::build_footer(content_hash, &zstd_payload)
};
let packed = inject::inject_payload(&stub_bytes, &section_body).map_err(|message| Error::Io {
context: "inject the packed payload into the stub",
source: std::io::Error::other(message),
})?;
std::fs::write(dest, &packed).map_err(|source| Error::Io {
context: "write the packed executable",
source,
})?;
#[cfg(unix)]
mark_executable(dest)?;
// Re-sign only a Mach-O output — gate on the STUB's format, not the host OS,
// so cross-packing an ELF/PE stub on a macOS host never hands a non-Mach-O
// file (whose trailing footer a future codesign could mutate) to codesign.
// `resign` is itself a no-op off macOS.
if is_macho64(&stub_bytes) {
inject::resign(dest).map_err(|message| Error::Io {
context: "re-sign the packed executable",
source: std::io::Error::other(message),
})?;
}
let after = std::fs::metadata(dest)
.map_err(|source| Error::Io {
context: "stat the packed executable's on-disk size",
source,
})?
.len();
Ok(PackOutcome::Packed {
before: src_len,
after,
})
}
/// A 64-bit little-endian Mach-O magic at the head of `bytes` — the same
/// dispatch [`inject::inject_payload`] makes internally, mirrored here so the
/// right payload wrapper (section body vs. footer) is built before the call and
/// so the runtime swap only hands a Mach-O to `codesign`.
pub(crate) fn is_macho64(bytes: &[u8]) -> bool {
bytes.get(0..4) == Some(&0xfeed_facfu32.to_le_bytes())
}
/// Set the owner/group/world execute bits on `path`, on top of whatever mode
/// it landed with.
#[cfg(unix)]
fn mark_executable(path: &Path) -> Result<(), Error> {
use std::os::unix::fs::PermissionsExt;
let mut perms = std::fs::metadata(path)
.map_err(|source| Error::Io {
context: "read the packed executable's permissions",
source,
})?
.permissions();
perms.set_mode(perms.mode() | 0o111);
std::fs::set_permissions(path, perms).map_err(|source| Error::Io {
context: "set the packed executable's execute bit",
source,
})
}
/// True when `path` is a packed stub — it carries a decmpfs payload (a
/// `SMOL/__DECMPFS` section on Mach-O, an EOF footer on ELF/PE). A plain or
/// already-materialized executable returns `false`.
pub fn contains_payload(path: &Path) -> bool {
section::read_self_section_bytes(path).is_some()
}
/// Runtime entry the packed stub calls from its `main`: resolve self → read the
/// payload → decompress → FS-compress the bytes to disk → atomically replace
/// `argv[0]` → re-sign (macOS) → `execve`. On success it does NOT return (the
/// process image is replaced); `Ok(false)` means "this binary is not a packed
/// stub, run your normal main". `Err` is a genuine I/O / integrity failure.
pub fn self_replace_and_exec(argv: &[String]) -> Result<bool, Error> {
replace::materialize_and_exec(argv)
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
/// A scratch dir unique to one test, cleaned up by the caller via
/// [`std::fs::remove_dir_all`] once the test finishes.
fn scratch_dir(label: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("decmpfs-pack-{label}-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("create scratch dir");
dir
}
#[cfg(target_os = "macos")]
fn synthetic_macho_stub_with_linkedit(
first_section_offset: u32,
linkedit_fileoff: u64,
linkedit_body: &[u8],
) -> Vec<u8> {
// Mirrors `inject`'s own test fixture: a minimal two-segment Mach-O
// (`__TEXT` anchoring the header-slack boundary, `__LINKEDIT` as the
// splice point) big enough for `inject_macho` to walk for real.
const MH_MAGIC_64: u32 = 0xfeed_facf;
const LC_SEGMENT_64: u32 = 0x19;
const MACH_HEADER_64_SIZE: usize = 32;
const SEGMENT_COMMAND_64_SIZE: usize = 72;
const NEW_LC_SIZE: usize = SEGMENT_COMMAND_64_SIZE + 80; // segment + one section_64
let linkedit_lc_off = MACH_HEADER_64_SIZE + NEW_LC_SIZE;
let linkedit_len = linkedit_body.len() as u64;
let mut m = vec![0u8; linkedit_fileoff as usize + linkedit_body.len()];
m[0..4].copy_from_slice(&MH_MAGIC_64.to_le_bytes());
m[16..20].copy_from_slice(&2u32.to_le_bytes()); // ncmds: __TEXT, __LINKEDIT
m[20..24].copy_from_slice(&((NEW_LC_SIZE + SEGMENT_COMMAND_64_SIZE) as u32).to_le_bytes());
let text = MACH_HEADER_64_SIZE;
m[text..text + 4].copy_from_slice(&LC_SEGMENT_64.to_le_bytes());
m[text + 4..text + 8].copy_from_slice(&(NEW_LC_SIZE as u32).to_le_bytes());
m[text + 8..text + 14].copy_from_slice(b"__TEXT");
m[text + 64..text + 68].copy_from_slice(&1u32.to_le_bytes()); // nsects
let text_sect = text + SEGMENT_COMMAND_64_SIZE;
m[text_sect..text_sect + 6].copy_from_slice(b"__text");
m[text_sect + 16..text_sect + 22].copy_from_slice(b"__TEXT");
m[text_sect + 48..text_sect + 52].copy_from_slice(&first_section_offset.to_le_bytes());
m[linkedit_lc_off..linkedit_lc_off + 4].copy_from_slice(&LC_SEGMENT_64.to_le_bytes());
m[linkedit_lc_off + 4..linkedit_lc_off + 8]
.copy_from_slice(&(SEGMENT_COMMAND_64_SIZE as u32).to_le_bytes());
m[linkedit_lc_off + 8..linkedit_lc_off + 18].copy_from_slice(b"__LINKEDIT");
m[linkedit_lc_off + 24..linkedit_lc_off + 32]
.copy_from_slice(&(0x1_0000_0000u64 + linkedit_fileoff).to_le_bytes());
m[linkedit_lc_off + 40..linkedit_lc_off + 48].copy_from_slice(&linkedit_fileoff.to_le_bytes());
m[linkedit_lc_off + 48..linkedit_lc_off + 56].copy_from_slice(&linkedit_len.to_le_bytes());
m[linkedit_fileoff as usize..linkedit_fileoff as usize + linkedit_body.len()]
.copy_from_slice(linkedit_body);
m
}
#[cfg(target_os = "macos")]
#[test]
fn pack_into_a_macho_stub_round_trips_through_read_self_section_bytes() {
let dir = scratch_dir("macho");
let stub_path = dir.join("stub.bin");
let src_path = dir.join("src.bin");
let dest_path = dir.join("dest.bin");
let stub = synthetic_macho_stub_with_linkedit(512, 600, b"LINKEDIT-CONTENT");
std::fs::write(&stub_path, &stub).expect("write stub");
let src_bytes = b"the original executable's bytes, repeated a bit to give zstd something to chew on. the original executable's bytes.".to_vec();
std::fs::write(&src_path, &src_bytes).expect("write src");
let outcome = pack_executable_with_stub(&stub_path, &src_path, &dest_path, &Gate::any())
.expect("pack succeeds");
let Some(after) = (match outcome {
PackOutcome::Packed { before, after } => {
assert_eq!(before, src_bytes.len() as u64);
Some(after)
}
PackOutcome::SkippedGate => None,
}) else {
panic!("Gate::any() must never skip");
};
assert_eq!(
after,
std::fs::metadata(&dest_path).expect("stat dest").len()
);
let got = section::read_self_section_bytes(&dest_path).expect("section found");
assert_eq!(got.content_hash, section::fnv1a64(&src_bytes));
let decompressed = zstd::stream::decode_all(got.payload.as_slice()).expect("zstd decode");
assert_eq!(decompressed, src_bytes);
use std::os::unix::fs::PermissionsExt;
let mode = std::fs::metadata(&dest_path)
.expect("stat dest")
.permissions()
.mode();
assert_ne!(mode & 0o111, 0, "packed executable must be executable");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn pack_into_an_elf_stub_appends_a_footer_that_round_trips() {
let dir = scratch_dir("elf");
let stub_path = dir.join("stub.bin");
let src_path = dir.join("src.bin");
let dest_path = dir.join("dest.bin");
// A minimal ELF-magic stub — `is_macho64` rejects it, so the packer routes
// it through `section::build_footer` + the append-only ELF/PE arm of
// `inject::inject_payload`, regardless of the host this test runs on.
let mut stub = vec![0u8; 64];
stub[0..4].copy_from_slice(b"\x7fELF");
std::fs::write(&stub_path, &stub).expect("write stub");
let src_bytes = b"another synthetic executable payload for the ELF/PE footer path".to_vec();
std::fs::write(&src_path, &src_bytes).expect("write src");
let outcome = pack_executable_with_stub(&stub_path, &src_path, &dest_path, &Gate::any())
.expect("pack succeeds");
assert_eq!(
outcome,
PackOutcome::Packed {
before: src_bytes.len() as u64,
after: (stub.len()
+ section::build_footer(0, &[]).len()
+ zstd::stream::encode_all(src_bytes.as_slice(), 19)
.expect("zstd encode")
.len()) as u64,
}
);
let dest_bytes = std::fs::read(&dest_path).expect("read dest");
let raw_footer = section::find_footer(&dest_bytes).expect("footer found");
// Wire format: `[zstd payload][content_hash u64 LE][payload_len u64 LE][MAGIC]`.
let payload_len = raw_footer
.len()
.checked_sub(24)
.expect("footer long enough");
let payload = &raw_footer[0..payload_len];
let hash_bytes: [u8; 8] = raw_footer[payload_len..payload_len + 8]
.try_into()
.expect("hash slice is 8 bytes");
let content_hash = u64::from_le_bytes(hash_bytes);
assert_eq!(content_hash, section::fnv1a64(&src_bytes));
let decompressed = zstd::stream::decode_all(payload).expect("zstd decode");
assert_eq!(decompressed, src_bytes);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn a_gate_miss_skips_and_writes_nothing() {
let dir = scratch_dir("gate-miss");
let stub_path = dir.join("stub.bin"); // never read: the gate rejects first.
let src_path = dir.join("src.bin");
let dest_path = dir.join("reject.bin");
std::fs::write(&src_path, b"some source bytes").expect("write src");
let gate = Gate::new(Some("*.selected"), None).expect("gate parses");
let outcome = pack_executable_with_stub(&stub_path, &src_path, &dest_path, &gate)
.expect("gate miss is not an error");
assert_eq!(outcome, PackOutcome::SkippedGate);
assert!(!dest_path.exists(), "a gate miss must write nothing");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn is_macho64_recognizes_only_the_64_bit_le_magic() {
assert!(is_macho64(&0xfeed_facfu32.to_le_bytes()));
assert!(!is_macho64(b"\x7fELF"));
assert!(!is_macho64(b"MZ"));
assert!(!is_macho64(b"sho"));
}
}
+320
src/exe/replace.rs
//! The runtime swap: read the packed stub's own payload, decompress, write the
//! real executable back to disk FS-compressed, atomically replace `argv[0]`,
//! re-sign (macOS), and `execve`.
//!
//! A running image can't be overwritten on Windows, so there the swap is
//! deferred: write the materialized binary alongside and schedule a
//! rename-on-next-start (MoveFileEx with MOVEFILE_DELAY_UNTIL_REBOOT is too
//! coarse; the runtime writes a `.pending` sibling and a tiny relauncher).
//!
//! Unix path: `std::os::unix::process::CommandExt::exec` replaces the process
//! image so control never returns on success.
#[cfg(windows)]
use std::path::Path;
use crate::Error;
#[cfg(windows)]
use crate::Gate;
use super::section::SectionData;
/// Read the current executable's own packed payload, if any. `Ok(None)` means
/// this binary is a plain executable (not a packed stub) — the caller runs its
/// normal `main`.
pub(crate) fn read_self_payload() -> Result<Option<SectionData>, Error> {
let exe = std::env::current_exe().map_err(|source| Error::Io {
context: "resolve current_exe to read the self payload",
source,
})?;
Ok(super::section::read_self_section_bytes(&exe))
}
/// Decompress `section`'s zstd payload and verify it against `content_hash`.
/// Pure and unit-testable independently of `current_exe`/the filesystem.
fn decode_verified(section: &SectionData) -> Result<Vec<u8>, Error> {
let raw = zstd::stream::decode_all(section.payload.as_slice()).map_err(|source| Error::Io {
context: "zstd-decompress the self payload",
source,
})?;
if super::section::fnv1a64(&raw) != section.content_hash {
return Err(Error::Io {
context: "verify self payload integrity",
source: std::io::Error::other("content hash mismatch: the packed payload is corrupt"),
});
}
Ok(raw)
}
/// Resolve `current_exe()` and the directory it lives in (the directory the
/// materialized binary must land in too, so the swap stays on one filesystem —
/// a same-filesystem rename is atomic and a same-filesystem `clonefile` is
/// available to the FS-compression writer).
fn resolve_exe_and_dir() -> Result<(std::path::PathBuf, std::path::PathBuf), Error> {
let exe = std::env::current_exe().map_err(|source| Error::Io {
context: "resolve current_exe for the runtime swap",
source,
})?;
let dir = exe
.parent()
.ok_or_else(|| Error::Io {
context: "resolve current_exe's parent directory for the runtime swap",
source: std::io::Error::other("current_exe has no parent directory"),
})?
.to_path_buf();
Ok((exe, dir))
}
/// Best-effort exclusive advisory lock over a stub's self-replace. Concurrent
/// first-loads of the SAME packed binary (a burst of parallel launches right
/// after install) would otherwise EACH run the expensive
/// decode+re-sign+FS-compress+rename; the lock lets one win while the rest wait,
/// then observe the completed swap and exec the real binary. Held for the
/// guard's lifetime: std's file-lock fd is `O_CLOEXEC`, so the lock releases at
/// `execve` (the re-exec'd real binary must not inherit it) and on any early
/// `return` (the guard drops → the `File` closes). A lock we can't create or
/// acquire (a read-only dir, an FS without advisory locks) yields an UNLOCKED
/// guard — the swap stays correct via the atomic rename, only the dedup is lost.
///
/// Unix-only: Windows defers the swap to a `.decmpfs-pending` sibling and spawns
/// it (never overwrites the running image), so its launches are independent and
/// the per-write atomicity already covers concurrency — a lock there would
/// wrongly serialize concurrent launches.
#[cfg(unix)]
struct SelfReplaceLock {
// Kept alive to hold the OS lock; dropped (closed → unlocked) at scope end.
_file: Option<std::fs::File>,
}
#[cfg(unix)]
impl SelfReplaceLock {
fn acquire(dir: &std::path::Path, file_name: &str) -> Self {
let path = dir.join(format!(".{file_name}.decmpfs.lock"));
let file = std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(&path)
.ok();
if let Some(f) = file.as_ref() {
// Blocking exclusive lock; an Err leaves the guard effectively unlocked
// (we fall through — the atomic rename still guarantees correctness).
let _ = f.lock();
}
Self { _file: file }
}
}
/// Materialize + swap + exec. Does not return on success (process image
/// replaced). `Ok(false)` means "this binary is not a packed stub, run your
/// normal main"; `Err` is a genuine I/O / integrity failure.
#[cfg(unix)]
pub(crate) fn materialize_and_exec(argv: &[String]) -> Result<bool, Error> {
use std::os::unix::process::CommandExt;
// Cheap pre-check: a plain executable (no payload) runs its normal main.
if read_self_payload()?.is_none() {
return Ok(false);
}
let (exe, dir) = resolve_exe_and_dir()?;
let file_name = exe
.file_name()
.ok_or_else(|| Error::Io {
context: "resolve current_exe's file name for the runtime swap",
source: std::io::Error::other("current_exe has no file name"),
})?
.to_string_lossy()
.into_owned();
// Serialize concurrent first-loads of this stub (see SelfReplaceLock). Released
// at execve (O_CLOEXEC) or when this guard drops on an early return.
let _lock = SelfReplaceLock::acquire(&dir, &file_name);
// Re-read UNDER the lock: a process that went before us may have already
// swapped the on-disk exe, so its payload section is gone. If so, skip the
// whole materialize and exec the already-real binary directly.
let Some(section) = read_self_payload()? else {
let source = std::process::Command::new(&exe)
.args(argv.get(1..).unwrap_or(&[]))
.exec();
return Err(Error::Io {
context: "exec the already-materialized binary",
source,
});
};
let raw = decode_verified(&section)?;
let temp = dir.join(format!(
".{file_name}.decmpfs-materializing-{}",
std::process::id()
));
// Order matters: write the raw bytes, sign, THEN FS-compress. `codesign`
// rewrites the whole file (embedding the signature), which strips any decmpfs
// compression — so signing must run on the plain bytes and compress_file
// applies decmpfs last. decmpfs is transparent on read, so the signature the
// kernel hands the loader stays valid.
std::fs::write(&temp, &raw).map_err(|source| Error::Io {
context: "write the materialized binary before compressing",
source,
})?;
let mode = std::fs::metadata(&exe)
.map_err(|source| Error::Io {
context: "read current_exe's mode to preserve it on the materialized binary",
source,
})?
.permissions();
std::fs::set_permissions(&temp, mode).map_err(|source| Error::Io {
context: "copy current_exe's mode onto the materialized binary",
source,
})?;
// Re-sign only a Mach-O payload — codesign can't sign a script or other
// non-Mach-O executable, and handing it one is a no-op at best.
#[cfg(target_os = "macos")]
if super::is_macho64(&raw) {
super::inject::resign(&temp).map_err(|message| Error::Io {
context: "re-sign the materialized binary",
source: std::io::Error::other(message),
})?;
}
// FS-compress in place, AFTER signing — the on-disk win the packer promised.
crate::compress_file(&temp)?;
std::fs::rename(&temp, &exe).map_err(|source| Error::Io {
context: "atomically rename the materialized binary over argv[0]",
source,
})?;
let source = std::process::Command::new(&exe)
.args(argv.get(1..).unwrap_or(&[]))
.exec();
// `exec` only returns on failure — success replaces this process image.
Err(Error::Io {
context: "exec the materialized binary",
source,
})
}
/// Windows can't overwrite its own running image, so the swap is deferred: the
/// materialized binary is written to a `.decmpfs-pending` sibling and launched
/// directly. The original stub on disk is untouched — the swap-over-original
/// completes on a later run (a follow-up can add a rename-on-reboot / relauncher
/// so the pending binary takes the original's name and the stub disappears; for
/// now every launch re-materializes and re-execs the pending copy).
#[cfg(windows)]
pub(crate) fn materialize_and_exec(argv: &[String]) -> Result<bool, Error> {
let Some(section) = read_self_payload()? else {
return Ok(false);
};
let raw = decode_verified(&section)?;
let (exe, _) = resolve_exe_and_dir()?;
let mut pending = exe.clone().into_os_string();
pending.push(".decmpfs-pending");
let pending = Path::new(&pending).to_path_buf();
crate::compress_bytes(&pending, &raw, &Gate::any())?;
std::process::Command::new(&pending)
.args(argv.get(1..).unwrap_or(&[]))
.spawn()
.map_err(|source| Error::Io {
context: "spawn the pending materialized binary",
source,
})?;
Ok(true)
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
#[test]
fn decode_verified_round_trips_a_valid_payload() {
let raw = b"hello from the materialized executable".to_vec();
let compressed = zstd::stream::encode_all(raw.as_slice(), 3).expect("zstd encode");
let section = SectionData {
content_hash: super::super::section::fnv1a64(&raw),
payload: compressed,
};
let got = decode_verified(&section).expect("decodes and verifies");
assert_eq!(got, raw);
}
#[test]
fn decode_verified_rejects_a_corrupted_hash() {
let raw = b"hello from the materialized executable".to_vec();
let compressed = zstd::stream::encode_all(raw.as_slice(), 3).expect("zstd encode");
let section = SectionData {
// Flip a bit — the decompressed bytes no longer match this hash.
content_hash: super::super::section::fnv1a64(&raw) ^ 1,
payload: compressed,
};
assert!(decode_verified(&section).is_err());
}
#[test]
fn decode_verified_rejects_a_non_zstd_payload() {
let section = SectionData {
content_hash: 0,
payload: b"not zstd at all".to_vec(),
};
assert!(decode_verified(&section).is_err());
}
#[test]
fn read_self_payload_returns_none_for_a_plain_test_binary() {
// The cargo test binary is a plain executable, not a packed stub, so it
// carries no `SMOL/__DECMPFS` section or EOF footer.
assert!(read_self_payload().expect("reads current_exe").is_none());
}
#[cfg(unix)]
#[test]
fn self_replace_lock_is_exclusive_and_releases_on_drop() {
let dir = std::env::temp_dir().join(format!("decmpfs-srlock-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let name = "toolstub";
let lock_path = dir.join(format!(".{name}.decmpfs.lock"));
{
let _guard = SelfReplaceLock::acquire(&dir, name);
// A distinct handle on the same lock file can't take it while the guard
// holds the exclusive lock (flock conflicts across open descriptions).
let other = std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(&lock_path)
.unwrap();
assert!(
other.try_lock().is_err(),
"lock is held exclusively while the guard is alive"
);
}
// The guard dropped → the File closed → the lock is free again.
let other = std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(&lock_path)
.unwrap();
assert!(
other.try_lock().is_ok(),
"lock released when the guard drops"
);
other.unlock().ok();
std::fs::remove_dir_all(&dir).ok();
}
#[cfg(unix)]
#[test]
fn self_replace_lock_falls_through_when_the_dir_is_unwritable() {
// A lock file we can't create yields an unlocked guard, no panic — the swap
// still works via the atomic rename, just without the dedup.
let guard = SelfReplaceLock::acquire(std::path::Path::new("/no/such/decmpfs/dir"), "x");
drop(guard);
}
}
+338
src/exe/section.rs
//! The `SMOL/__DECMPFS` section / EOF-footer wire format, shared by the packer
//! (write) and the runtime (read-self). One source of truth for the ABI.
//!
//! Section body (Mach-O, signable): `[MAGIC][content_hash u64 LE][zstd payload]`.
//! Footer (ELF/PE, appended): `[zstd payload][content_hash u64 LE][payload_len
//! u64 LE][MAGIC]` at EOF, so the runtime seeks the tail, reads `payload_len`,
//! and validates the trailing magic.
//!
//! Ported from napi-rs `crates/decmpfs/src/section.rs`; parse is hand-rolled,
//! length-guarded, no `object` crate (the crate stays dep-lean + panic=abort).
use std::path::Path;
/// Head of the payload. Distinguishes our section/footer from stray bytes and
/// fails closed on a malformed image.
pub(crate) const SECTION_MAGIC: &[u8; 8] = b"DCMPFSX1";
/// The validated payload extracted from a packed stub.
#[allow(dead_code)] // fields read by the replace stage's consumer
pub(crate) struct SectionData {
/// FNV-1a of the RAW executable — names the materialized file + verifies decode.
pub content_hash: u64,
/// The zstd payload (the compressed raw executable).
pub payload: Vec<u8>,
}
/// Assemble the section body the packer injects (Mach-O path).
#[allow(dead_code)] // wired in the inject stage
pub(crate) fn build_section_payload(content_hash: u64, zstd_payload: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(16 + zstd_payload.len());
out.extend_from_slice(SECTION_MAGIC);
out.extend_from_slice(&content_hash.to_le_bytes());
out.extend_from_slice(zstd_payload);
out
}
/// Parse a Mach-O section body `[MAGIC][hash][payload]`. Pure, unit-testable.
#[cfg(any(target_os = "macos", test))]
pub(crate) fn parse_section_payload(raw: &[u8]) -> Option<SectionData> {
if raw.len() < 16 || &raw[0..8] != SECTION_MAGIC {
return None;
}
Some(SectionData {
content_hash: u64::from_le_bytes(raw[8..16].try_into().ok()?),
payload: raw[16..].to_vec(),
})
}
/// Assemble the EOF footer the packer appends (ELF/PE path): the payload, then
/// its content hash, then its length (so the runtime can find the payload's
/// start by walking back from EOF), then the trailing magic.
#[allow(dead_code)] // wired in the inject stage
pub(crate) fn build_footer(content_hash: u64, zstd_payload: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(zstd_payload.len() + 24);
out.extend_from_slice(zstd_payload);
out.extend_from_slice(&content_hash.to_le_bytes());
out.extend_from_slice(&(zstd_payload.len() as u64).to_le_bytes());
out.extend_from_slice(SECTION_MAGIC);
out
}
/// Parse the footer region located by [`find_footer`]: trailing MAGIC, then
/// `payload_len`, then `content_hash`, then the payload fills everything before
/// it. Pure, unit-testable against [`build_footer`] without a real ELF/PE.
#[allow(dead_code)] // used on non-macOS hosts in production; exercised by tests everywhere
fn parse_footer_payload(raw: &[u8]) -> Option<SectionData> {
if raw.len() < 24 {
return None;
}
let magic_off = raw.len().checked_sub(8)?;
if &raw[magic_off..] != SECTION_MAGIC {
return None;
}
let len_off = magic_off.checked_sub(8)?;
let payload_len = u64::from_le_bytes(raw.get(len_off..len_off + 8)?.try_into().ok()?) as usize;
let hash_off = len_off.checked_sub(8)?;
if hash_off != payload_len {
return None;
}
Some(SectionData {
content_hash: u64::from_le_bytes(raw.get(hash_off..hash_off + 8)?.try_into().ok()?),
payload: raw.get(0..hash_off)?.to_vec(),
})
}
/// FNV-1a (64-bit) — names the cache/materialize target without decoding.
#[allow(dead_code)] // wired in the replace stage
pub(crate) fn fnv1a64(bytes: &[u8]) -> u64 {
let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
for &b in bytes {
hash ^= b as u64;
hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
}
hash
}
/// Read `self_path` from disk and extract its packed payload: the Mach-O
/// `SMOL/__DECMPFS` section on macOS, the EOF footer everywhere else. `None`
/// if the file can't be read, has no such section/footer, or it's malformed —
/// the caller treats that as "not a packed stub".
#[allow(dead_code)] // wired in the replace stage
pub(crate) fn read_self_section_bytes(self_path: &Path) -> Option<SectionData> {
let bytes = std::fs::read(self_path).ok()?;
#[cfg(target_os = "macos")]
{
parse_section_payload(find_section(&bytes)?)
}
#[cfg(not(target_os = "macos"))]
{
parse_footer_payload(find_footer(&bytes)?)
}
}
/// A null-padded fixed-width name slot equals `want` (Mach-O's 16-byte
/// `sectname`/`segname`): the leading bytes match and the remainder is NUL.
#[cfg(target_os = "macos")]
fn name_eq(slot: &[u8], want: &[u8]) -> bool {
slot.len() >= want.len()
&& &slot[..want.len()] == want
&& slot[want.len()..].iter().all(|&b| b == 0)
}
// ---------------------------------------------------------------------------
// macOS — 64-bit little-endian Mach-O (every darwin target is x86_64/arm64,
// both LE). Walk load commands → SMOL segment → __DECMPFS section.
// ---------------------------------------------------------------------------
#[cfg(target_os = "macos")]
pub(crate) fn find_section(bytes: &[u8]) -> Option<&[u8]> {
const MH_MAGIC_64: u32 = 0xfeed_facf;
const LC_SEGMENT_64: u32 = 0x19;
if u32::from_le_bytes(bytes.get(0..4)?.try_into().ok()?) != MH_MAGIC_64 {
return None;
}
let ncmds = u32::from_le_bytes(bytes.get(16..20)?.try_into().ok()?);
// mach_header_64 is 32 bytes; load commands follow.
let mut off = 32usize;
for _ in 0..ncmds {
let cmd = u32::from_le_bytes(bytes.get(off..off + 4)?.try_into().ok()?);
let cmdsize = u32::from_le_bytes(bytes.get(off + 4..off + 8)?.try_into().ok()?) as usize;
if cmdsize < 8 {
return None;
}
if cmd == LC_SEGMENT_64 && name_eq(bytes.get(off + 8..off + 24)?, b"SMOL") {
// segment_command_64: cmd,cmdsize,segname[16],vmaddr,vmsize,fileoff,
// filesize,maxprot,initprot,nsects,flags — nsects at off+64, sections at off+72.
let nsects = u32::from_le_bytes(bytes.get(off + 64..off + 68)?.try_into().ok()?);
let mut soff = off + 72;
for _ in 0..nsects {
// section_64: sectname[16],segname[16],addr(8),size(8),offset(4),...(80 total).
if name_eq(bytes.get(soff..soff + 16)?, b"__DECMPFS") {
let size = u64::from_le_bytes(bytes.get(soff + 40..soff + 48)?.try_into().ok()?) as usize;
let offset =
u32::from_le_bytes(bytes.get(soff + 48..soff + 52)?.try_into().ok()?) as usize;
return bytes.get(offset..offset.checked_add(size)?);
}
soff = soff.checked_add(80)?;
}
}
off = off.checked_add(cmdsize)?;
}
None
}
// ---------------------------------------------------------------------------
// ELF/PE — no per-format object surgery needed: the payload rides an appended
// EOF footer `[payload][content_hash][payload_len][MAGIC]`. Read the fixed
// trailer back-to-front and slice the payload out by its declared length.
// ---------------------------------------------------------------------------
// Not target-gated: the trailer bytes carry no OS-specific structure, so the
// parser is portable. [`read_self_section_bytes`] is what restricts it to
// non-macOS hosts at the call site.
#[allow(dead_code)] // used on non-macOS hosts in production; exercised by tests everywhere
pub(crate) fn find_footer(bytes: &[u8]) -> Option<&[u8]> {
const TRAILER: usize = 24; // content_hash(8) + payload_len(8) + MAGIC(8)
if bytes.len() < TRAILER {
return None;
}
let magic_off = bytes.len().checked_sub(8)?;
if &bytes[magic_off..] != SECTION_MAGIC {
return None;
}
let len_off = magic_off.checked_sub(8)?;
let payload_len = u64::from_le_bytes(bytes.get(len_off..len_off + 8)?.try_into().ok()?) as usize;
let hash_off = len_off.checked_sub(8)?;
let payload_start = hash_off.checked_sub(payload_len)?;
bytes.get(payload_start..bytes.len())
}
/// Build a minimal Mach-O image carrying one `SMOL/__DECMPFS` section whose
/// bytes are exactly `body`, so [`find_section`]'s load-command/section-header
/// arithmetic is validated against a known layout without a real binary. The
/// real-binary round-trip is covered by the packer-phase integration test.
#[cfg(all(test, target_os = "macos"))]
pub(crate) fn synthetic_object_with_section(body: &[u8]) -> Vec<u8> {
// header(32) + one LC_SEGMENT_64 (72 segment + 80 section = 152) = 184, then
// the section body appended at offset 184.
let mut m = vec![0u8; 184];
m[0..4].copy_from_slice(&0xfeed_facfu32.to_le_bytes()); // MH_MAGIC_64
m[16..20].copy_from_slice(&1u32.to_le_bytes()); // ncmds
m[32..36].copy_from_slice(&0x19u32.to_le_bytes()); // LC_SEGMENT_64
m[36..40].copy_from_slice(&152u32.to_le_bytes()); // cmdsize
m[40..44].copy_from_slice(b"SMOL"); // segname
m[96..100].copy_from_slice(&1u32.to_le_bytes()); // nsects (off 32 + 64)
let s = 104; // sections start at off 32 + 72
m[s..s + 9].copy_from_slice(b"__DECMPFS"); // sectname
m[s + 16..s + 20].copy_from_slice(b"SMOL"); // section's segname
m[s + 40..s + 48].copy_from_slice(&(body.len() as u64).to_le_bytes()); // size
m[s + 48..s + 52].copy_from_slice(&184u32.to_le_bytes()); // offset
m.extend_from_slice(body);
m
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
#[test]
fn build_and_parse_section_payload_round_trip() {
let body = build_section_payload(0xfeed_face_dead_beef, b"the-zstd-bytes");
let got = parse_section_payload(&body).expect("parses");
assert_eq!(got.content_hash, 0xfeed_face_dead_beef);
assert_eq!(got.payload, b"the-zstd-bytes");
// Empty payload is still a valid (header-only) body.
assert!(parse_section_payload(&build_section_payload(0, b"")).is_some());
}
#[test]
fn parse_section_payload_rejects_bad_input() {
assert!(parse_section_payload(b"short").is_none());
assert!(parse_section_payload(b"WRONGMAG\0\0\0\0\0\0\0\0").is_none());
}
#[test]
fn build_and_find_footer_round_trip() {
let footer = build_footer(0xfeed_face_dead_beef, b"the-zstd-bytes");
// The footer trails arbitrary leading bytes, exactly as it does at the end
// of a real ELF/PE executable.
let mut bytes = b"leading executable bytes before the footer".to_vec();
bytes.extend_from_slice(&footer);
let raw = find_footer(&bytes).expect("footer found");
let got = parse_footer_payload(raw).expect("parses");
assert_eq!(got.content_hash, 0xfeed_face_dead_beef);
assert_eq!(got.payload, b"the-zstd-bytes");
// Empty payload is still a valid (header-only) footer.
let empty = build_footer(0, b"");
assert!(parse_footer_payload(find_footer(&empty).expect("footer found")).is_some());
}
#[test]
fn find_footer_rejects_malformed_or_short_input() {
assert!(find_footer(b"short").is_none());
assert!(find_footer(&[0u8; 23]).is_none());
// Trailing magic corrupted.
let mut bad_magic = build_footer(1, b"x");
let last = bad_magic.len() - 1;
bad_magic[last] = b'!';
assert!(find_footer(&bad_magic).is_none());
// payload_len lies about the payload's true length.
let mut bad_len = build_footer(1, b"xyz");
let len_off = bad_len.len() - 16;
bad_len[len_off..len_off + 8].copy_from_slice(&999u64.to_le_bytes());
assert!(find_footer(&bad_len).is_none());
}
#[test]
fn fnv1a64_known_vector() {
// FNV-1a 64-bit offset basis is the hash of the empty string.
assert_eq!(fnv1a64(b""), 0xcbf2_9ce4_8422_2325);
// Published FNV-1a 64-bit test vector for the single byte "a".
assert_eq!(fnv1a64(b"a"), 0xaf63_dc4c_8601_ec8c);
}
#[cfg(target_os = "macos")]
#[test]
fn find_section_reads_from_a_synthetic_macho() {
let body = build_section_payload(0x0123_4567_89ab_cdef, b"the-zstd-payload-bytes");
let obj = synthetic_object_with_section(&body);
let raw = find_section(&obj).expect("section found");
let got = parse_section_payload(raw).expect("parses");
assert_eq!(got.content_hash, 0x0123_4567_89ab_cdef);
assert_eq!(got.payload, b"the-zstd-payload-bytes");
}
#[cfg(target_os = "macos")]
#[test]
fn find_section_rejects_non_macho() {
assert!(find_section(b"not a binary at all").is_none());
assert!(find_section(&[]).is_none());
}
#[cfg(target_os = "macos")]
#[test]
fn read_self_section_bytes_reads_a_real_file() {
let body = build_section_payload(0x0102_0304_0506_0708, b"stub-payload");
let obj = synthetic_object_with_section(&body);
let dir = std::env::temp_dir().join(format!("decmpfs-section-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("create temp dir");
let path = dir.join("stub.bin");
std::fs::write(&path, &obj).expect("write synthetic object");
let got = read_self_section_bytes(&path).expect("section found");
assert_eq!(got.content_hash, 0x0102_0304_0506_0708);
assert_eq!(got.payload, b"stub-payload");
std::fs::remove_dir_all(&dir).ok();
}
#[cfg(not(target_os = "macos"))]
#[test]
fn read_self_section_bytes_reads_a_footer_from_a_real_file() {
let mut bytes = b"leading executable bytes".to_vec();
bytes.extend_from_slice(&build_footer(0x0102_0304_0506_0708, b"stub-payload"));
let dir = std::env::temp_dir().join(format!("decmpfs-footer-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("create temp dir");
let path = dir.join("stub.bin");
std::fs::write(&path, &bytes).expect("write synthetic footer file");
let got = read_self_section_bytes(&path).expect("footer found");
assert_eq!(got.content_hash, 0x0102_0304_0506_0708);
assert_eq!(got.payload, b"stub-payload");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn read_self_section_bytes_rejects_missing_file() {
let missing = std::env::temp_dir().join("decmpfs-section-does-not-exist-at-all.bin");
assert!(read_self_section_bytes(&missing).is_none());
}
}
+605
src/remove.rs
//! `rm` — a fast recursive remove mirroring Node's `fs.rm` / `fs.rmSync`,
//! tuned for APFS/decmpfs.
//!
//! API parity (no extra knobs): options are exactly Node's `recursive`, `force`,
//! `maxRetries`, `retryDelay`. Semantics match `fs.rm`:
//! - a missing path throws unless `force`;
//! - a directory with `recursive: false` throws `EISDIR`;
//! - `recursive: true` does the `rm -rf`; `maxRetries`/`retryDelay` (linear
//! backoff on EBUSY/EMFILE/ENFILE/ENOTEMPTY/EPERM) apply ONLY when recursive,
//! as in Node.
//!
//! Safety: adapted from socket-lib `safeDelete`, MINUS its socket-owned
//! allowlist (temp / cacache / ~/.socket) — just the universal guard: removing
//! the current directory, one of its ancestors, or the filesystem root is
//! refused unless `force` is set (Node's own option, doubling as the override —
//! no extra knob). Descendants and unrelated siblings are unaffected.
//!
//! Speed: a decmpfs file unlinks like any other (its resource-fork xattr drops
//! with the inode), so DELETE has no compression angle. MEASURED on APFS (this
//! machine, ~12k files), `rm` is filesystem-metadata-bound — directory-entry
//! mutations serialize on the container lock, so BOTH a single `removefile(3)`
//! (~5% slower) and a parallel top-level fan-out (~15-20% slower) LOSE to
//! `std::fs::remove_dir_all`. So this is a correct Node-parity wrapper over
//! `remove_dir_all`, already at that floor — the DELETE win is parity, not a
//! codec trick (contrast WRITE: parallel LZVN, 6.5x).
use std::path::Path;
use crate::Error;
/// Node `fs.rm` options — same four fields, same defaults, nothing extra.
#[derive(Clone, Copy)]
pub struct RmOptions {
pub recursive: bool,
pub force: bool,
pub max_retries: u32,
pub retry_delay_ms: u64,
}
impl Default for RmOptions {
fn default() -> Self {
// Node defaults: recursive false, force false, maxRetries 0, retryDelay 100.
Self {
recursive: false,
force: false,
max_retries: 0,
retry_delay_ms: 100,
}
}
}
fn is_not_found(e: &std::io::Error) -> bool {
e.kind() == std::io::ErrorKind::NotFound
}
// The errno set Node retries in recursive mode.
#[cfg(unix)]
fn retryable(e: &std::io::Error) -> bool {
matches!(
e.raw_os_error(),
Some(c)
if c == libc::EBUSY
|| c == libc::EMFILE
|| c == libc::ENFILE
|| c == libc::ENOTEMPTY
|| c == libc::EPERM
)
}
#[cfg(windows)]
fn retryable(e: &std::io::Error) -> bool {
// ACCESS_DENIED, SHARING_VIOLATION, LOCK_VIOLATION, DIR_NOT_EMPTY.
matches!(e.raw_os_error(), Some(5) | Some(32) | Some(33) | Some(145))
}
/// Run one removal op, applying Node's force (swallow ENOENT) and — only when
/// recursive — the retry/backoff loop.
fn with_policy<F: FnMut() -> std::io::Result<()>>(
mut op: F,
opts: &RmOptions,
) -> std::io::Result<()> {
let mut attempt: u32 = 0;
loop {
match op() {
Ok(()) => return Ok(()),
Err(e) if is_not_found(&e) && opts.force => return Ok(()),
Err(e) if opts.recursive && attempt < opts.max_retries && retryable(&e) => {
attempt += 1;
// Linear backoff: retryDelay ms longer each try (Node's wording).
std::thread::sleep(std::time::Duration::from_millis(
opts.retry_delay_ms.saturating_mul(u64::from(attempt)),
));
}
Err(e) => return Err(e),
}
}
}
fn io(context: &'static str, source: std::io::Error) -> Error {
Error::Io { context, source }
}
/// What `path` is, from a no-follow `symlink_metadata`, collapsed to the three
/// cases `rm` dispatches on. Keeping the fs probe down to this small enum is what
/// lets the seam below be a trivial mock.
enum EntryKind {
Missing,
Dir,
Other,
}
/// The filesystem operations `rm` performs, behind a trait so every error arm is
/// unit-testable with a mock instead of a hard-to-provoke real fault. Prod uses
/// `OsRmFs` — thin `std::fs` / `std::env` delegations with no added logic, so the
/// logic that needs testing all lives in `rm_with` / `guard_cwd_and_root`.
#[cfg_attr(test, mockall::automock)]
trait RmFs {
fn kind(&self, path: &Path) -> std::io::Result<EntryKind>;
fn remove_file(&self, path: &Path) -> std::io::Result<()>;
fn remove_tree(&self, path: &Path) -> std::io::Result<()>;
fn cwd(&self) -> std::io::Result<std::path::PathBuf>;
fn canonicalize(&self, path: &Path) -> std::io::Result<std::path::PathBuf>;
}
struct OsRmFs;
impl RmFs for OsRmFs {
fn kind(&self, path: &Path) -> std::io::Result<EntryKind> {
match std::fs::symlink_metadata(path) {
Ok(md) if md.is_dir() => Ok(EntryKind::Dir),
Ok(_) => Ok(EntryKind::Other),
Err(e) if is_not_found(&e) => Ok(EntryKind::Missing),
Err(e) => Err(e),
}
}
fn remove_file(&self, path: &Path) -> std::io::Result<()> {
std::fs::remove_file(path)
}
/// One recursive delete of a subtree. MEASURED on APFS (this machine, 14 cores,
/// ~12k files): neither a single `removefile(3)` (~4-5% slower) nor a parallel
/// top-level fan-out (~15-20% slower) beats `std::fs::remove_dir_all` —
/// directory-entry mutations serialize on the container's metadata lock, so
/// `rm` is filesystem-bound and `remove_dir_all` (openat + unlinkat, no path
/// re-resolution) is already at that floor. Unlike WRITE (parallel LZVN = 6.5x),
/// DELETE has no userspace codec win, so the simplest correct call is the fast
/// one, on every platform.
fn remove_tree(&self, path: &Path) -> std::io::Result<()> {
std::fs::remove_dir_all(path)
}
// Real Io seam: the safe-delete guard needs the process cwd to refuse deleting
// it or its ancestors (socket-lib safeDelete model). The seam is mocked in
// tests, so this is the one production read of the cwd. A fleet cascade added
// std::env::current_dir to clippy's disallowed_methods.
#[allow(clippy::disallowed_methods)]
fn cwd(&self) -> std::io::Result<std::path::PathBuf> {
std::env::current_dir()
}
fn canonicalize(&self, path: &Path) -> std::io::Result<std::path::PathBuf> {
std::fs::canonicalize(path)
}
}
/// PURE safe-delete guard (socket-lib `safeDelete` model, minus the socket-owned
/// allowlist): is `target` the current directory, an ANCESTOR of it, or the
/// filesystem root? Deleting any of those is almost always a mistake. `cwd` is
/// injected so the policy is unit-testable without touching the process cwd. A
/// sibling or a descendant of cwd is allowed.
fn is_cwd_ancestor_or_root(target: &Path, cwd: &Path) -> bool {
// A path with no parent is a filesystem root ("/", "C:\").
if target.parent().is_none() {
return true;
}
// `target` is cwd or an ancestor of cwd iff cwd is prefixed by target.
cwd == target || cwd.starts_with(target)
}
/// Refuse to remove the cwd, one of its ancestors, or the root — unless `force`.
/// This is the safe-delete guard adapted from socket-lib: NO socket-specific
/// allowlist (temp / cacache / ~/.socket), just the universal ancestor + root
/// protection, with Node's own `force` as the override (no extra option).
fn guard_cwd_and_root<F: RmFs>(fs: &F, path: &Path, opts: &RmOptions) -> Result<(), Error> {
if opts.force {
return Ok(());
}
// Resolve real paths for the comparison; a missing target (canonicalize fails)
// falls back to its given path — it can't be an ancestor of cwd anyway.
let target = fs.canonicalize(path).unwrap_or_else(|_| path.to_path_buf());
let cwd = fs
.cwd()
.and_then(|c| fs.canonicalize(&c))
.unwrap_or_default();
if is_cwd_ancestor_or_root(&target, &cwd) {
return Err(io(
"refusing to remove the current directory, an ancestor, or the root — pass force to override",
std::io::Error::from(std::io::ErrorKind::PermissionDenied),
));
}
Ok(())
}
/// Node `fs.rm(path, options)`. A file/symlink is a single unlink. A directory
/// needs `recursive` (else `EISDIR`, as in Node); recursive delete is
/// `std::fs::remove_dir_all` — MEASURED as the floor on APFS.
pub fn rm(path: &Path, opts: &RmOptions) -> Result<(), Error> {
rm_with(&OsRmFs, path, opts)
}
/// The generic core: `rm` over an injectable filesystem so tests exercise every
/// arm with a mock. Prod calls it with `OsRmFs`.
fn rm_with<F: RmFs>(fs: &F, path: &Path, opts: &RmOptions) -> Result<(), Error> {
guard_cwd_and_root(fs, path, opts)?;
match fs.kind(path) {
Ok(EntryKind::Missing) if opts.force => Ok(()),
Ok(EntryKind::Missing) => Err(Error::NotFound(path.to_path_buf())),
Ok(EntryKind::Other) => with_policy(|| fs.remove_file(path), opts).map_err(|e| io("unlink", e)),
// Node throws EISDIR for a directory without recursive.
Ok(EntryKind::Dir) if !opts.recursive => Err(io(
"path is a directory (pass recursive)",
std::io::Error::from_raw_os_error(eisdir()),
)),
Ok(EntryKind::Dir) => {
with_policy(|| fs.remove_tree(path), opts).map_err(|e| io("remove tree", e))
}
Err(e) => Err(io("lstat", e)),
}
}
#[cfg(unix)]
fn eisdir() -> i32 {
libc::EISDIR
}
#[cfg(windows)]
fn eisdir() -> i32 {
// ERROR_DIRECTORY — "The directory name is invalid" (closest Win32 analog).
267
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
fn seed_tree(root: &Path, dirs: usize, per: usize) {
std::fs::create_dir_all(root).unwrap();
for d in 0..dirs {
let sub = root.join(format!("pkg-{d}"));
std::fs::create_dir_all(sub.join("nested")).unwrap();
for f in 0..per {
std::fs::write(sub.join(format!("f{f}.js")), b"module.exports=1\n").unwrap();
std::fs::write(sub.join("nested").join(format!("g{f}.js")), b"x\n").unwrap();
}
}
}
#[test]
fn matches_node_rm_semantics() {
let root = std::env::temp_dir().join(format!("decmpfs-rm-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&root);
seed_tree(&root, 4, 3);
// recursive:false on a directory throws (EISDIR parity).
assert!(rm(&root, &RmOptions::default()).is_err());
// a symlink is unlinked, not followed.
let keep = std::env::temp_dir().join(format!("decmpfs-rm-keep-{}", std::process::id()));
std::fs::write(&keep, b"keep").unwrap();
#[cfg(unix)]
std::os::unix::fs::symlink(&keep, root.join("link")).unwrap();
let rf = RmOptions {
recursive: true,
force: true,
..RmOptions::default()
};
rm(&root, &rf).unwrap();
assert!(!root.exists(), "tree cleared");
assert!(keep.exists(), "symlink target must survive");
// force: a missing path is Ok; without force it's NotFound.
rm(&root, &rf).unwrap();
assert!(matches!(
rm(&root, &RmOptions::default()),
Err(Error::NotFound(_))
));
// a single file.
let f = std::env::temp_dir().join(format!("decmpfs-rm-one-{}", std::process::id()));
std::fs::write(&f, b"x").unwrap();
rm(&f, &RmOptions::default()).unwrap();
assert!(!f.exists());
let _ = std::fs::remove_file(&keep);
}
#[test]
fn safe_guard_blocks_cwd_ancestors_and_root() {
use std::path::Path;
let cwd = Path::new("/a/b/c");
// cwd itself, an ancestor, and the root are refused.
assert!(is_cwd_ancestor_or_root(Path::new("/a/b/c"), cwd), "cwd");
assert!(is_cwd_ancestor_or_root(Path::new("/a/b"), cwd), "ancestor");
assert!(is_cwd_ancestor_or_root(Path::new("/a"), cwd), "ancestor");
assert!(is_cwd_ancestor_or_root(Path::new("/"), cwd), "root");
// a descendant of cwd and an unrelated sibling are allowed.
assert!(
!is_cwd_ancestor_or_root(Path::new("/a/b/c/build"), cwd),
"descendant allowed"
);
assert!(
!is_cwd_ancestor_or_root(Path::new("/a/b/other"), cwd),
"sibling allowed"
);
}
// Retry/backoff + force policy — the fs op is injected as a closure, so these
// branches are covered WITHOUT touching disk (DI is the mock).
#[cfg(unix)]
#[test]
fn with_policy_covers_retry_force_and_non_retryable() {
use std::io::Error as IoErr;
// retryDelay 0 keeps the test instant.
let recursive = RmOptions {
recursive: true,
force: false,
max_retries: 3,
retry_delay_ms: 0,
};
// A retryable errno (EBUSY) under recursive: retried until it succeeds.
let mut tries = 0;
let ok = with_policy(
|| {
tries += 1;
if tries < 3 {
Err(IoErr::from_raw_os_error(libc::EBUSY))
} else {
Ok(())
}
},
&recursive,
);
assert!(ok.is_ok());
assert_eq!(tries, 3, "retried twice then succeeded");
// Same error, NOT recursive: Node ignores retries → fails on the first try.
let mut n = 0;
let non_recursive = RmOptions {
recursive: false,
..recursive
};
let err = with_policy(
|| {
n += 1;
Err::<(), _>(IoErr::from_raw_os_error(libc::EBUSY))
},
&non_recursive,
);
assert!(err.is_err());
assert_eq!(n, 1, "no retry when not recursive");
// force swallows a missing path.
let forced = with_policy(
|| Err(IoErr::from(std::io::ErrorKind::NotFound)),
&RmOptions {
force: true,
..RmOptions::default()
},
);
assert!(forced.is_ok());
// A non-retryable errno surfaces immediately even under recursive.
let mut k = 0;
let hard = with_policy(
|| {
k += 1;
Err::<(), _>(IoErr::from_raw_os_error(libc::EACCES))
},
&recursive,
);
assert!(hard.is_err());
assert_eq!(k, 1, "non-retryable is not retried");
// retryable() classifies the errno set.
assert!(retryable(&IoErr::from_raw_os_error(libc::ENOTEMPTY)));
assert!(!retryable(&IoErr::from_raw_os_error(libc::EACCES)));
}
#[test]
// The test must read the real cwd to prove the guard refuses deleting it.
#[allow(clippy::disallowed_methods)]
fn rm_refuses_cwd_without_force_but_force_overrides_the_guard() {
// Removing the real cwd is blocked by the guard (this does NOT delete it).
let cwd = std::env::current_dir().unwrap();
assert!(
rm(&cwd, &RmOptions::default()).is_err(),
"guard must refuse removing the cwd"
);
// force bypasses the guard — proven WITHOUT touching cwd: a fresh temp file
// (not an ancestor) removes fine, and force is the documented override.
let f = std::env::temp_dir().join(format!("decmpfs-guard-{}", std::process::id()));
std::fs::write(&f, b"x").unwrap();
let forced = RmOptions {
force: true,
..RmOptions::default()
};
rm(&f, &forced).unwrap();
assert!(!f.exists());
}
// A path far from cwd, so the guard lets rm_with reach its dispatch.
fn mock_target() -> std::path::PathBuf {
std::path::PathBuf::from("/mock-target/x")
}
// Wire a MockRmFs so guard_cwd_and_root passes: canonicalize is identity and
// cwd is an unrelated dir, so `mock_target` is neither cwd nor an ancestor.
fn mock_guard_passes(m: &mut MockRmFs) {
m.expect_canonicalize().returning(|p| Ok(p.to_path_buf()));
m.expect_cwd()
.returning(|| Ok(std::path::PathBuf::from("/mock-cwd")));
}
fn eacces() -> std::io::Error {
std::io::Error::from(std::io::ErrorKind::PermissionDenied)
}
// rm_with over a mocked filesystem — each dispatch + error arm without disk.
#[test]
fn rm_with_surfaces_a_non_not_found_kind_error() {
let mut m = MockRmFs::new();
mock_guard_passes(&mut m);
m.expect_kind().returning(|_| Err(eacces()));
assert!(matches!(
rm_with(&m, &mock_target(), &RmOptions::default()),
Err(Error::Io {
context: "lstat",
..
})
));
}
#[test]
fn rm_with_missing_is_force_ok_else_not_found() {
let mut ok = MockRmFs::new();
ok.expect_kind().returning(|_| Ok(EntryKind::Missing));
let forced = RmOptions {
force: true,
..RmOptions::default()
};
// force short-circuits the guard AND swallows the missing path.
assert!(rm_with(&ok, &mock_target(), &forced).is_ok());
let mut miss = MockRmFs::new();
mock_guard_passes(&mut miss);
miss.expect_kind().returning(|_| Ok(EntryKind::Missing));
assert!(matches!(
rm_with(&miss, &mock_target(), &RmOptions::default()),
Err(Error::NotFound(_))
));
}
#[test]
fn rm_with_dir_needs_recursive_then_removes_tree() {
let mut eisdir = MockRmFs::new();
mock_guard_passes(&mut eisdir);
eisdir.expect_kind().returning(|_| Ok(EntryKind::Dir));
assert!(matches!(
rm_with(&eisdir, &mock_target(), &RmOptions::default()),
Err(Error::Io { context, .. }) if context.contains("directory")
));
let mut tree_err = MockRmFs::new();
mock_guard_passes(&mut tree_err);
tree_err.expect_kind().returning(|_| Ok(EntryKind::Dir));
tree_err.expect_remove_tree().returning(|_| Err(eacces()));
let rf = RmOptions {
recursive: true,
..RmOptions::default()
};
assert!(matches!(
rm_with(&tree_err, &mock_target(), &rf),
Err(Error::Io {
context: "remove tree",
..
})
));
let mut tree_ok = MockRmFs::new();
mock_guard_passes(&mut tree_ok);
tree_ok.expect_kind().returning(|_| Ok(EntryKind::Dir));
tree_ok.expect_remove_tree().returning(|_| Ok(()));
assert!(rm_with(&tree_ok, &mock_target(), &rf).is_ok());
}
#[test]
fn rm_with_file_unlink_ok_and_error() {
let mut ok = MockRmFs::new();
mock_guard_passes(&mut ok);
ok.expect_kind().returning(|_| Ok(EntryKind::Other));
ok.expect_remove_file().returning(|_| Ok(()));
assert!(rm_with(&ok, &mock_target(), &RmOptions::default()).is_ok());
let mut err = MockRmFs::new();
mock_guard_passes(&mut err);
err.expect_kind().returning(|_| Ok(EntryKind::Other));
err.expect_remove_file().returning(|_| Err(eacces()));
assert!(matches!(
rm_with(&err, &mock_target(), &RmOptions::default()),
Err(Error::Io {
context: "unlink",
..
})
));
}
#[test]
fn guard_falls_back_when_canonicalize_and_cwd_fail() {
let mut m = MockRmFs::new();
// Both the target canonicalize and cwd() fail → the guard uses the raw path
// and an empty cwd (the unwrap_or_else / unwrap_or_default fallbacks). A
// non-ancestor path still proceeds to the dispatch.
m.expect_canonicalize().returning(|_| Err(eacces()));
m.expect_cwd().returning(|| Err(eacces()));
m.expect_kind().returning(|_| Ok(EntryKind::Missing));
assert!(matches!(
rm_with(&m, &mock_target(), &RmOptions::default()),
Err(Error::NotFound(_))
));
}
#[test]
fn guard_blocks_when_target_is_an_ancestor_of_cwd() {
let mut m = MockRmFs::new();
// target "/" is the root → an ancestor of any cwd → refused (no force).
m.expect_canonicalize().returning(|p| Ok(p.to_path_buf()));
m.expect_cwd()
.returning(|| Ok(std::path::PathBuf::from("/mock-cwd")));
assert!(matches!(
rm_with(&m, std::path::Path::new("/"), &RmOptions::default()),
Err(Error::Io { context, .. }) if context.contains("refusing")
));
}
#[test]
fn os_rmfs_kind_surfaces_a_non_not_found_error() {
// A path whose parent is a FILE makes symlink_metadata fail non-NotFound
// (ENOTDIR), exercising OsRmFs::kind's real error passthrough.
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("f");
std::fs::write(&file, b"x").unwrap();
let bogus = file.join("child");
assert!(rm(&bogus, &RmOptions::default()).is_err());
}
#[test]
fn os_rmfs_removes_a_real_tree_and_file() {
// Covers OsRmFs's happy delegations (kind Dir/Other, remove_tree/remove_file)
// against a real temp fs with auto-cleanup.
let dir = tempfile::tempdir().unwrap();
let sub = dir.path().join("sub");
std::fs::create_dir_all(sub.join("deep")).unwrap();
std::fs::write(sub.join("a.txt"), b"a").unwrap();
let rf = RmOptions {
recursive: true,
..RmOptions::default()
};
rm(&sub, &rf).unwrap();
assert!(!sub.exists());
let f = dir.path().join("solo");
std::fs::write(&f, b"x").unwrap();
rm(&f, &RmOptions::default()).unwrap();
assert!(!f.exists());
}
// Opt-in perf probe: parallel rm vs std::fs::remove_dir_all on a big tree.
// cargo test -p decmpfs rmrf_probe -- --ignored --nocapture
#[test]
#[ignore]
fn rmrf_probe() {
let base = std::env::temp_dir().join(format!("decmpfs-rmrf-{}", std::process::id()));
let a = base.join("parallel");
let b = base.join("std");
for d in [&a, &b] {
seed_tree(d, 300, 20);
}
let cores = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1);
let rf = RmOptions {
recursive: true,
force: true,
..RmOptions::default()
};
let t0 = std::time::Instant::now();
rm(&a, &rf).unwrap();
let par = t0.elapsed().as_secs_f64() * 1e3;
let t1 = std::time::Instant::now();
std::fs::remove_dir_all(&b).unwrap();
let base_ms = t1.elapsed().as_secs_f64() * 1e3;
eprintln!(
"rmrf ~12k files — decmpfs::rm ({cores} cores avail): {par:.1} ms | std::fs::remove_dir_all: {base_ms:.1} ms"
);
let _ = std::fs::remove_dir_all(&base);
}
}
+317
src/stream.rs
//! Incremental, atomic writes into the platform's transparent compression.
use std::io::Write;
use std::path::{Path, PathBuf};
use crate::{backend, verify, Error, Gate, Outcome, SkipReason, Support, UnsupportedReason};
enum PlainOutcome {
Skipped(SkipReason),
Unsupported(UnsupportedReason),
}
enum StreamInner {
#[cfg(target_os = "macos")]
Macos(backend::StreamingWriter),
#[cfg(target_os = "macos")]
MacosBuffered(Vec<u8>),
File(std::fs::File),
Closed,
}
/// An atomic incremental writer for OS-transparent filesystem compression.
///
/// The expected logical length is required before the first byte. macOS uses it
/// to reserve the decmpfs block-offset table; btrfs and NTFS use it to enforce
/// complete publication. Dropping or aborting the writer removes its sibling
/// temp, leaving any existing destination untouched.
pub struct DecmpfsWriter {
target: PathBuf,
temp: PathBuf,
expected_len: u64,
written: u64,
inner: StreamInner,
plain_outcome: Option<PlainOutcome>,
finished: bool,
}
fn stream_error(context: &'static str, kind: std::io::ErrorKind) -> Error {
Error::Io {
context,
source: std::io::Error::from(kind),
}
}
fn unique_temp(path: &Path) -> Result<PathBuf, Error> {
static STREAM_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let dir = path.parent().ok_or_else(|| {
stream_error(
"stream target has no parent",
std::io::ErrorKind::InvalidInput,
)
})?;
let name = path.file_name().map_or_else(
|| std::borrow::Cow::Borrowed("stream"),
|n| n.to_string_lossy(),
);
let seq = STREAM_SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|duration| duration.as_nanos())
.unwrap_or(0);
Ok(dir.join(format!(
".{name}.decmpfs-stream-{}-{nanos}-{seq}.tmp",
std::process::id()
)))
}
fn create_plain(temp: &Path) -> Result<std::fs::File, Error> {
std::fs::OpenOptions::new()
.read(true)
.write(true)
.create_new(true)
.open(temp)
.map_err(|source| Error::Io {
context: "create streaming temp",
source,
})
}
fn publish(temp: &Path, target: &Path) -> Result<(), Error> {
#[cfg(target_os = "windows")]
{
backend::publish_stream(temp, target)
}
#[cfg(not(target_os = "windows"))]
{
std::fs::rename(temp, target).map_err(|source| Error::Io {
context: "publish streaming temp",
source,
})
}
}
impl DecmpfsWriter {
/// Open an incremental writer. The destination is published only by
/// [`finish`](Self::finish) after exactly `expected_len` bytes arrive.
pub fn create(path: &Path, expected_len: u64, gate: &Gate) -> Result<Self, Error> {
let normalized = path.to_string_lossy().replace('\\', "/");
let temp = unique_temp(path)?;
if !gate.matches(&normalized, expected_len) {
return Ok(Self {
target: path.to_path_buf(),
temp: temp.clone(),
expected_len,
written: 0,
inner: StreamInner::File(create_plain(&temp)?),
plain_outcome: Some(PlainOutcome::Skipped(SkipReason::GateExcluded)),
finished: false,
});
}
let probe = path.parent().unwrap_or(path);
let support =
backend::detect(probe).unwrap_or(Support::Unsupported(UnsupportedReason::Filesystem));
match support {
Support::Supported => {
#[cfg(target_os = "macos")]
let inner = match usize::try_from(expected_len) {
Ok(len) if len <= backend::STREAMING_THRESHOLD => {
StreamInner::MacosBuffered(Vec::with_capacity(len))
}
Ok(len) => StreamInner::Macos(backend::StreamingWriter::new(&temp, len)?),
Err(_) => StreamInner::File(create_plain(&temp)?),
};
#[cfg(any(target_os = "linux", target_os = "windows"))]
let (inner, plain_outcome) = {
let file = create_plain(&temp)?;
match backend::prepare_stream(&file) {
Ok(()) => (StreamInner::File(file), None),
Err(_) => {
drop(file);
let _ = std::fs::remove_file(&temp);
(
StreamInner::File(create_plain(&temp)?),
Some(PlainOutcome::Skipped(SkipReason::IntegrityRevert)),
)
}
}
};
#[cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows")))]
let (inner, plain_outcome) = (
StreamInner::File(create_plain(&temp)?),
Some(PlainOutcome::Unsupported(UnsupportedReason::PlatformBuild)),
);
#[cfg(target_os = "macos")]
let plain_outcome = if usize::try_from(expected_len).is_ok() {
None
} else {
Some(PlainOutcome::Skipped(SkipReason::TooLarge))
};
Ok(Self {
target: path.to_path_buf(),
temp,
expected_len,
written: 0,
inner,
plain_outcome,
finished: false,
})
}
Support::AlreadyCompressed => Ok(Self {
target: path.to_path_buf(),
temp: temp.clone(),
expected_len,
written: 0,
inner: StreamInner::File(create_plain(&temp)?),
plain_outcome: Some(PlainOutcome::Unsupported(UnsupportedReason::Filesystem)),
finished: false,
}),
Support::Unsupported(reason) => Ok(Self {
target: path.to_path_buf(),
temp: temp.clone(),
expected_len,
written: 0,
inner: StreamInner::File(create_plain(&temp)?),
plain_outcome: Some(PlainOutcome::Unsupported(reason)),
finished: false,
}),
}
}
/// Consume one raw chunk without retaining it after the current filesystem
/// compression block has been completed.
pub fn write_chunk(&mut self, buf: &[u8]) -> Result<(), Error> {
let next = self
.written
.checked_add(buf.len() as u64)
.filter(|&len| len <= self.expected_len)
.ok_or_else(|| {
stream_error(
"stream exceeds expected length",
std::io::ErrorKind::InvalidData,
)
})?;
match &mut self.inner {
#[cfg(target_os = "macos")]
StreamInner::Macos(writer) => writer.write_all(buf)?,
#[cfg(target_os = "macos")]
StreamInner::MacosBuffered(buffer) => buffer.extend_from_slice(buf),
StreamInner::File(file) => file.write_all(buf).map_err(|source| Error::Io {
context: "write streaming temp",
source,
})?,
StreamInner::Closed => {
return Err(stream_error(
"write closed stream",
std::io::ErrorKind::BrokenPipe,
));
}
}
self.written = next;
Ok(())
}
/// Remove the incomplete sibling temp and leave the destination untouched.
pub fn abort(mut self) -> Result<(), Error> {
self.inner = StreamInner::Closed;
self.finished = true;
match std::fs::remove_file(&self.temp) {
Ok(()) => Ok(()),
Err(source) if source.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(source) => Err(Error::Io {
context: "abort streaming temp",
source,
}),
}
}
/// Flush, verify the received length, and atomically publish the destination.
pub fn finish(mut self) -> Result<Outcome, Error> {
if self.written != self.expected_len {
return Err(stream_error(
"finish incomplete stream",
std::io::ErrorKind::UnexpectedEof,
));
}
#[cfg(target_os = "macos")]
let macos_compressed = match &mut self.inner {
StreamInner::Macos(writer) => Some(writer.finish()?),
StreamInner::MacosBuffered(buffer) => {
backend::apply_bytes(&self.temp, buffer, None)?;
Some(backend::is_already_compressed(&self.temp).unwrap_or(false))
}
_ => None,
};
#[cfg(not(target_os = "macos"))]
let macos_compressed: Option<bool> = None;
if let StreamInner::File(file) = &mut self.inner {
file
.flush()
.and_then(|()| file.sync_all())
.map_err(|source| Error::Io {
context: "finish streaming temp",
source,
})?;
}
self.inner = StreamInner::Closed;
publish(&self.temp, &self.target)?;
self.finished = true;
if let Some(plain) = self.plain_outcome.take() {
return Ok(match plain {
PlainOutcome::Skipped(reason) => Outcome::Skipped { reason },
PlainOutcome::Unsupported(reason) => Outcome::Unsupported { reason },
});
}
if macos_compressed == Some(false) {
return Ok(Outcome::NoGain {
before: self.expected_len,
after: verify::on_disk_bytes(&self.target)?,
});
}
let after = verify::on_disk_bytes(&self.target)?;
let signal = backend::compressed_on_disk(&self.target).ok().flatten();
if signal.unwrap_or(after < self.expected_len) {
Ok(Outcome::Compressed {
before: self.expected_len,
after,
})
} else {
Ok(Outcome::NoGain {
before: self.expected_len,
after,
})
}
}
}
impl Write for DecmpfsWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.write_chunk(buf).map_err(std::io::Error::other)?;
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
match &mut self.inner {
#[cfg(target_os = "macos")]
StreamInner::Macos(_) => Ok(()),
#[cfg(target_os = "macos")]
StreamInner::MacosBuffered(_) => Ok(()),
StreamInner::File(file) => file.flush(),
StreamInner::Closed => Err(std::io::Error::from(std::io::ErrorKind::BrokenPipe)),
}
}
}
impl Drop for DecmpfsWriter {
fn drop(&mut self) {
if !self.finished {
self.inner = StreamInner::Closed;
let _ = std::fs::remove_file(&self.temp);
}
}
}
+107
tests/exe_selfreplace.rs
//! End-to-end proof of the self-replacing runtime (feature `exe`, unix):
//! pack a real payload into the `decmpfs-stub` binary, RUN the packed stub, and
//! assert it (a) produced the payload's own output, (b) left the on-disk file
//! FS-compressed when the filesystem supports it, and (c) on a second run takes
//! the plain-exec path (the stub is gone — the file IS the payload now).
//!
//! Cargo sets `CARGO_BIN_EXE_decmpfs-stub` for integration tests to the built
//! stub path; the `exe` feature is what builds that bin, so this file is empty
//! without it.
#![cfg(all(unix, feature = "exe"))]
use std::os::unix::fs::PermissionsExt;
use std::path::Path;
use std::process::Command;
fn on_disk_bytes(path: &Path) -> u64 {
use std::os::unix::fs::MetadataExt;
std::fs::metadata(path)
.expect("stat")
.blocks()
.saturating_mul(512)
}
#[test]
fn packed_stub_materializes_then_execs_payload_and_compresses_when_supported() {
let stub = env!("CARGO_BIN_EXE_decmpfs-stub");
let dir = std::env::temp_dir().join(format!("decmpfs-e2e-{}", std::process::id()));
std::fs::create_dir_all(&dir).expect("scratch dir");
// The payload is a shell script that echoes a marker + its args, padded with
// a big, highly-compressible comment — large enough (~1.6 MB) that the
// on-disk win is unambiguous against APFS's block rounding.
let payload = dir.join("payload.sh");
let filler = "# ".to_string() + &"decmpfs ".repeat(200_000);
std::fs::write(
&payload,
format!("#!/bin/sh\n{filler}\necho \"MATERIALIZED $*\"\n"),
)
.expect("write payload");
std::fs::set_permissions(&payload, std::fs::Permissions::from_mode(0o755))
.expect("chmod payload");
// Pack it into a copy of the stub.
let packed = dir.join("packed");
let outcome = decmpfs::exe::pack_executable_with_stub(
Path::new(stub),
&payload,
&packed,
&decmpfs::Gate::any(),
)
.expect("pack succeeds");
assert!(
matches!(outcome, decmpfs::exe::PackOutcome::Packed { .. }),
"Gate::any() must pack, got {outcome:?}",
);
let compression_supported = matches!(
decmpfs::probe(&packed),
Ok(decmpfs::Support::Supported | decmpfs::Support::AlreadyCompressed)
);
// First run: the stub materializes the payload over itself and execs it.
let first = Command::new(&packed)
.arg("hello")
.output()
.expect("run packed stub");
assert!(
first.status.success(),
"first run failed: {}",
String::from_utf8_lossy(&first.stderr),
);
assert_eq!(
String::from_utf8_lossy(&first.stdout).trim(),
"MATERIALIZED hello",
"first run must exec the materialized payload",
);
// The file on disk is now the payload. APFS and btrfs should compress it;
// unsupported filesystems such as a stock Linux CI runner's ext4 still
// exercise the materialize-and-exec fallback and intentionally land plain.
let logical = std::fs::metadata(&packed).expect("stat").len();
let physical = on_disk_bytes(&packed);
if compression_supported {
assert!(
physical < logical,
"materialized file should be FS-compressed: {physical} on disk vs {logical} logical",
);
}
// Second run: the file IS the payload now (no packed section), so the stub
// runtime is never re-entered — it just runs as the plain executable.
let second = Command::new(&packed)
.arg("again")
.output()
.expect("run materialized file");
assert!(second.status.success());
assert_eq!(
String::from_utf8_lossy(&second.stdout).trim(),
"MATERIALIZED again",
);
// The now-plain file carries no packed payload.
assert!(
!decmpfs::exe::contains_payload(&packed),
"after materialize the file carries no packed payload",
);
std::fs::remove_dir_all(&dir).ok();
}
+2
-2
.cargo_vcs_info.json
{
"git": {
"sha1": "55fb470a720c03e8cf4feb76c600b7cc1a277838"
"sha1": "9e4134a23d4805e2e2c4b739f33847728f600dd4"
},
"path_in_vcs": ""
"path_in_vcs": "crates/decmpfs"
}
+345
-1
Cargo.lock

@@ -6,2 +6,35 @@ # This file is automatically @generated by Cargo.

[[package]]
name = "anstyle"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
[[package]]
name = "autocfg"
version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "bit-set"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"
dependencies = [
"bit-vec",
]
[[package]]
name = "bit-vec"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"
[[package]]
name = "bitflags"
version = "2.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8"
[[package]]
name = "block-buffer"

@@ -54,6 +87,9 @@ version = "0.10.4"

name = "decmpfs"
version = "0.1.0"
version = "0.1.2"
dependencies = [
"libc",
"mockall",
"proptest",
"sha2",
"tempfile",
"windows-sys",

@@ -74,2 +110,24 @@ "zstd",

[[package]]
name = "downcast"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1435fa1053d8b2fbbe9be7e97eca7f33d37b28409959813daefc1446a14247f1"
[[package]]
name = "errno"
version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys",
]
[[package]]
name = "fastrand"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6"
[[package]]
name = "find-msvc-tools"

@@ -81,2 +139,23 @@ version = "0.1.9"

[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "fragile"
version = "2.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8878864ba14bb86e818a412bfd6f18f9eabd4ec0f008a28e8f7eb61db532fcf9"
dependencies = [
"futures-core",
]
[[package]]
name = "futures-core"
version = "0.3.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d"
[[package]]
name = "generic-array"

@@ -120,2 +199,49 @@ version = "0.14.7"

[[package]]
name = "linux-raw-sys"
version = "0.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "mockall"
version = "0.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a6ceddfe3ce334925e96bf420fdb2dcee5bed6c632a168ece622676dadeaf8a"
dependencies = [
"cfg-if",
"downcast",
"fragile",
"mockall_derive",
"predicates",
"predicates-tree",
]
[[package]]
name = "mockall_derive"
version = "0.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9cfe16fbe8a314aeec0b861ac24e60b1e123e97634bab045475b9d6a18416fd8"
dependencies = [
"cfg-if",
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
]
[[package]]
name = "once_cell"
version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "pkg-config"

@@ -127,2 +253,80 @@ version = "0.3.33"

[[package]]
name = "ppv-lite86"
version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
dependencies = [
"zerocopy",
]
[[package]]
name = "predicates"
version = "3.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ada8f2932f28a27ee7b70dd6c1c39ea0675c55a36879ab92f3a715eaa1e63cfe"
dependencies = [
"anstyle",
"predicates-core",
]
[[package]]
name = "predicates-core"
version = "1.0.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cad38746f3166b4031b1a0d39ad9f954dd291e7854fcc0eed52ee41a0b50d144"
[[package]]
name = "predicates-tree"
version = "1.0.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0de1b847b39c8131db0467e9df1ff60e6d0562ab8e9a16e568ad0fdb372e2f2"
dependencies = [
"predicates-core",
"termtree",
]
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "proptest"
version = "1.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744"
dependencies = [
"bit-set",
"bit-vec",
"bitflags",
"num-traits",
"rand",
"rand_chacha",
"rand_xorshift",
"regex-syntax",
"rusty-fork",
"tempfile",
"unarray",
]
[[package]]
name = "quick-error"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quote"
version = "1.0.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368"
dependencies = [
"proc-macro2",
]
[[package]]
name = "r-efi"

@@ -134,2 +338,71 @@ version = "5.3.0"

[[package]]
name = "rand"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41"
dependencies = [
"rand_chacha",
"rand_core",
]
[[package]]
name = "rand_chacha"
version = "0.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
dependencies = [
"ppv-lite86",
"rand_core",
]
[[package]]
name = "rand_core"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
dependencies = [
"getrandom",
]
[[package]]
name = "rand_xorshift"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a"
dependencies = [
"rand_core",
]
[[package]]
name = "regex-syntax"
version = "0.8.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
[[package]]
name = "rustix"
version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
dependencies = [
"bitflags",
"errno",
"libc",
"linux-raw-sys",
"windows-sys",
]
[[package]]
name = "rusty-fork"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2"
dependencies = [
"fnv",
"quick-error",
"tempfile",
"wait-timeout",
]
[[package]]
name = "sha2"

@@ -152,2 +425,32 @@ version = "0.10.9"

[[package]]
name = "syn"
version = "2.0.118"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "tempfile"
version = "3.27.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom",
"once_cell",
"rustix",
"windows-sys",
]
[[package]]
name = "termtree"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f50febec83f5ee1df3015341d8bd429f2d1cc62bcba7ea2076759d315084683"
[[package]]
name = "typenum"

@@ -159,2 +462,14 @@ version = "1.20.1"

[[package]]
name = "unarray"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94"
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "version_check"

@@ -166,2 +481,11 @@ version = "0.9.5"

[[package]]
name = "wait-timeout"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11"
dependencies = [
"libc",
]
[[package]]
name = "wasip2"

@@ -255,2 +579,22 @@ version = "1.0.4+wasi-0.2.12"

[[package]]
name = "zerocopy"
version = "0.8.55"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5a105cd7b140f6eeec8acff2ea38135d3cab283ada58540f629fe51e46696eb"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.8.55"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fe976fb70c78cd64cccfe3a6fc142244e8a77b70959b30faf9d0ac37ee228eb"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "zstd"

@@ -257,0 +601,0 @@ version = "0.13.3"

+33
-3
Cargo.toml

@@ -15,4 +15,4 @@ # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO

name = "decmpfs"
version = "0.1.0"
build = false
version = "0.1.2"
build = "build.rs"
autolib = false

@@ -24,3 +24,3 @@ autobins = false

description = "Apply OS-level transparent filesystem compression (APFS decmpfs / btrfs / NTFS) to a file in place."
readme = "README.md"
readme = false
keywords = [

@@ -43,2 +43,6 @@ "apfs",

default = []
exe = [
"dep:zstd",
"dep:sha2",
]

@@ -49,2 +53,7 @@ [lib]

[[bin]]
name = "decmpfs-stub"
path = "src/bin/decmpfs-stub.rs"
required-features = ["exe"]
[[test]]

@@ -54,2 +63,6 @@ name = "btrfs"

[[test]]
name = "exe_selfreplace"
path = "tests/exe_selfreplace.rs"
[dependencies.sha2]

@@ -63,2 +76,11 @@ version = "=0.10.9"

[dev-dependencies.mockall]
version = "=0.15.0"
[dev-dependencies.proptest]
version = "=1.11.0"
[dev-dependencies.tempfile]
version = "=3.27.0"
[target.'cfg(any(target_os = "linux", target_os = "macos"))'.dependencies.libc]

@@ -75,1 +97,9 @@ version = "=0.2.186"

]
[lints.rust.unexpected_cfgs]
level = "warn"
priority = 0
check-cfg = [
"cfg(coverage)",
"cfg(coverage_nightly)",
]
+59
-4
src/addon.rs

@@ -247,5 +247,54 @@ //! `addon` feature — unwrap a napi `--compress` hybrid back to the raw `.node`.

#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use proptest::prelude::*;
use super::*;
proptest! {
// Tier 1 round-trip: any raw addon, wrapped into a well-formed pressed-data
// section, decodes back to the exact original bytes (header parse + zstd inflate
// + SHA-512 gate is the identity over a validly-framed blob).
#[test]
fn decode_round_trips_arbitrary_payload(
raw in prop::collection::vec(any::<u8>(), 1..8192),
has_config in any::<bool>(),
) {
let section = synth_section(&raw, has_config);
let decoded = decode_pressed_data(&section);
prop_assert_eq!(decoded.as_deref(), Some(raw.as_slice()));
}
// The decoder never panics on arbitrary bytes — the offset arithmetic, size
// guards, and zstd frame decode must fail closed to a graceful `None`.
#[test]
fn decode_never_panics(data in prop::collection::vec(any::<u8>(), 0..4096)) {
let _ = decode_pressed_data(&data);
}
// The container walk + decode never panics on arbitrary bytes.
#[test]
fn unwrap_never_panics(data in prop::collection::vec(any::<u8>(), 0..4096)) {
let _ = unwrap_if_hybrid(&data);
}
// Tamper-evidence: a single flipped byte anywhere in a valid section either still
// decodes to the EXACT original (the flip landed in an ignored field — cache key
// / platform metadata) or is rejected with `None`. It can NEVER yield different
// bytes — the SHA-512 gate makes a silently-wrong decode impossible.
#[test]
fn tampering_never_yields_wrong_bytes(
raw in prop::collection::vec(any::<u8>(), 1..2048),
idx in any::<prop::sample::Index>(),
xor in 1u8..=255,
) {
let mut section = synth_section(&raw, false);
let i = idx.index(section.len());
section[i] ^= xor;
if let Some(out) = decode_pressed_data(&section) {
prop_assert_eq!(out, raw);
}
}
}
/// Build a valid pressed-data section blob from `raw` (the addon) so the header

@@ -278,3 +327,6 @@ /// parse + zstd decode + SHA-512 check round-trip without a real binary.

let section = synth_section(&raw, false);
assert_eq!(decode_pressed_data(&section).as_deref(), Some(raw.as_slice()));
assert_eq!(
decode_pressed_data(&section).as_deref(),
Some(raw.as_slice())
);
}

@@ -286,3 +338,6 @@

let section = synth_section(&raw, true);
assert_eq!(decode_pressed_data(&section).as_deref(), Some(raw.as_slice()));
assert_eq!(
decode_pressed_data(&section).as_deref(),
Some(raw.as_slice())
);
}

@@ -334,3 +389,3 @@

bin[16..20].copy_from_slice(&1u32.to_le_bytes()); // ncmds = 1
// segment_command_64 at offset 32
// segment_command_64 at offset 32
let seg = 32;

@@ -341,3 +396,3 @@ bin[seg..seg + 4].copy_from_slice(&LC_SEGMENT_64.to_le_bytes());

bin[seg + 64..seg + 68].copy_from_slice(&1u32.to_le_bytes()); // nsects = 1
// section_64 at offset seg + 72
// section_64 at offset seg + 72
let sect = seg + 72;

@@ -344,0 +399,0 @@ bin[sect..sect + 14].copy_from_slice(b"__PRESSED_DATA");

+81
-2
src/gate.rs

@@ -261,5 +261,78 @@ //! The install-time gate: decide whether a given file should be OS-compressed.

#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use proptest::prelude::*;
use super::*;
proptest! {
// Tier 1 — the size-predicate parser never panics on ANY string. `spec` is a
// manifest-supplied value (`compress = ">= 1MB"`), so arbitrary text reaches it;
// a graceful `Err(GateParseError)` is the only non-`Ok` outcome.
#[test]
fn size_predicate_parse_never_panics(s in ".*") {
let _ = SizePredicate::parse(&s);
}
// Round-trip: a bare-number spec parses back to the exact threshold under both
// operators (formatting the struct's value then reparsing is the identity).
#[test]
fn bare_number_spec_round_trips(n in any::<u64>()) {
prop_assert_eq!(
SizePredicate::parse(&format!(">{n}")),
Ok(SizePredicate::GreaterThan(n))
);
prop_assert_eq!(
SizePredicate::parse(&format!(">= {n}")),
Ok(SizePredicate::AtLeast(n))
);
}
// Oracle: a decimal-unit literal equals `number * 1000^k` whenever it can't
// overflow (bounded so the product stays inside u64).
#[test]
fn decimal_unit_scales_like_the_oracle(n in 0u64..=1_000_000) {
prop_assert_eq!(parse_size(&format!("{n}KB")), Ok(n * 1_000));
prop_assert_eq!(parse_size(&format!("{n}MB")), Ok(n * 1_000_000));
prop_assert_eq!(parse_size(&format!("{n}kib")), Ok(n * 1024));
}
// `Gate::matches` never panics on an untrusted path text of any bytes/length —
// the path segment can carry a downloaded package's name.
#[test]
fn gate_matches_never_panics(text in ".*", len in any::<u64>()) {
let _ = Gate::default().matches(&text, len);
let _ = Gate::any().matches(&text, len);
}
// Oracle: a metachar-free pattern is a plain literal — it matches iff the text
// is byte-identical.
#[test]
fn literal_glob_matches_iff_equal(
lit in "[a-zA-Z0-9._-]{0,32}",
text in "[a-zA-Z0-9._-]{0,32}",
) {
prop_assert_eq!(glob_match(&lit, &text), lit == text);
}
// `**` is the match-everything pattern: it accepts any path text, including one
// that spans separators.
#[test]
fn double_star_matches_any_path(text in ".*") {
prop_assert!(glob_match("**", &text));
}
// The matcher terminates and never panics on arbitrary pattern AND text. The
// pattern alphabet is bounded (few metachars, short) so this Tier-1 property
// can't wander into the algorithmic-complexity search that Tier 2 (the
// `gate_glob` fuzz target) owns.
#[test]
fn glob_match_never_panics(
pattern in "[a-z/.?*]{0,12}",
text in "[a-z/.]{0,24}",
) {
let _ = glob_match(&pattern, &text);
}
}
#[test]

@@ -286,4 +359,7 @@ fn parses_units_case_insensitively() {

assert_eq!(parse_size("10PB"), Err(GateParseError::Unit));
assert_eq!(parse_size("99999999999999999999GB"), Err(GateParseError::Number));
assert_eq!(
parse_size("99999999999999999999GB"),
Err(GateParseError::Number)
);
assert_eq!(
parse_size("18446744073709551615KB"),

@@ -324,3 +400,6 @@ Err(GateParseError::Overflow)

fn glob_matches_node_addons_anywhere() {
assert!(glob_match("**/*.node", "node_modules/foo/build/Release/addon.node"));
assert!(glob_match(
"**/*.node",
"node_modules/foo/build/Release/addon.node"
));
assert!(glob_match("**/*.node", "addon.node"));

@@ -327,0 +406,0 @@ assert!(glob_match("*.node", "addon.node"));

+615
-19
src/lib.rs

@@ -22,2 +22,7 @@ //! `decmpfs` — apply the operating system's transparent per-file compression to a file

#![cfg_attr(not(test), deny(clippy::unwrap_used, clippy::expect_used))]
// On a nightly `cargo llvm-cov` run, cargo-llvm-cov sets `coverage_nightly`,
// enabling `#[coverage(off)]` so test-only code is dropped from the report and it
// reflects PRODUCTION coverage. A no-op on stable (the cfg is unset), so ordinary
// builds and `cargo test` are unaffected.
#![cfg_attr(coverage_nightly, feature(coverage_attribute))]

@@ -226,2 +231,170 @@ use std::path::Path;

/// What a [`copy_file`] did — a SUCCESS shape, same contract as [`Outcome`]:
/// `Err` is reserved for genuine I/O failures; the copy itself always lands.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum CopyOutcome {
/// Copy-on-write clone (`clonefile` / `FICLONE`) — the extents are shared,
/// so the source's compression state carries over at zero cost.
Cloned { compressed: bool },
/// Byte copy plus one-pass recompression at the destination (a compressed
/// source that could not be cloned — cross-volume, non-reflink FS).
CopiedCompressed { before: u64, after: u64 },
/// Byte copy landed plain: the source was plain, the destination FS has no
/// transparent compression, or a fail-soft skip (`skipped` carries the
/// reason when the source WAS compressed but the state could not follow).
CopiedPlain { skipped: Option<SkipReason> },
}
/// Attempt a copy-on-write clone of `src` at `dest` (`clonefile(2)` on macOS,
/// the `FICLONE` ioctl on Linux) — the zero-cost way to copy a compressed file
/// WITH its compression. `Ok(true)` = cloned; `Ok(false)` = this pairing can't
/// clone (cross-volume, non-reflink FS, an existing destination on macOS,
/// Windows) and the caller decides the fallback — [`copy_file`] is the
/// clone-then-fallback composition, and a Node-`COPYFILE_FICLONE_FORCE`-shaped
/// caller turns `false` into its error.
pub fn try_clone_file(src: &Path, dest: &Path) -> Result<bool, Error> {
if !src.exists() {
return Err(Error::NotFound(src.to_path_buf()));
}
Os.clone_file(src, dest)
}
/// Copy `src` to `dest` preserving transparent filesystem compression — the
/// `fs.copyFile` the OS never shipped. A plain byte copy silently re-inflates
/// a compressed file (the kernel hands every reader the full logical bytes,
/// and that is what gets written back out); this copy keeps the on-disk
/// savings.
///
/// Strategy, in order:
/// 1. A copy-on-write clone (`clonefile(2)` on macOS, the `FICLONE` ioctl on
/// Linux) shares the extents, so a compressed source stays compressed at
/// zero cost — and a plain source clones plain.
/// 2. When cloning is impossible (cross-volume, non-reflink FS, Windows), the
/// logical bytes are copied and, if the source was compressed, the
/// destination is written back compressed via the same guarded one-pass
/// path as [`compress_bytes`].
/// 3. A plain source is copied plain — a copy never changes compression state.
///
/// `fs.copyFile` parity: an existing `dest` is replaced, and the source's
/// permission bits carry over. Fail-soft like the rest of the crate: on any
/// backend skip the plain copy still stands, reported in the outcome.
pub fn copy_file(src: &Path, dest: &Path) -> Result<CopyOutcome, Error> {
copy_file_with(&Os, src, dest)
}
/// `copy_file` over an injectable [`Backend`] — production always threads
/// [`Os`]; tests drive the clone and fallback arms with fakes.
fn copy_file_with<B: Backend>(backend: &B, src: &Path, dest: &Path) -> Result<CopyOutcome, Error> {
if !src.exists() {
return Err(Error::NotFound(src.to_path_buf()));
}
if dest.exists() {
// A copy onto itself (same path, a hardlink, a symlink alias) is a no-op
// success — otherwise the replace step below would clobber the copy's own
// source. The extents are 100% shared, which is what `Cloned` states.
if is_same_file(src, dest) {
return Ok(CopyOutcome::Cloned {
compressed: backend.is_already_compressed(src).unwrap_or(false),
});
}
// clonefile refuses an existing destination; replace-by-default is the
// `fs.copyFile` contract this mirrors.
std::fs::remove_file(dest).map_err(|source| Error::Io {
context: "replace existing destination",
source,
})?;
}
// On a filesystem with no compression signal this is a capability gap, not a
// failure — treat it as "plain source" and keep copying.
let compressed_src = backend.is_already_compressed(src).unwrap_or(false);
if backend.clone_file(src, dest)? {
return Ok(CopyOutcome::Cloned {
compressed: compressed_src,
});
}
// A normal read hands back the full logical bytes regardless of the
// source's on-disk representation.
let content = std::fs::read(src).map_err(|source| Error::Io {
context: "read copy source",
source,
})?;
let mode = std::fs::metadata(src).ok().map(|meta| meta.permissions());
if !compressed_src {
plain_write(dest, &content)?;
if let Some(mode) = mode {
let _ = std::fs::set_permissions(dest, mode);
}
return Ok(CopyOutcome::CopiedPlain { skipped: None });
}
let outcome = compress_bytes_with(backend, dest, &content, &Gate::any())?;
if let Some(mode) = mode {
let _ = std::fs::set_permissions(dest, mode);
}
Ok(match outcome {
Outcome::Compressed { before, after } | Outcome::NoGain { before, after } => {
CopyOutcome::CopiedCompressed { before, after }
}
// Unreachable from a fresh destination (compress_bytes maps an
// already-compressed detect to a plain write), kept total + truthful.
Outcome::AlreadyCompressed { before } => CopyOutcome::CopiedCompressed {
before,
after: before,
},
Outcome::Unsupported { .. } => CopyOutcome::CopiedPlain { skipped: None },
Outcome::Skipped { reason } => CopyOutcome::CopiedPlain {
skipped: Some(reason),
},
})
}
/// True when both paths name the same underlying file — same dev+inode on
/// unix, or same volume serial + file index on Windows. Guards [`copy_file`]'s
/// replace-by-default from removing its own source.
#[cfg(unix)]
fn is_same_file(a: &Path, b: &Path) -> bool {
use std::os::unix::fs::MetadataExt;
match (std::fs::metadata(a), std::fs::metadata(b)) {
(Ok(meta_a), Ok(meta_b)) => meta_a.dev() == meta_b.dev() && meta_a.ino() == meta_b.ino(),
_ => false,
}
}
#[cfg(windows)]
fn is_same_file(a: &Path, b: &Path) -> bool {
use std::os::windows::io::AsRawHandle;
use windows_sys::Win32::Storage::FileSystem::{
GetFileInformationByHandle, BY_HANDLE_FILE_INFORMATION,
};
fn identity(path: &Path) -> Option<(u32, u64)> {
let file = std::fs::File::open(path).ok()?;
let mut info = unsafe { std::mem::zeroed::<BY_HANDLE_FILE_INFORMATION>() };
if unsafe { GetFileInformationByHandle(file.as_raw_handle() as _, &mut info) } == 0 {
return None;
}
Some((
info.dwVolumeSerialNumber,
(u64::from(info.nFileIndexHigh) << 32) | u64::from(info.nFileIndexLow),
))
}
match (identity(a), identity(b)) {
(Some(id_a), Some(id_b)) => id_a == id_b,
// Preserve the exact-path fast path if a file-information query is denied.
_ => match (std::fs::canonicalize(a), std::fs::canonicalize(b)) {
(Ok(canon_a), Ok(canon_b)) => canon_a == canon_b,
_ => false,
},
}
}
#[cfg(not(any(unix, windows)))]
fn is_same_file(a: &Path, b: &Path) -> bool {
match (std::fs::canonicalize(a), std::fs::canonicalize(b)) {
(Ok(canon_a), Ok(canon_b)) => canon_a == canon_b,
_ => false,
}
}
/// Fail-soft plain atomic write: sibling temp + fsync + rename. The never-break-the

@@ -261,9 +434,67 @@ /// -install floor under every `compress_bytes` fallback.

/// Filesystem-compression state of a path — one call that coalesces the
/// otherwise-separate size + backend-signal reads (the compress/copy paths
/// previously did a `stat` AND an `lstat`/attr read per file). Follows symlinks:
/// compression is a property of the target file, never a symlink.
pub struct Stat {
/// FS-compressed on disk. Uses the backend's authoritative signal where it has
/// one (`UF_COMPRESSED` on APFS, FIEMAP-encoded extents on btrfs, the
/// compressed attribute on NTFS); elsewhere inferred from allocated < logical.
pub compressed: bool,
/// Apparent (logical) size — constant whether or not the file is compressed.
pub logical: u64,
/// Allocated (physical) bytes on disk — where the compression win shows.
pub physical: u64,
}
/// Inspect the FS-compression state of `path` (see [`Stat`]).
pub fn stat(path: &Path) -> Result<Stat, Error> {
stat_with(&Os, path)
}
/// [`stat`] over an injectable [`Backend`] so the no-signal (allocated-bytes)
/// inference arm is testable without a real filesystem.
fn stat_with<B: Backend>(backend: &B, path: &Path) -> Result<Stat, Error> {
let meta = std::fs::metadata(path).map_err(|source| Error::Io {
context: "stat",
source,
})?;
let logical = meta.len();
// One metadata read yields both size + allocation on unix (the coalesce);
// Windows needs GetCompressedFileSizeW for the post-compression allocation.
#[cfg(unix)]
let physical = {
use std::os::unix::fs::MetadataExt;
meta.blocks().saturating_mul(512)
};
#[cfg(not(unix))]
let physical = verify::on_disk_bytes(path)?;
// Prefer the backend's authoritative signal; fall back to the
// allocated-vs-logical inference when there is no signal (e.g. NTFS) OR the
// probe isn't supported on this filesystem (e.g. FIEMAP on tmpfs) — a stat is
// an inspection and must never fail over a best-effort compression check.
let compressed = match backend.compressed_on_disk(path) {
Ok(Some(signal)) => signal,
Ok(None) | Err(_) => logical > 0 && physical < logical,
};
Ok(Stat {
compressed,
logical,
physical,
})
}
#[cfg(feature = "addon")]
pub mod addon;
#[cfg(feature = "exe")]
pub mod exe;
mod gate;
mod remove;
mod safety;
mod stream;
mod verify;
pub use gate::{Gate, GateParseError, SizePredicate, DEFAULT_GLOB};
pub use remove::{rm, RmOptions};
pub use stream::DecmpfsWriter;

@@ -289,6 +520,11 @@ #[cfg(target_os = "linux")]

/// cost in a release build).
#[cfg_attr(test, mockall::automock)]
pub(crate) trait Backend {
fn detect(&self, path: &Path) -> Result<Support, Error>;
fn is_already_compressed(&self, path: &Path) -> Result<bool, Error>;
fn apply_inplace(&self, path: &Path) -> Result<(), Error>;
/// Compress `path` in place. `snapshot` is the already-read file contents (the
/// caller holds them for rollback); backends that rewrite via temp+rename reuse
/// it instead of reading the file a second time, and backends that flag the
/// existing file in place (Windows) ignore it.
fn apply_inplace(&self, path: &Path, snapshot: &[u8]) -> Result<(), Error>;
fn apply_bytes(

@@ -301,2 +537,8 @@ &self,

fn compressed_on_disk(&self, path: &Path) -> Result<Option<bool>, Error>;
/// Copy-on-write clone. `Ok(false)` = "cannot clone here" → the caller falls
/// back to a byte copy. Defaulted so fakes without a clone path exercise the
/// fallback arms.
fn clone_file(&self, _src: &Path, _dest: &Path) -> Result<bool, Error> {
Ok(false)
}
}

@@ -314,4 +556,4 @@

}
fn apply_inplace(&self, path: &Path) -> Result<(), Error> {
backend::apply_inplace(path)
fn apply_inplace(&self, path: &Path, snapshot: &[u8]) -> Result<(), Error> {
backend::apply_inplace(path, snapshot)
}

@@ -329,2 +571,5 @@ fn apply_bytes(

}
fn clone_file(&self, src: &Path, dest: &Path) -> Result<bool, Error> {
backend::clone_file(src, dest)
}
}

@@ -337,14 +582,15 @@

pub(crate) detect: Support,
/// `None` → apply succeeds; `Some(errno)` → apply fails with that OS error.
pub(crate) apply_errno: Option<i32>,
/// `None` → apply succeeds; `Some(kind)` → apply fails portably with that kind.
pub(crate) apply_error: Option<std::io::ErrorKind>,
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
impl FakeBackend {
fn apply_result(&self) -> Result<(), Error> {
match self.apply_errno {
match self.apply_error {
None => Ok(()),
Some(errno) => Err(Error::Io {
Some(kind) => Err(Error::Io {
context: "fake apply",
source: std::io::Error::from_raw_os_error(errno),
source: std::io::Error::from(kind),
}),

@@ -356,2 +602,3 @@ }

#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
impl Backend for FakeBackend {

@@ -364,3 +611,3 @@ fn detect(&self, _path: &Path) -> Result<Support, Error> {

}
fn apply_inplace(&self, _path: &Path) -> Result<(), Error> {
fn apply_inplace(&self, _path: &Path, _snapshot: &[u8]) -> Result<(), Error> {
self.apply_result()

@@ -381,4 +628,4 @@ }

#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {

@@ -389,2 +636,5 @@ use super::*;

let dir = std::env::temp_dir().join(format!("decmpfs-{tag}-{}", std::process::id()));
// A pid-recycled leftover FILE at this path makes create_dir_all fail
// with AlreadyExists; clear it so the scratch dir always materializes.
let _ = std::fs::remove_file(&dir);
std::fs::create_dir_all(&dir).unwrap();

@@ -493,6 +743,3 @@ dir

assert!(
matches!(
out,
Ok(Outcome::Compressed { .. } | Outcome::NoGain { .. })
),
matches!(out, Ok(Outcome::Compressed { .. } | Outcome::NoGain { .. })),
"one-pass APFS write → applied, got {out:?}"

@@ -614,3 +861,6 @@ );

let out = compress_bytes(&target, &fake_addon(), &Gate::any());
assert!(out.is_err(), "cannot write a file over a directory, got {out:?}");
assert!(
out.is_err(),
"cannot write a file over a directory, got {out:?}"
);
assert!(target.is_dir(), "the directory is left intact");

@@ -620,2 +870,50 @@ std::fs::remove_dir_all(&dir).ok();

#[test]
fn stat_reports_size_and_uncompressed_for_a_plain_file() {
let dir = scratch("stat-plain");
let path = dir.join("f");
std::fs::write(&path, vec![0u8; 4096]).unwrap();
let s = stat(&path).unwrap();
assert_eq!(s.logical, 4096, "logical == the written bytes");
assert!(s.physical > 0, "allocated bytes reported");
assert!(
!s.compressed,
"a freshly-written plain file is not FS-compressed"
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn stat_reflects_a_compressed_file_where_supported() {
let dir = scratch("stat-comp");
let path = dir.join("addon.node");
let content = vec![0xABu8; 128 * 1024];
let outcome = compress_bytes(&path, &content, &Gate::any()).unwrap();
let s = stat(&path).unwrap();
assert_eq!(
s.logical,
content.len() as u64,
"logical == the written bytes"
);
assert_eq!(
std::fs::read(&path).unwrap(),
content,
"content round-trips"
);
// Where the FS actually compressed (APFS / btrfs / NTFS), stat must reflect
// it; on an unsupported FS the outcome isn't Compressed and we only assert
// the size + content invariants above.
if matches!(outcome, Outcome::Compressed { .. }) {
assert!(
s.compressed,
"a Compressed outcome → stat reports compressed"
);
assert!(
s.physical < s.logical,
"allocation shrank below the logical size"
);
}
std::fs::remove_dir_all(&dir).ok();
}
// A read-only parent dir: the guarded backend write hits EACCES (classify_skip →

@@ -652,3 +950,3 @@ // Skipped), then the plain-write fallback also can't write → `Err`. Root bypasses

detect: Support::AlreadyCompressed,
apply_errno: None,
apply_error: None,
};

@@ -670,6 +968,9 @@ assert!(matches!(

detect: Support::Unsupported(UnsupportedReason::Filesystem),
apply_errno: None,
apply_error: None,
};
let out = compress_bytes_with(&backend, &path, &content, &Gate::any());
assert!(matches!(out, Ok(Outcome::Unsupported { .. })), "got {out:?}");
assert!(
matches!(out, Ok(Outcome::Unsupported { .. })),
"got {out:?}"
);
assert_eq!(std::fs::read(&path).unwrap(), content, "bytes landed plain");

@@ -688,3 +989,3 @@ std::fs::remove_dir_all(&dir).ok();

detect: Support::Supported,
apply_errno: Some(13), // EACCES
apply_error: Some(std::io::ErrorKind::PermissionDenied),
};

@@ -704,2 +1005,297 @@ let out = compress_bytes_with(&backend, &path, &content, &Gate::any());

}
#[test]
fn copy_file_errors_when_the_source_is_missing() {
let dir = scratch("copy-missing");
let out = copy_file(&dir.join("absent.node"), &dir.join("dest.node"));
assert!(matches!(out, Err(Error::NotFound(_))));
std::fs::remove_dir_all(&dir).ok();
}
/// A fallback fake: no clone path (trait default), reports the source
/// compressed, and its apply actually writes — so the guarded one-pass copy
/// arm runs end to end and classifies via the backend signal.
struct RecompressingFake;
impl Backend for RecompressingFake {
fn detect(&self, _path: &Path) -> Result<Support, Error> {
Ok(Support::Supported)
}
fn is_already_compressed(&self, _path: &Path) -> Result<bool, Error> {
Ok(true)
}
fn apply_inplace(&self, _path: &Path, _snapshot: &[u8]) -> Result<(), Error> {
Ok(())
}
fn apply_bytes(
&self,
path: &Path,
content: &[u8],
_mode: Option<std::fs::Permissions>,
) -> Result<(), Error> {
std::fs::write(path, content).map_err(|source| Error::Io {
context: "fake write",
source,
})
}
fn compressed_on_disk(&self, _path: &Path) -> Result<Option<bool>, Error> {
Ok(Some(true))
}
}
#[test]
fn copy_file_recompresses_at_the_destination_when_it_cannot_clone() {
let dir = scratch("copy-recompress");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
let content = fake_addon();
std::fs::write(&src, &content).unwrap();
let out = copy_file_with(&RecompressingFake, &src, &dest).unwrap();
assert!(
matches!(out, CopyOutcome::CopiedCompressed { .. }),
"got {out:?}"
);
assert_eq!(
std::fs::read(&dest).unwrap(),
content,
"bytes are identical"
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn copy_file_with_mock_backend_takes_the_clone_fast_path() {
// mockall MockBackend mocks the fs backend seam (no real syscalls); tempfile
// gives a real, isolated, auto-cleaned src fixture. clone_file → true
// short-circuits copy_file_with to the zero-cost Cloned outcome.
let dir = tempfile::tempdir().unwrap();
let src = dir.path().join("a.node");
std::fs::write(&src, b"native").unwrap();
let dest = dir.path().join("b.node");
let mut backend = MockBackend::new();
backend
.expect_is_already_compressed()
.returning(|_| Ok(true));
backend.expect_clone_file().returning(|_, _| Ok(true));
let out = copy_file_with(&backend, &src, &dest).unwrap();
assert!(
matches!(out, CopyOutcome::Cloned { compressed: true }),
"clone fast-path → Cloned; got {out:?}"
);
}
#[test]
fn copy_file_copies_a_plain_source_plain_and_replaces_the_destination() {
let dir = scratch("copy-plain");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
let content = fake_addon();
std::fs::write(&src, &content).unwrap();
std::fs::write(&dest, b"stale destination").unwrap();
let backend = FakeBackend {
detect: Support::Supported,
apply_error: None,
};
let out = copy_file_with(&backend, &src, &dest).unwrap();
assert_eq!(out, CopyOutcome::CopiedPlain { skipped: None });
assert_eq!(
std::fs::read(&dest).unwrap(),
content,
"destination replaced"
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn copy_file_lands_plain_and_reports_the_skip_when_recompression_fails() {
struct SkippingFake;
impl Backend for SkippingFake {
fn detect(&self, _path: &Path) -> Result<Support, Error> {
Ok(Support::Supported)
}
fn is_already_compressed(&self, _path: &Path) -> Result<bool, Error> {
Ok(true)
}
fn apply_inplace(&self, _path: &Path, _snapshot: &[u8]) -> Result<(), Error> {
Ok(())
}
fn apply_bytes(
&self,
_path: &Path,
_content: &[u8],
_mode: Option<std::fs::Permissions>,
) -> Result<(), Error> {
Err(Error::Io {
context: "fake apply",
source: std::io::Error::from(std::io::ErrorKind::PermissionDenied),
})
}
fn compressed_on_disk(&self, _path: &Path) -> Result<Option<bool>, Error> {
Ok(Some(false))
}
}
let dir = scratch("copy-skip");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
let content = fake_addon();
std::fs::write(&src, &content).unwrap();
let out = copy_file_with(&SkippingFake, &src, &dest).unwrap();
assert!(
matches!(out, CopyOutcome::CopiedPlain { skipped: Some(_) }),
"got {out:?}"
);
assert_eq!(std::fs::read(&dest).unwrap(), content, "bytes still landed");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn copy_file_onto_itself_is_a_noop_reported_as_cloned() {
let dir = scratch("copy-self");
let src = dir.join("src.node");
let content = fake_addon();
std::fs::write(&src, &content).unwrap();
let backend = FakeBackend {
detect: Support::Supported,
apply_error: None,
};
let out = copy_file_with(&backend, &src, &src).unwrap();
assert!(matches!(out, CopyOutcome::Cloned { .. }), "got {out:?}");
assert_eq!(std::fs::read(&src).unwrap(), content, "source untouched");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn is_same_file_sees_hardlinks_and_distinct_files() {
let dir = scratch("same-file");
let a = dir.join("a.node");
let b = dir.join("b.node");
std::fs::write(&a, b"bytes").unwrap();
std::fs::write(&b, b"bytes").unwrap();
assert!(is_same_file(&a, &a), "identical path");
assert!(!is_same_file(&a, &b), "distinct files");
let link = dir.join("a-link.node");
std::fs::hard_link(&a, &link).unwrap();
assert!(is_same_file(&a, &link), "hardlink shares the inode");
assert!(
!is_same_file(&a, &dir.join("absent.node")),
"a missing path is never the same file"
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn copy_file_onto_a_hardlink_is_a_noop_reported_as_cloned() {
let dir = scratch("copy-hardlink");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
let content = fake_addon();
std::fs::write(&src, &content).unwrap();
std::fs::hard_link(&src, &dest).unwrap();
let out = copy_file(&src, &dest).unwrap();
assert!(matches!(out, CopyOutcome::Cloned { .. }), "got {out:?}");
assert_eq!(std::fs::read(&src).unwrap(), content, "source untouched");
assert_eq!(std::fs::read(&dest).unwrap(), content, "hardlink untouched");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn copy_file_errors_when_the_destination_cannot_be_replaced() {
let dir = scratch("copy-dest-dir");
let src = dir.join("src.node");
std::fs::write(&src, fake_addon()).unwrap();
// A directory at `dest` makes the replace step's remove_file fail.
let dest = dir.join("dest.node");
std::fs::create_dir(&dest).unwrap();
let backend = FakeBackend {
detect: Support::Supported,
apply_error: None,
};
let out = copy_file_with(&backend, &src, &dest);
assert!(
matches!(
out,
Err(Error::Io {
context: "replace existing destination",
..
})
),
"got {out:?}"
);
std::fs::remove_dir_all(&dir).ok();
}
#[cfg(unix)]
#[test]
fn copy_file_errors_when_the_source_is_unreadable() {
use std::os::unix::fs::PermissionsExt;
let dir = scratch("copy-unreadable");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
std::fs::write(&src, fake_addon()).unwrap();
std::fs::set_permissions(&src, std::fs::Permissions::from_mode(0o000)).unwrap();
let backend = FakeBackend {
detect: Support::Supported,
apply_error: None,
};
let out = copy_file_with(&backend, &src, &dest);
std::fs::set_permissions(&src, std::fs::Permissions::from_mode(0o644)).ok();
assert!(
matches!(
out,
Err(Error::Io {
context: "read copy source",
..
})
),
"got {out:?}"
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn try_clone_file_errors_when_the_source_is_missing() {
let dir = scratch("clone-missing");
let out = try_clone_file(&dir.join("absent.node"), &dir.join("dest.node"));
assert!(matches!(out, Err(Error::NotFound(_))));
std::fs::remove_dir_all(&dir).ok();
}
#[cfg(target_os = "macos")]
#[test]
fn try_clone_file_clones_on_apfs_and_declines_an_existing_destination() {
let dir = scratch("clone-try");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
std::fs::write(&src, fake_addon()).unwrap();
assert!(try_clone_file(&src, &dest).unwrap(), "fresh clone");
// clonefile refuses an existing destination — reported as cannot-clone,
// never an error.
assert!(!try_clone_file(&src, &dest).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[cfg(target_os = "macos")]
#[test]
fn copy_file_clones_a_compressed_source_on_apfs() {
let dir = scratch("copy-clone");
let src = dir.join("src.node");
let dest = dir.join("dest.node");
let content = fake_addon();
let wrote = compress_bytes(&src, &content, &Gate::any()).unwrap();
// Only meaningful when the scratch volume actually compressed the source.
if !matches!(wrote, Outcome::Compressed { .. }) {
std::fs::remove_dir_all(&dir).ok();
return;
}
let out = copy_file(&src, &dest).unwrap();
assert_eq!(out, CopyOutcome::Cloned { compressed: true });
assert!(backend::is_already_compressed(&dest).unwrap());
assert_eq!(
std::fs::read(&dest).unwrap(),
content,
"bytes are identical"
);
std::fs::remove_dir_all(&dir).ok();
}
}
+157
-9
src/linux.rs

@@ -89,2 +89,6 @@ //! Linux backend — btrfs (Stage 1); bcachefs later. ext4/xfs/ZFS cannot be done.

pub(crate) fn prepare_stream(file: &std::fs::File) -> Result<(), Error> {
request_codec(file.as_raw_fd())
}
pub(crate) fn detect(path: &Path) -> Result<Support, Error> {

@@ -174,11 +178,28 @@ if statfs_type(path)? == BTRFS_SUPER_MAGIC {

pub(crate) fn apply_inplace(path: &Path) -> Result<(), Error> {
let data = std::fs::read(path).map_err(|source| Error::Io {
context: "read",
source,
})?;
pub(crate) fn apply_inplace(path: &Path, snapshot: &[u8]) -> Result<(), Error> {
// `snapshot` is the file's bytes the caller already read for rollback — reuse
// it instead of a second full read.
let mode = std::fs::metadata(path).map(|m| m.permissions()).ok();
apply_bytes(path, &data, mode)
apply_bytes(path, snapshot, mode)
}
/// A collision-resistant sibling temp path for the atomic write. PID +
/// wall-clock nanos + a process-local counter means two threads/processes
/// compressing the SAME destination never derive the same temp name (a PID-only
/// name with create+truncate would interleave their writes, or silently adopt a
/// crashed run's stale temp). Paired with `create_new`, a collision is detected
/// rather than truncated through. Mirrors the macOS backend's scheme.
fn unique_tmp(dir: &Path, name: &str) -> std::path::PathBuf {
static TMP_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let seq = TMP_SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
dir.join(format!(
".{name}.decmpfs-{}-{nanos}-{seq}.tmp",
std::process::id()
))
}
/// Write `content` to `path` as a fresh btrfs-compressed file in ONE pass: create

@@ -204,3 +225,3 @@ /// the sibling temp, request the codec on the EMPTY file (so the bytes compress as

);
let tmp = dir.join(format!(".{name}.decmpfs-{}.tmp", std::process::id()));
let tmp = unique_tmp(dir, &name);

@@ -211,4 +232,3 @@ let result = (|| {

.write(true)
.create(true)
.truncate(true)
.create_new(true)
.open(&tmp)

@@ -237,2 +257,11 @@ .map_err(|source| Error::Io {

}
// Preserve ownership across the rewrite. As root (a global npm install, a
// Docker build) the temp is owned by the current euid, so the rename would
// otherwise change the file's owner. Match the original's uid/gid.
// Best-effort: a no-op for a new path and for a non-root process (chown to
// another owner is EPERM — but then the file was already ours).
if let Ok(meta) = std::fs::metadata(path) {
use std::os::unix::fs::MetadataExt;
let _ = std::os::unix::fs::chown(&tmp, Some(meta.uid()), Some(meta.gid()));
}
}

@@ -254,1 +283,120 @@

}
/// Copy-on-write clone via the `FICLONE` ioctl (btrfs / XFS reflink) — shares
/// the extents AND their compression property, so a compressed source stays
/// compressed at zero cost. `Ok(false)` means "cannot clone here"
/// (cross-device, non-reflink FS, …) and the caller falls back to a byte copy;
/// a failed clone removes the empty destination it created.
pub(crate) fn clone_file(src: &Path, dest: &Path) -> Result<bool, Error> {
use std::os::fd::AsRawFd;
let src_file = match std::fs::File::open(src) {
Ok(file) => file,
Err(_) => return Ok(false),
};
let dest_file = match std::fs::File::create(dest) {
Ok(file) => file,
Err(_) => return Ok(false),
};
// FICLONE = _IOW(0x94, 9, int) — stable since Linux 4.5.
const FICLONE: libc::c_ulong = 0x4004_9409;
let cloned = unsafe { libc::ioctl(dest_file.as_raw_fd(), FICLONE, src_file.as_raw_fd()) } == 0;
if !cloned {
drop(dest_file);
let _ = std::fs::remove_file(dest);
}
Ok(cloned)
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
fn scratch(tag: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("decmpfs-linux-{tag}-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
dir
}
// btrfs is required to exercise apply_bytes / is_already_compressed; ext4/tmpfs
// CI runners report Unsupported and the compression path is never reached.
fn on_btrfs(dir: &Path) -> bool {
let probe = dir.join(".btrfs-probe");
std::fs::write(&probe, b"x").unwrap();
let yes = matches!(detect(&probe), Ok(Support::Supported));
std::fs::remove_file(&probe).ok();
yes
}
#[test]
fn unique_tmp_never_collides_and_is_well_formed() {
let dir = Path::new("/tmp");
let a = unique_tmp(dir, "addon.node");
let b = unique_tmp(dir, "addon.node");
assert_ne!(a, b, "successive temps must differ (the seq counter)");
for p in [&a, &b] {
let f = p.file_name().unwrap().to_string_lossy();
assert!(
f.starts_with(".addon.node.decmpfs-"),
"unexpected temp name: {f}"
);
assert!(f.ends_with(".tmp"), "unexpected temp name: {f}");
assert_eq!(p.parent().unwrap(), dir);
}
}
#[test]
fn detect_is_ok_on_a_regular_temp_path() {
// btrfs → Supported, ext4/tmpfs → Unsupported; only a real I/O error Errs.
assert!(detect(&std::env::temp_dir()).is_ok());
}
#[test]
fn a_fresh_plain_file_is_not_already_compressed() {
let dir = scratch("iac");
let path = dir.join("f");
std::fs::write(&path, b"plain").unwrap();
assert!(!is_already_compressed(&path).unwrap_or(false));
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn clone_file_declines_a_missing_source() {
let dir = scratch("clone");
assert!(!clone_file(&dir.join("missing"), &dir.join("dest")).unwrap());
assert!(!dir.join("dest").exists(), "the empty dest is cleaned up");
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn apply_bytes_round_trips_and_concurrent_writers_converge_on_btrfs() {
let dir = scratch("rt");
if !on_btrfs(&dir) {
std::fs::remove_dir_all(&dir).ok();
return;
}
let content = vec![0xEEu8; 128 * 1024];
let dest = dir.join("addon.node");
// Eight writers racing on the SAME destination must all converge to identical
// bytes — the create_new + unique-temp scheme prevents interleaved writes.
std::thread::scope(|s| {
for _ in 0..8 {
let dest = dest.clone();
let content = content.clone();
s.spawn(move || {
let _ = apply_bytes(&dest, &content, None);
});
}
});
assert_eq!(
std::fs::read(&dest).unwrap(),
content,
"bytes survive the race"
);
assert!(
is_already_compressed(&dest).unwrap(),
"landed btrfs-compressed"
);
std::fs::remove_dir_all(&dir).ok();
}
}
+1251
-95
src/macos.rs
//! macOS backend — APFS/HFS+ decmpfs transparent compression.
//!
//! decmpfs is an undocumented kernel ABI; afsctool and `ditto --hfsCompression` are
//! the references. We write the LZVN resource-fork variant (compression type 8): the
//! kernel decompresses on read(), so the file keeps its logical size + stays a
//! loadable native binary. LZVN comes from the system libcompression framework — no
//! Rust codec dep, so the macOS stub stays tiny.
//! the references. We write the LZVN (type 8) and LZFSE (type 12) resource-fork
//! variants: the kernel decompresses on read(), so the file keeps its logical size
//! and stays a loadable native binary. Both codecs come from the system
//! libcompression library, so the macOS stub gains no Rust codec dependency.
//!
//! Codec choice is LZVN, not LZFSE (type 12) or ZLIB (type 4): for a load-once
//! `.node` we favor decode speed over ratio, and LZVN is the fastest of the three
//! to decompress (LZFSE wins ~10-15% on size but decodes slower). LZVN has shipped
//! in libcompression since 10.11, so there's no availability fallback to make — the
//! only fallback path is the ephemeral cache when decmpfs can't be applied at all.
//! Common `.node` files take the speed-first LZVN path, with LZFSE as a no-gain
//! fallback. Large assets stream ratio-first LZFSE blocks directly into the named
//! resource fork, avoiding both a multi-gigabyte output allocation and `setxattr`'s
//! `E2BIG` ceiling. LZVN is the last fallback there for unusual codec-specific data.
//!
//! Layout written (verified by the kernel-roundtrip test):
//! xattr com.apple.decmpfs = [magic u32 LE][type=8 LZVN u32 LE][rawSize u64 LE]
//! xattr com.apple.ResourceFork = [(numBlocks+1) u32 LE offset table][LZVN blocks]
//! Built on a sibling temp (empty data fork + those xattrs + UF_COMPRESSED), then
//! atomically renamed over the original — never an in-place truncate, so a crash
//! can't leave a 0-byte file.
//! xattr com.apple.decmpfs = [magic u32 LE][type=8/12 u32 LE][rawSize u64 LE]
//! xattr com.apple.ResourceFork = [(numBlocks+1) u32 LE offsets][codec blocks]
//! A winning fork is built on an empty sibling temp with those xattrs and
//! UF_COMPRESSED; an expanding fork becomes an ordinary sibling data fork. Either
//! form is atomically renamed over the original — never an in-place truncate, so
//! a crash can't leave a 0-byte file.

@@ -29,15 +29,52 @@ use std::os::fd::AsRawFd;

const DECMPFS_MAGIC: u32 = 0x636d_7066; // 'cmpf' (XNU sys/decmpfs.h); LE on disk = "fpmc"
// Type 8 = LZVN-in-resource-fork — what `ditto --hfsCompression` writes and the
// kernel reliably reads. The resource fork is a flat offset table (NOT the zlib
// type-4 resource-fork-with-map format).
const CMP_LZVN_RESOURCE_FORK: u32 = 8;
const BLOCK: usize = 0x1_0000; // 64 KiB
const XATTR_NOFOLLOW: libc::c_int = 0x0001;
// decmpfs resource-fork offsets cap at u32; stay well under 4 GiB so the offsets
// and the kernel's own checks never overflow.
const MAX_RAW: u64 = 3_900_000_000;
const COMPRESSION_LZVN: i32 = 0x900;
const COMPRESSION_LZFSE: i32 = 0x801;
// 2026-07-16 — The data supports keeping a 64 MiB in-memory fast path for now.
// The sampled Darwin ARM64 Vite ecosystem topped out at SWC 36.563 MiB, followed
// by Rolldown 15.6–17.9 MiB, Oxlint 14.4 MiB, Lightning CSS 8.1 MiB, and Oxc
// bindings at 1.9–5.7 MiB. Keeping <=64 MiB on parallel LZVN + one `setxattr`
// leaves 27 MiB of headroom above that observed upper tail; larger assets stream
// to `..namedfork/rsrc` so peak output memory stays bounded. Re-benchmark this
// boundary as native-addon sizes or the streaming implementation changes.
pub(crate) const STREAMING_THRESHOLD: usize = 64 * 1024 * 1024;
fn should_stream_resource_fork(raw_len: usize, threshold: usize) -> bool {
raw_len > threshold
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Codec {
Lzvn,
Lzfse,
}
impl Codec {
const fn compression_type(self) -> u32 {
match self {
Self::Lzvn => 8,
Self::Lzfse => 12,
}
}
const fn algorithm(self) -> i32 {
match self {
Self::Lzvn => COMPRESSION_LZVN,
Self::Lzfse => COMPRESSION_LZFSE,
}
}
}
#[link(name = "compression")]
extern "C" {
fn compression_decode_buffer(
dst_buffer: *mut u8,
dst_size: usize,
src_buffer: *const u8,
src_size: usize,
scratch_buffer: *mut u8,
algorithm: i32,
) -> usize;
fn compression_encode_buffer(

@@ -54,13 +91,7 @@ dst_buffer: *mut u8,

/// Reject files past the decmpfs u32-offset ceiling (→ Skipped(TooLarge) via
/// safety::classify_skip on EFBIG). Pure, so the limit is testable without a 4 GiB
/// file.
fn within_decmpfs_limit(len: u64) -> Result<(), Error> {
if len > MAX_RAW {
return Err(Error::Io {
context: "file too large for decmpfs",
source: std::io::Error::from_raw_os_error(libc::EFBIG),
});
fn resource_fork_too_large() -> Error {
Error::Io {
context: "decmpfs resource fork exceeds u32 offsets",
source: std::io::Error::from_raw_os_error(libc::EFBIG),
}
Ok(())
}

@@ -126,7 +157,6 @@

/// LZVN-encode `src` into a kernel-decodable block. libcompression emits a valid
/// Encode `src` into one kernel-decodable block. libcompression emits a valid
/// frame even for incompressible input (slightly larger than `src`), so every
/// block decodes the same way — there is no bare "stored" block. Returns `None`
/// only if encoding fails outright (treated as a hard error upstream).
fn compress_block(src: &[u8], scratch: &mut [u8]) -> Option<Vec<u8>> {
/// block decodes the same way. `None` means the codec declined outright.
fn compress_block_with_codec(src: &[u8], scratch: &mut [u8], codec: Codec) -> Option<Vec<u8>> {
// Headroom for the worst case (incompressible data expands a little).

@@ -141,3 +171,3 @@ let mut dst = vec![0u8; src.len() + src.len() / 16 + 1024];

scratch.as_mut_ptr(),
COMPRESSION_LZVN,
codec.algorithm(),
)

@@ -152,22 +182,128 @@ };

// The compatibility wrapper keeps the focused LZVN unit tests terse.
#[cfg(test)]
fn compress_block(src: &[u8], scratch: &mut [u8]) -> Option<Vec<u8>> {
compress_block_with_codec(src, scratch, Codec::Lzvn)
}
#[derive(Debug, PartialEq, Eq)]
enum ResourceForkPlan {
/// The fork would not make the file smaller. Keep the ordinary data fork.
Plain,
/// The encoded fork is smaller and every offset fits the on-disk u32 table.
Compressed { table_len: usize, total_len: usize },
}
fn resource_fork_table_len(num_blocks: usize) -> Result<usize, Error> {
num_blocks
.checked_add(1)
.and_then(|entries| entries.checked_mul(std::mem::size_of::<u32>()))
.ok_or_else(resource_fork_too_large)
}
/// Decide from lengths alone whether the fork is useful and representable. The
/// raw length in `com.apple.decmpfs` is u64; only offsets inside the resource
/// fork are u32. This allows a raw file beyond the old 3.9 GB cutoff whenever its
/// encoded fork is smaller than both the raw file and `u32::MAX`.
fn plan_resource_fork(
raw_len: usize,
num_blocks: usize,
encoded_len: usize,
) -> Result<ResourceForkPlan, Error> {
let table_len = resource_fork_table_len(num_blocks)?;
let total_len = table_len
.checked_add(encoded_len)
.ok_or_else(resource_fork_too_large)?;
if total_len >= raw_len {
return Ok(ResourceForkPlan::Plain);
}
if total_len > u32::MAX as usize {
return Err(resource_fork_too_large());
}
Ok(ResourceForkPlan::Compressed {
table_len,
total_len,
})
}
fn compress_blocks(raw: &[u8], codec: Codec) -> Option<Vec<Vec<u8>>> {
let num_blocks = raw.len().div_ceil(BLOCK).max(1);
let scratch_len = unsafe { compression_encode_scratch_buffer_size(codec.algorithm()) };
// The 64 KiB blocks are independent, so fan them across cores. Each worker
// owns its libcompression scratch buffer. Contiguous regions keep the output
// in block order without a sort.
let workers = if std::env::var_os("DECMPFS_SERIAL").is_some() {
1
} else {
std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1)
.min(num_blocks)
};
if workers <= 1 || num_blocks < 8 {
let mut scratch = vec![0u8; scratch_len];
return raw
.chunks(BLOCK)
.map(|chunk| compress_block_with_codec(chunk, &mut scratch, codec))
.collect();
}
let bytes_per_worker = num_blocks.div_ceil(workers) * BLOCK;
let parts: Vec<Option<Vec<Vec<u8>>>> = std::thread::scope(|scope| {
let handles: Vec<_> = raw
.chunks(bytes_per_worker)
.map(|region| {
scope.spawn(move || {
let mut scratch = vec![0u8; scratch_len];
region
.chunks(BLOCK)
.map(|chunk| compress_block_with_codec(chunk, &mut scratch, codec))
.collect::<Option<Vec<Vec<u8>>>>()
})
})
.collect();
handles
.into_iter()
.map(|handle| handle.join().ok().flatten())
.collect()
});
let mut out = Vec::with_capacity(num_blocks);
for part in parts {
out.extend(part?);
}
Some(out)
}
/// Build the com.apple.ResourceFork blob for `raw` in the LZVN/LZFSE decmpfs
/// layout (what `ditto` writes): `(numBlocks+1)` u32 LE offsets, then the blocks.
/// `offset[0]` = table size; `offset[i+1]` = end of block i; last = total size.
fn build_resource_fork(raw: &[u8]) -> Option<Vec<u8>> {
/// `Ok(None)` means this codec did not shrink the input.
fn build_resource_fork_with_codec(raw: &[u8], codec: Codec) -> Result<Option<Vec<u8>>, Error> {
let num_blocks = raw.len().div_ceil(BLOCK).max(1);
let scratch_len = unsafe { compression_encode_scratch_buffer_size(COMPRESSION_LZVN) };
let mut scratch = vec![0u8; scratch_len];
let Some(blocks) = compress_blocks(raw, codec) else {
return Ok(None);
};
let blocks = raw
.chunks(BLOCK)
.map(|chunk| compress_block(chunk, &mut scratch))
.collect::<Option<Vec<Vec<u8>>>>()?;
let encoded_len = blocks
.iter()
.try_fold(0usize, |sum, block| sum.checked_add(block.len()))
.ok_or_else(resource_fork_too_large)?;
let ResourceForkPlan::Compressed {
table_len,
total_len,
} = plan_resource_fork(raw.len(), num_blocks, encoded_len)?
else {
return Ok(None);
};
let table_len = (num_blocks + 1) * 4;
let mut out = Vec::with_capacity(table_len + blocks.iter().map(Vec::len).sum::<usize>());
let mut out = Vec::with_capacity(total_len);
// Offset table: numBlocks+1 entries. offset[i] is where block i starts.
let mut offset = table_len as u32;
let mut offset = u32::try_from(table_len).map_err(|_| resource_fork_too_large())?;
out.extend_from_slice(&offset.to_le_bytes());
for block in &blocks {
offset += block.len() as u32;
offset = offset
.checked_add(u32::try_from(block.len()).map_err(|_| resource_fork_too_large())?)
.ok_or_else(resource_fork_too_large)?;
out.extend_from_slice(&offset.to_le_bytes());

@@ -178,5 +314,664 @@ }

}
Some(out)
debug_assert_eq!(out.len(), total_len);
Ok(Some(out))
}
/// The existing speed-first LZVN builder, retained as a focused test seam.
#[cfg(test)]
fn build_resource_fork(raw: &[u8]) -> Result<Option<Vec<u8>>, Error> {
build_resource_fork_with_codec(raw, Codec::Lzvn)
}
struct InMemoryResourceFork {
codec: Codec,
bytes: Vec<u8>,
}
/// Common addons prefer LZVN decode speed. Only a no-gain LZVN attempt pays for
/// the stronger LZFSE pass.
fn build_in_memory_resource_fork(raw: &[u8]) -> Result<Option<InMemoryResourceFork>, Error> {
for codec in [Codec::Lzvn, Codec::Lzfse] {
if let Some(bytes) = build_resource_fork_with_codec(raw, codec)? {
return Ok(Some(InMemoryResourceFork { codec, bytes }));
}
}
Ok(None)
}
/// Stream one codec into the temp file's named resource fork. One single-slot
/// channel per worker keeps at most one encoded block per core waiting for the
/// ordered writer, so output memory is bounded while libcompression stays
/// parallel. `Ok(false)` means no gain, an unrepresentable u32 fork, or a codec
/// decline; the caller may retry the original bytes with another codec.
fn write_streaming_resource_fork(path: &Path, raw: &[u8], codec: Codec) -> Result<bool, Error> {
use std::io::{Seek, Write};
use std::sync::atomic::{AtomicBool, Ordering};
let num_blocks = raw.len().div_ceil(BLOCK).max(1);
let table_len = resource_fork_table_len(num_blocks)?;
if table_len >= raw.len() || table_len > u32::MAX as usize {
return Ok(false);
}
let fork_path = path.join("..namedfork").join("rsrc");
let mut file = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(fork_path)
.map_err(|source| Error::Io {
context: "open resource fork",
source,
})?;
file.set_len(table_len as u64).map_err(|source| Error::Io {
context: "reserve resource-fork table",
source,
})?;
file
.seek(std::io::SeekFrom::Start(table_len as u64))
.map_err(|source| Error::Io {
context: "seek resource-fork payload",
source,
})?;
// Coalesce codec blocks into larger writes. Besides syscall overhead, one
// write per 64 KiB block makes APFS allocate several extra MiB of extents on
// a multi-gigabyte fork even when its logical compressed bytes are identical.
let mut writer = std::io::BufWriter::with_capacity(1 << 20, file);
let workers = if std::env::var_os("DECMPFS_SERIAL").is_some() {
1
} else {
std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1)
.min(num_blocks)
};
let scratch_len = unsafe { compression_encode_scratch_buffer_size(codec.algorithm()) };
let cancelled = AtomicBool::new(false);
let mut offsets = Vec::with_capacity(num_blocks + 1);
offsets.push(u32::try_from(table_len).map_err(|_| resource_fork_too_large())?);
let mut offset = table_len;
let won = std::thread::scope(|scope| -> Result<bool, Error> {
let mut receivers = Vec::with_capacity(workers);
for worker in 0..workers {
let (sender, receiver) = std::sync::mpsc::sync_channel(1);
receivers.push(receiver);
let cancelled = &cancelled;
scope.spawn(move || {
let mut scratch = vec![0u8; scratch_len];
let mut block_index = worker;
while block_index < num_blocks && !cancelled.load(Ordering::Relaxed) {
let start = block_index * BLOCK;
let end = start.saturating_add(BLOCK).min(raw.len());
let encoded = compress_block_with_codec(&raw[start..end], &mut scratch, codec);
if sender.send(encoded).is_err() {
break;
}
block_index += workers;
}
});
}
let result = (|| -> Result<bool, Error> {
for block_index in 0..num_blocks {
let Some(block) = receivers[block_index % workers].recv().ok().flatten() else {
return Ok(false);
};
let Some(next_offset) = offset.checked_add(block.len()) else {
return Ok(false);
};
// The fork can only grow from here. Stop early rather than writing a
// multi-gigabyte losing candidate before trying the fallback codec.
if next_offset >= raw.len() || next_offset > u32::MAX as usize {
return Ok(false);
}
writer.write_all(&block).map_err(|source| Error::Io {
context: "write resource-fork block",
source,
})?;
offset = next_offset;
offsets.push(u32::try_from(offset).map_err(|_| resource_fork_too_large())?);
}
Ok(true)
})();
cancelled.store(true, Ordering::Relaxed);
drop(receivers);
result
})?;
if !won {
return Ok(false);
}
debug_assert_eq!(offsets.len(), num_blocks + 1);
let mut table = Vec::with_capacity(table_len);
for offset in offsets {
table.extend_from_slice(&offset.to_le_bytes());
}
debug_assert_eq!(table.len(), table_len);
writer
.seek(std::io::SeekFrom::Start(0))
.and_then(|_| writer.write_all(&table))
.and_then(|_| writer.flush())
.map_err(|source| Error::Io {
context: "finish resource fork",
source,
})?;
writer.get_ref().sync_all().map_err(|source| Error::Io {
context: "sync resource fork",
source,
})?;
Ok(true)
}
/// Large assets favor ratio-first LZFSE so we do not write and discard a huge
/// LZVN candidate like Gemini's. LZVN remains a codec-specific fallback.
fn build_streaming_resource_fork(path: &Path, raw: &[u8]) -> Result<Option<Codec>, Error> {
for codec in [Codec::Lzfse, Codec::Lzvn] {
if write_streaming_resource_fork(path, raw, codec)? {
return Ok(Some(codec));
}
}
Ok(None)
}
enum StreamingState {
Encoding(StreamingEncoding),
Plain(std::fs::File),
Closed,
}
struct StreamingEncoding {
file: std::fs::File,
fork: std::io::BufWriter<std::fs::File>,
scratch: Vec<u8>,
partial: Vec<u8>,
offsets: Vec<u32>,
encoded_offset: usize,
}
/// Incremental macOS writer used by the public streaming API. Raw input is held
/// only until the current 64 KiB block is complete; winning LZFSE blocks land
/// directly in the named resource fork. If the fork stops winning, its completed
/// blocks are decoded into a plain sibling and subsequent input streams there.
pub(crate) struct StreamingWriter {
path: std::path::PathBuf,
expected_len: usize,
written: usize,
state: StreamingState,
complete: bool,
}
impl StreamingEncoding {
fn write_block(&mut self, raw: &[u8], expected_len: usize) -> Result<bool, Error> {
let Some(encoded) = compress_block_with_codec(raw, &mut self.scratch, Codec::Lzfse) else {
return Ok(false);
};
// Verify every encoder result while the matching raw block is still in
// memory. The final kernel oracle then only has to prove the decmpfs layout.
let mut decoded = vec![0u8; raw.len()];
let decoded_len = unsafe {
compression_decode_buffer(
decoded.as_mut_ptr(),
decoded.len(),
encoded.as_ptr(),
encoded.len(),
std::ptr::null_mut(),
Codec::Lzfse.algorithm(),
)
};
if decoded_len != raw.len() || decoded != raw {
return Ok(false);
}
let Some(next_offset) = self.encoded_offset.checked_add(encoded.len()) else {
return Ok(false);
};
if next_offset >= expected_len || next_offset > u32::MAX as usize {
return Ok(false);
}
use std::io::Write;
self.fork.write_all(&encoded).map_err(|source| Error::Io {
context: "write streaming resource-fork block",
source,
})?;
self.encoded_offset = next_offset;
self
.offsets
.push(u32::try_from(next_offset).map_err(|_| resource_fork_too_large())?);
Ok(true)
}
}
fn streaming_fallback_path(path: &Path) -> std::path::PathBuf {
static FALLBACK_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let seq = FALLBACK_SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let name = path.file_name().map_or_else(
|| std::borrow::Cow::Borrowed("stream"),
|n| n.to_string_lossy(),
);
path.with_file_name(format!(".{name}.plain-{}-{seq}.tmp", std::process::id()))
}
fn decode_streaming_prefix(
path: &Path,
encoding: &mut StreamingEncoding,
current: &[u8],
expected_len: usize,
) -> Result<(std::path::PathBuf, std::fs::File), Error> {
use std::io::{Read, Seek, Write};
encoding.fork.flush().map_err(|source| Error::Io {
context: "flush streaming resource fork",
source,
})?;
encoding
.fork
.get_ref()
.sync_all()
.map_err(|source| Error::Io {
context: "sync streaming resource fork",
source,
})?;
let fork_path = path.join("..namedfork").join("rsrc");
let mut fork = std::fs::File::open(fork_path).map_err(|source| Error::Io {
context: "open streaming resource fork for fallback",
source,
})?;
let fallback = streaming_fallback_path(path);
let mut plain = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create_new(true)
.open(&fallback)
.map_err(|source| Error::Io {
context: "create streaming plain fallback",
source,
})?;
let decoded = (|| -> Result<(), Error> {
for (block_index, pair) in encoding.offsets.windows(2).enumerate() {
let start = pair[0] as u64;
let encoded_len = (pair[1] - pair[0]) as usize;
let mut encoded = vec![0u8; encoded_len];
fork
.seek(std::io::SeekFrom::Start(start))
.and_then(|_| fork.read_exact(&mut encoded))
.map_err(|source| Error::Io {
context: "read streaming resource fork for fallback",
source,
})?;
let raw_len = expected_len
.saturating_sub(block_index.saturating_mul(BLOCK))
.min(BLOCK);
let mut raw = vec![0u8; raw_len];
let raw_len = unsafe {
compression_decode_buffer(
raw.as_mut_ptr(),
raw.len(),
encoded.as_ptr(),
encoded.len(),
std::ptr::null_mut(),
Codec::Lzfse.algorithm(),
)
};
if raw_len != raw.len() {
return Err(Error::Io {
context: "decode streaming resource fork for fallback",
source: std::io::Error::from(std::io::ErrorKind::InvalidData),
});
}
plain.write_all(&raw).map_err(|source| Error::Io {
context: "write streaming plain fallback",
source,
})?;
}
plain.write_all(current).map_err(|source| Error::Io {
context: "write current streaming fallback block",
source,
})
})();
if let Err(err) = decoded {
drop(plain);
let _ = std::fs::remove_file(&fallback);
return Err(err);
}
Ok((fallback, plain))
}
fn streaming_kernel_matches(
path: &Path,
encoding: &StreamingEncoding,
expected_len: usize,
) -> Result<bool, Error> {
use std::io::{Read, Seek};
let mut logical = match std::fs::File::open(path) {
Ok(file) => file,
Err(_) => return Ok(false),
};
let fork_path = path.join("..namedfork").join("rsrc");
let mut fork = std::fs::File::open(fork_path).map_err(|source| Error::Io {
context: "open finished streaming resource fork",
source,
})?;
for (block_index, pair) in encoding.offsets.windows(2).enumerate() {
let encoded_len = (pair[1] - pair[0]) as usize;
let mut encoded = vec![0u8; encoded_len];
fork
.seek(std::io::SeekFrom::Start(pair[0] as u64))
.and_then(|_| fork.read_exact(&mut encoded))
.map_err(|source| Error::Io {
context: "read finished streaming resource fork",
source,
})?;
let raw_len = expected_len
.saturating_sub(block_index.saturating_mul(BLOCK))
.min(BLOCK);
let mut decoded = vec![0u8; raw_len];
let decoded_len = unsafe {
compression_decode_buffer(
decoded.as_mut_ptr(),
decoded.len(),
encoded.as_ptr(),
encoded.len(),
std::ptr::null_mut(),
Codec::Lzfse.algorithm(),
)
};
if decoded_len != raw_len {
return Ok(false);
}
let mut kernel = vec![0u8; raw_len];
if logical.read_exact(&mut kernel).is_err() || kernel != decoded {
return Ok(false);
}
}
let mut extra = [0u8; 1];
Ok(logical.read(&mut extra).is_ok_and(|len| len == 0))
}
impl StreamingWriter {
pub(crate) fn new(path: &Path, expected_len: usize) -> Result<Self, Error> {
let num_blocks = expected_len.div_ceil(BLOCK).max(1);
let table_len = resource_fork_table_len(num_blocks)?;
if expected_len == 0 || table_len >= expected_len || table_len > u32::MAX as usize {
let file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create_new(true)
.open(path)
.map_err(|source| Error::Io {
context: "create streaming plain temp",
source,
})?;
return Ok(Self {
path: path.to_path_buf(),
expected_len,
written: 0,
state: StreamingState::Plain(file),
complete: false,
});
}
let file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create_new(true)
.open(path)
.map_err(|source| Error::Io {
context: "create streaming decmpfs temp",
source,
})?;
let fork_file = (|| -> Result<std::fs::File, Error> {
let fork_path = path.join("..namedfork").join("rsrc");
let mut fork_file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create(true)
.truncate(true)
.open(fork_path)
.map_err(|source| Error::Io {
context: "open streaming resource fork",
source,
})?;
use std::io::Seek;
fork_file
.set_len(table_len as u64)
.map_err(|source| Error::Io {
context: "reserve streaming resource-fork table",
source,
})?;
fork_file
.seek(std::io::SeekFrom::Start(table_len as u64))
.map_err(|source| Error::Io {
context: "seek streaming resource-fork payload",
source,
})?;
Ok(fork_file)
})();
let fork_file = match fork_file {
Ok(fork_file) => fork_file,
Err(error) => {
drop(file);
let _ = std::fs::remove_file(path);
return Err(error);
}
};
let scratch_len = unsafe { compression_encode_scratch_buffer_size(Codec::Lzfse.algorithm()) };
Ok(Self {
path: path.to_path_buf(),
expected_len,
written: 0,
state: StreamingState::Encoding(StreamingEncoding {
file,
fork: std::io::BufWriter::with_capacity(1 << 20, fork_file),
scratch: vec![0u8; scratch_len],
partial: Vec::with_capacity(BLOCK),
offsets: vec![u32::try_from(table_len).map_err(|_| resource_fork_too_large())?],
encoded_offset: table_len,
}),
complete: false,
})
}
fn switch_to_plain(&mut self, current: &[u8]) -> Result<(), Error> {
let StreamingState::Encoding(mut encoding) =
std::mem::replace(&mut self.state, StreamingState::Closed)
else {
return Err(Error::Io {
context: "switch streaming writer to plain",
source: std::io::Error::from(std::io::ErrorKind::InvalidInput),
});
};
let (fallback, mut plain) =
decode_streaming_prefix(&self.path, &mut encoding, current, self.expected_len)?;
drop(encoding);
if let Err(source) = std::fs::remove_file(&self.path) {
let _ = std::fs::remove_file(&fallback);
return Err(Error::Io {
context: "remove streaming decmpfs temp",
source,
});
}
if let Err(source) = std::fs::rename(&fallback, &self.path) {
let _ = std::fs::remove_file(&fallback);
return Err(Error::Io {
context: "adopt streaming plain fallback",
source,
});
}
use std::io::Seek;
plain
.seek(std::io::SeekFrom::End(0))
.map_err(|source| Error::Io {
context: "seek streaming plain fallback",
source,
})?;
self.state = StreamingState::Plain(plain);
Ok(())
}
pub(crate) fn write_all(&mut self, mut input: &[u8]) -> Result<(), Error> {
let next_written = self
.written
.checked_add(input.len())
.filter(|&len| len <= self.expected_len)
.ok_or_else(|| Error::Io {
context: "stream exceeds expected length",
source: std::io::Error::from(std::io::ErrorKind::InvalidData),
})?;
while !input.is_empty() {
match &mut self.state {
StreamingState::Plain(file) => {
use std::io::Write;
file.write_all(input).map_err(|source| Error::Io {
context: "write streaming plain temp",
source,
})?;
input = &[];
}
StreamingState::Encoding(encoding) => {
let take = (BLOCK - encoding.partial.len()).min(input.len());
encoding.partial.extend_from_slice(&input[..take]);
input = &input[take..];
if encoding.partial.len() == BLOCK {
let block = std::mem::replace(&mut encoding.partial, Vec::with_capacity(BLOCK));
if !encoding.write_block(&block, self.expected_len)? {
self.switch_to_plain(&block)?;
}
}
}
StreamingState::Closed => {
return Err(Error::Io {
context: "write closed streaming writer",
source: std::io::Error::from(std::io::ErrorKind::BrokenPipe),
});
}
}
}
self.written = next_written;
Ok(())
}
pub(crate) fn finish(&mut self) -> Result<bool, Error> {
if self.written != self.expected_len {
return Err(Error::Io {
context: "finish incomplete streaming writer",
source: std::io::Error::from(std::io::ErrorKind::UnexpectedEof),
});
}
let partial = match &mut self.state {
StreamingState::Encoding(encoding) if !encoding.partial.is_empty() => Some(
std::mem::replace(&mut encoding.partial, Vec::with_capacity(BLOCK)),
),
_ => None,
};
if let Some(block) = partial {
let won = match &mut self.state {
StreamingState::Encoding(encoding) => encoding.write_block(&block, self.expected_len)?,
_ => false,
};
if !won {
self.switch_to_plain(&block)?;
}
}
let compressed = match std::mem::replace(&mut self.state, StreamingState::Closed) {
StreamingState::Plain(file) => {
file.sync_all().map_err(|source| Error::Io {
context: "sync streaming plain temp",
source,
})?;
false
}
StreamingState::Encoding(mut encoding) => {
use std::io::{Seek, Write};
let mut table = Vec::with_capacity(encoding.offsets.len() * std::mem::size_of::<u32>());
for offset in &encoding.offsets {
table.extend_from_slice(&offset.to_le_bytes());
}
encoding
.fork
.seek(std::io::SeekFrom::Start(0))
.and_then(|_| encoding.fork.write_all(&table))
.and_then(|_| encoding.fork.flush())
.map_err(|source| Error::Io {
context: "finish streaming resource fork",
source,
})?;
encoding
.fork
.get_ref()
.sync_all()
.map_err(|source| Error::Io {
context: "sync finished streaming resource fork",
source,
})?;
let cpath = cstring(&self.path)?;
setxattr(
&cpath,
c"com.apple.decmpfs",
&decmpfs_header(Codec::Lzfse, self.expected_len),
)?;
if unsafe { libc::fchflags(encoding.file.as_raw_fd(), UF_COMPRESSED) } != 0 {
return Err(io("fchflags streaming temp"));
}
encoding.file.sync_all().map_err(|source| Error::Io {
context: "sync streaming decmpfs temp",
source,
})?;
if streaming_kernel_matches(&self.path, &encoding, self.expected_len)? {
true
} else {
let (fallback, plain) =
decode_streaming_prefix(&self.path, &mut encoding, &[], self.expected_len)?;
drop(encoding);
std::fs::remove_file(&self.path).map_err(|source| Error::Io {
context: "remove failed streaming decmpfs oracle",
source,
})?;
if let Err(source) = std::fs::rename(&fallback, &self.path) {
let _ = std::fs::remove_file(&fallback);
return Err(Error::Io {
context: "publish streaming oracle fallback",
source,
});
}
plain.sync_all().map_err(|source| Error::Io {
context: "sync streaming oracle fallback",
source,
})?;
false
}
}
StreamingState::Closed => {
return Err(Error::Io {
context: "finish closed streaming writer",
source: std::io::Error::from(std::io::ErrorKind::BrokenPipe),
});
}
};
self.complete = true;
Ok(compressed)
}
}
impl Drop for StreamingWriter {
fn drop(&mut self) {
if !self.complete {
self.state = StreamingState::Closed;
let _ = std::fs::remove_file(&self.path);
}
}
}
fn decmpfs_header(codec: Codec, raw_len: usize) -> [u8; 16] {
let mut header = [0u8; 16];
header[..4].copy_from_slice(&DECMPFS_MAGIC.to_le_bytes());
header[4..8].copy_from_slice(&codec.compression_type().to_le_bytes());
header[8..].copy_from_slice(&(raw_len as u64).to_le_bytes());
header
}
fn setxattr(path: &std::ffi::CStr, name: &std::ffi::CStr, value: &[u8]) -> Result<(), Error> {

@@ -199,8 +994,3 @@ let rc = unsafe {

pub(crate) fn apply_inplace(path: &Path) -> Result<(), Error> {
let raw = std::fs::read(path).map_err(|source| Error::Io {
context: "read",
source,
})?;
pub(crate) fn apply_inplace(path: &Path, snapshot: &[u8]) -> Result<(), Error> {
// Fail-soft: skip if we can't write the original (by mode or ownership) — the

@@ -213,4 +1003,6 @@ // temp+rename below would otherwise replace even a file we can't open for write.

// `snapshot` is the file's bytes the caller already read for rollback — reuse
// it instead of a second full read.
let mode = std::fs::metadata(path).map(|m| m.permissions()).ok();
apply_bytes(path, &raw, mode)
apply_bytes(path, snapshot, mode)
}

@@ -220,7 +1012,8 @@

/// write-then-read-back. The decmpfs is built directly from `content`, dropped on a
/// sibling temp (empty data fork + the two xattrs + UF_COMPRESSED), then atomically
/// renamed over `path`. A crash can only leave the original or the finished file;
/// the rename also gives a fresh inode, the copy-break from any pnpm CAS hardlink
/// siblings. This is the one-pass core both `compress_bytes` (no original) and
/// `apply_inplace` (read first) share.
/// sibling temp (empty data fork + the two xattrs + UF_COMPRESSED when compression
/// wins; ordinary data fork on no gain), then atomically renamed over `path`. A
/// crash can only leave the original or the finished file; the rename also gives a
/// fresh inode, the copy-break from any pnpm CAS hardlink siblings. This is the
/// one-pass core both `compress_bytes` (no original) and `apply_inplace` (read
/// first) share.
pub(crate) fn apply_bytes(

@@ -231,12 +1024,17 @@ path: &Path,

) -> Result<(), Error> {
within_decmpfs_limit(content.len() as u64)?;
apply_bytes_with_streaming_threshold(path, content, mode, STREAMING_THRESHOLD)
}
let mut header = Vec::with_capacity(16);
header.extend_from_slice(&DECMPFS_MAGIC.to_le_bytes());
header.extend_from_slice(&CMP_LZVN_RESOURCE_FORK.to_le_bytes());
header.extend_from_slice(&(content.len() as u64).to_le_bytes());
let resource_fork = build_resource_fork(content).ok_or_else(|| Error::Io {
context: "lzvn encode",
source: std::io::Error::from(std::io::ErrorKind::InvalidData),
})?;
fn apply_bytes_with_streaming_threshold(
path: &Path,
content: &[u8],
mode: Option<std::fs::Permissions>,
streaming_threshold: usize,
) -> Result<(), Error> {
let stream = should_stream_resource_fork(content.len(), streaming_threshold);
let in_memory_resource_fork = if stream {
None
} else {
build_in_memory_resource_fork(content)?
};

@@ -248,19 +1046,73 @@ let dir = path.parent().ok_or_else(|| io("parent"))?;

.to_string_lossy();
let tmp = dir.join(format!(".{name}.decmpfs-{}.tmp", std::process::id()));
// Uniqueness beyond the PID: a crash can leave `.name.decmpfs-<pid>.tmp`, and a
// later run that reuses that PID (common after reboot) would fail `create_new`
// forever. PID + wall-clock nanos + a process-local counter makes a stale
// sibling collision astronomically unlikely while keeping `create_new`'s
// concurrent-writer safety.
static TMP_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let seq = TMP_SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
let tmp = dir.join(format!(
".{name}.decmpfs-{}-{nanos}-{seq}.tmp",
std::process::id()
));
let build = (|| -> Result<(), Error> {
let file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.create_new(true)
.open(&tmp)
.map_err(|source| Error::Io {
context: "create temp",
let create_temp = || {
std::fs::OpenOptions::new()
.read(true)
.write(true)
.create_new(true)
.open(&tmp)
.map_err(|source| Error::Io {
context: "create temp",
source,
})
};
let mut file = create_temp()?;
let ctmp = cstring(&tmp)?;
let codec = if stream {
build_streaming_resource_fork(&tmp, content)?
} else if let Some(resource_fork) = &in_memory_resource_fork {
// Install the payload before publishing metadata that tells the kernel to
// decode it. The temp inode is not visible at the destination yet either
// way, but this ordering also keeps direct temp-path observers safe.
setxattr(&ctmp, c"com.apple.ResourceFork", &resource_fork.bytes)?;
Some(resource_fork.codec)
} else {
None
};
if let Some(codec) = codec {
setxattr(
&ctmp,
c"com.apple.decmpfs",
&decmpfs_header(codec, content.len()),
)?;
if unsafe { libc::fchflags(file.as_raw_fd(), UF_COMPRESSED) } != 0 {
return Err(io("fchflags"));
}
} else {
// A losing streamed attempt left a partial resource fork on the temp.
// Recreate the inode rather than publish a plain file with stale fork data.
if stream {
drop(file);
std::fs::remove_file(&tmp).map_err(|source| Error::Io {
context: "remove losing streamed temp",
source,
})?;
file = create_temp()?;
}
use std::io::Write;
file.write_all(content).map_err(|source| Error::Io {
context: "plain temp write",
source,
})?;
let ctmp = cstring(&tmp)?;
setxattr(&ctmp, c"com.apple.decmpfs", &header)?;
setxattr(&ctmp, c"com.apple.ResourceFork", &resource_fork)?;
if unsafe { libc::fchflags(file.as_raw_fd(), UF_COMPRESSED) } != 0 {
return Err(io("fchflags"));
file.sync_all().map_err(|source| Error::Io {
context: "plain temp sync",
source,
})?;
}

@@ -277,2 +1129,12 @@ Ok(())

}
// Preserve ownership across the rewrite. Running as root (a global npm install,
// a Docker build) the temp is created owned by the current euid, so the rename
// would otherwise change the file's owner. Match the original's uid/gid.
// Best-effort: a no-op for a new path (nothing to preserve) and for a non-root
// process (chown to another owner is EPERM — but then the file was already
// ours, so nothing changes).
if let Ok(meta) = std::fs::metadata(path) {
use std::os::unix::fs::MetadataExt;
let _ = std::os::unix::fs::chown(&tmp, Some(meta.uid()), Some(meta.gid()));
}
std::fs::rename(&tmp, path).map_err(|source| {

@@ -287,3 +1149,15 @@ let _ = std::fs::remove_file(&tmp);

/// Copy-on-write clone via `clonefile(2)` — shares the extents AND the decmpfs
/// state, so a compressed source stays compressed at zero cost. `Ok(false)`
/// means "cannot clone here" (cross-volume, unsupported FS, …) and the caller
/// falls back to a byte copy; a failed clonefile never leaves a partial
/// destination.
pub(crate) fn clone_file(src: &Path, dest: &Path) -> Result<bool, Error> {
let csrc = cstring(src)?;
let cdest = cstring(dest)?;
Ok(unsafe { libc::clonefile(csrc.as_ptr(), cdest.as_ptr(), 0) } == 0)
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {

@@ -312,3 +1186,3 @@ use super::*;

);
apply_inplace(&path).unwrap();
apply_inplace(&path, &raw).unwrap();
assert!(is_already_compressed(&path).unwrap(), "UF_COMPRESSED set");

@@ -330,2 +1204,88 @@ assert_eq!(

#[test]
fn incremental_writer_streams_lzfse_blocks_into_a_kernel_readable_file() {
let dir =
std::env::temp_dir().join(format!("decmpfs-incremental-oracle-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("model.bin");
let raw = b"incremental lzfse resource fork ".repeat((2 << 20) / 34 + 1);
let mut writer = StreamingWriter::new(&path, raw.len()).unwrap();
for chunk in raw.chunks(17_003) {
writer.write_all(chunk).unwrap();
}
assert!(writer.finish().unwrap(), "compressible stream must win");
assert!(is_already_compressed(&path).unwrap());
assert_eq!(std::fs::read(&path).unwrap(), raw);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn incremental_writer_reconstructs_plain_bytes_when_compression_loses() {
let dir = std::env::temp_dir().join(format!(
"decmpfs-incremental-fallback-{}",
std::process::id()
));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("random.bin");
let mut raw = Vec::with_capacity(2 << 20);
let mut x: u64 = 0x9e37_79b9_7f4a_7c15;
while raw.len() < (2 << 20) {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
raw.extend_from_slice(&x.to_le_bytes());
}
let mut writer = StreamingWriter::new(&path, raw.len()).unwrap();
for chunk in raw.chunks(17_003) {
writer.write_all(chunk).unwrap();
}
assert!(
!writer.finish().unwrap(),
"incompressible stream stays plain"
);
assert!(!is_already_compressed(&path).unwrap());
assert_eq!(std::fs::read(&path).unwrap(), raw);
std::fs::remove_dir_all(&dir).ok();
}
// Opt-in perf probe (ignored in CI — timing is machine-specific). Reports the
// decmpfs write time for a ~40 MiB addon; run serial vs parallel with
// cargo test -p decmpfs write_time -- --ignored --nocapture
// DECMPFS_SERIAL=1 cargo test -p decmpfs write_time -- --ignored --nocapture
#[test]
#[ignore]
fn write_time_probe() {
let dir = std::env::temp_dir().join(format!("decmpfs-time-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("addon.node");
let mut raw: Vec<u8> = Vec::with_capacity(40 << 20);
let mut x: u64 = 0x9e37_79b9_7f4a_7c15;
while raw.len() < (40 << 20) {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
raw.extend_from_slice(&x.to_le_bytes());
raw.extend_from_slice(b"native addon .node text segment padding ");
}
if !matches!(detect(&dir), Ok(Support::Supported)) {
std::fs::remove_dir_all(&dir).ok();
return;
}
let cores = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1);
let serial = std::env::var_os("DECMPFS_SERIAL").is_some();
let start = std::time::Instant::now();
apply_bytes(&path, &raw, None).unwrap();
let ms = start.elapsed().as_secs_f64() * 1e3;
eprintln!(
"decmpfs write {}MiB — {} ({} cores): {:.1} ms",
raw.len() >> 20,
if serial { "serial" } else { "parallel" },
cores,
ms,
);
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn detect_and_flags_error_on_a_missing_path() {

@@ -351,7 +1311,17 @@ let p = std::path::Path::new("/no/such/decmpfs/path/x.bin");

let path = dir.join("f.bin");
std::fs::write(&path, b"\x7fELF unreadable").unwrap();
let content = b"\x7fELF unreadable";
std::fs::write(&path, content).unwrap();
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o000)).unwrap();
let out = apply_inplace(&path);
// apply_inplace no longer reads the file (the caller passes the snapshot it
// already holds); the fail-soft guard is now the W_OK access check, which
// rejects a file we cannot write before the temp+rename would replace it.
let out = apply_inplace(&path, content);
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).ok();
assert!(matches!(out, Err(Error::Io { context: "read", .. })));
assert!(matches!(
out,
Err(Error::Io {
context: "access",
..
})
));
std::fs::remove_dir_all(&dir).ok();

@@ -363,3 +1333,9 @@ }

let out = setxattr(c"/no/such/decmpfs/path", c"com.apple.decmpfs", b"x");
assert!(matches!(out, Err(Error::Io { context: "setxattr", .. })));
assert!(matches!(
out,
Err(Error::Io {
context: "setxattr",
..
})
));
}

@@ -376,2 +1352,107 @@

#[test]
fn build_resource_fork_zero_length_is_no_gain() {
assert!(
build_resource_fork(&[]).unwrap().is_none(),
"a resource fork cannot make an empty file smaller"
);
}
#[test]
fn streaming_threshold_keeps_vite_native_addons_on_the_fast_path() {
// The largest Darwin ARM64 addon in the 2026-07-16 Vite-family sample was
// SWC at 36.563 MiB. The complete observed set must stay comfortably below
// the in-memory cutoff, while the first byte beyond it streams.
assert!(!should_stream_resource_fork(37 << 20, STREAMING_THRESHOLD));
assert!(!should_stream_resource_fork(
STREAMING_THRESHOLD,
STREAMING_THRESHOLD
));
assert!(should_stream_resource_fork(
STREAMING_THRESHOLD + 1,
STREAMING_THRESHOLD
));
}
#[test]
fn kernel_roundtrips_forced_streaming_lzfse() {
let dir = std::env::temp_dir().join(format!("decmpfs-streaming-oracle-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("f.bin");
let raw = b"streamed lzfse decmpfs resource fork oracle ".repeat((2 << 20) / 46 + 1);
std::fs::write(&path, &raw).unwrap();
if matches!(detect(&path).unwrap(), Support::Supported) {
apply_bytes_with_streaming_threshold(&path, &raw, None, 0).unwrap();
assert!(is_already_compressed(&path).unwrap(), "UF_COMPRESSED set");
assert_eq!(
std::fs::read(&path).unwrap(),
raw,
"kernel read-back must decode the streamed type-12 resource fork"
);
let cpath = cstring(&path).unwrap();
let mut header = [0u8; 16];
let len = unsafe {
libc::getxattr(
cpath.as_ptr(),
c"com.apple.decmpfs".as_ptr(),
header.as_mut_ptr().cast(),
header.len(),
0,
XATTR_NOFOLLOW | 0x0020, // XATTR_SHOWCOMPRESSION
)
};
assert_eq!(len, header.len() as isize);
assert_eq!(u32::from_le_bytes(header[4..8].try_into().unwrap()), 12);
}
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn in_memory_path_falls_back_to_lzfse_when_lzvn_has_no_gain() {
// Skewed symbol frequencies give LZFSE's entropy coder something to exploit
// without manufacturing the repeated strings that LZVN specializes in.
let mut raw = Vec::with_capacity(1 << 20);
let mut x: u64 = 0x9e37_79b9_7f4a_7c15;
while raw.len() < raw.capacity() {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
raw.push(if x.is_multiple_of(4) {
0
} else {
(x >> 32) as u8
});
}
assert!(
build_resource_fork_with_codec(&raw, Codec::Lzvn)
.unwrap()
.is_none(),
"fixture must reach the fallback"
);
let candidate = build_in_memory_resource_fork(&raw)
.unwrap()
.expect("LZFSE should exploit the skewed symbols");
assert_eq!(candidate.codec, Codec::Lzfse);
assert!(candidate.bytes.len() < raw.len());
}
#[test]
fn build_resource_fork_last_offset_equals_length() {
// Invariant across sizes that actually encode: the final table offset equals
// the total blob length. (Tiny/incompressible inputs return None — the codec
// declines — which is a separate, correct path.)
for size in [512usize, BLOCK, BLOCK + 1, BLOCK * 3 + 7] {
let raw = vec![0x41u8; size];
let Some(rf) = build_resource_fork(&raw).unwrap() else {
continue;
};
let num_blocks = size.div_ceil(BLOCK);
let last_idx = num_blocks * 4; // offset[num_blocks] is the last entry
let last = u32::from_le_bytes(rf[last_idx..last_idx + 4].try_into().unwrap()) as usize;
assert_eq!(last, rf.len(), "size {size}: last offset != buffer length");
}
}
#[test]
fn cstring_rejects_an_interior_nul() {

@@ -408,6 +1489,28 @@ use std::os::unix::ffi::OsStrExt;

#[test]
fn within_decmpfs_limit_rejects_oversized() {
// No 4 GiB file needed — the predicate is pure.
assert!(within_decmpfs_limit(MAX_RAW).is_ok());
match within_decmpfs_limit(MAX_RAW + 1).unwrap_err() {
fn resource_fork_plan_accepts_raw_files_beyond_the_old_limit() {
// The raw byte count is stored as u64. Only resource-fork offsets are u32,
// so a >3.9 GB input is valid whenever its encoded fork fits in u32.
let raw_len = 4_100_000_000usize;
let num_blocks = raw_len.div_ceil(BLOCK);
assert!(matches!(
plan_resource_fork(raw_len, num_blocks, 3_000_000_000).unwrap(),
ResourceForkPlan::Compressed { .. }
));
}
#[test]
fn resource_fork_plan_accepts_raw_files_beyond_four_gib_when_the_fork_fits() {
let raw_len = 5_000_000_000usize;
let num_blocks = raw_len.div_ceil(BLOCK);
assert!(matches!(
plan_resource_fork(raw_len, num_blocks, 3_000_000_000).unwrap(),
ResourceForkPlan::Compressed { .. }
));
}
#[test]
fn resource_fork_plan_rejects_a_compressed_fork_past_u32() {
let raw_len = 5_000_000_000usize;
let num_blocks = raw_len.div_ceil(BLOCK);
match plan_resource_fork(raw_len, num_blocks, 4_400_000_000).unwrap_err() {
Error::Io { source, .. } => assert_eq!(source.raw_os_error(), Some(libc::EFBIG)),

@@ -418,5 +1521,28 @@ other => panic!("expected EFBIG Io, got {other:?}"),

// Incompressible data → blocks are stored verbatim (compress_block returns the
// chunk). The kernel must still read them back identically.
#[test]
fn gemini_nano_lzvn_resource_fork_is_no_gain() {
// Chrome 150's v3Nano weights.bin measured with this exact 64 KiB LZVN
// encoder: the encoded blocks expand enough to cross the u32 fork ceiling.
assert_eq!(
plan_resource_fork(4_269_932_544, 65_154, 4_364_775_458).unwrap(),
ResourceForkPlan::Plain
);
}
#[test]
fn gemini_nano_lzfse_resource_fork_fits_and_wins() {
// The streamed type-12 run encoded the same 65,154 blocks to this payload;
// with its 260,620-byte offset table the fork is safely below u32::MAX.
assert_eq!(
plan_resource_fork(4_269_932_544, 65_154, 3_598_249_560).unwrap(),
ResourceForkPlan::Compressed {
table_len: 260_620,
total_len: 3_598_510_180,
}
);
}
// Incompressible data → LZVN would expand the resource fork, so keep an
// ordinary data fork. The bytes and compression-state signal must agree.
#[test]
fn kernel_roundtrips_incompressible_blocks() {

@@ -436,11 +1562,41 @@ let dir = std::env::temp_dir().join(format!("decmpfs-raw-{}", std::process::id()));

if matches!(detect(&path).unwrap(), Support::Supported) {
apply_inplace(&path).unwrap();
assert!(matches!(
crate::compress_file(&path).unwrap(),
crate::Outcome::NoGain { .. }
));
assert_eq!(
std::fs::read(&path).unwrap(),
raw,
"verbatim blocks read back"
"plain fallback reads back identically"
);
assert!(
!is_already_compressed(&path).unwrap(),
"no-gain input must not carry UF_COMPRESSED"
);
}
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn apply_bytes_preserves_ownership_of_an_overwritten_file() {
// Non-root exercises the chown path over an existing file — owner is our own
// uid, so preservation is a no-op we assert stays stable + non-corrupting.
// The root path (a file owned by a different uid) is verified in CI.
let dir = std::env::temp_dir().join(format!("decmpfs-own-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("f");
std::fs::write(&path, vec![0u8; 4096]).unwrap();
if !matches!(detect(&path), Ok(Support::Supported)) {
std::fs::remove_dir_all(&dir).ok();
return;
}
use std::os::unix::fs::MetadataExt;
let before_uid = std::fs::metadata(&path).unwrap().uid();
let content = vec![0xABu8; 8192];
apply_bytes(&path, &content, None).unwrap();
let meta = std::fs::metadata(&path).unwrap();
assert_eq!(meta.uid(), before_uid, "owner preserved across the rewrite");
assert_eq!(std::fs::read(&path).unwrap(), content, "content intact");
std::fs::remove_dir_all(&dir).ok();
}
}
+73
-38
src/safety.rs

@@ -37,3 +37,3 @@ //! Shared orchestration that gates every backend identically. Written once here

// -> Skipped(Busy). A genuine, unclassifiable I/O error still propagates.
if let Err(err) = backend.apply_inplace(path) {
if let Err(err) = backend.apply_inplace(path, &snapshot) {
if let Error::Io { source, .. } = &err {

@@ -63,3 +63,3 @@ if let Some(reason) = classify_skip(source) {

if verify::magic_prefix(path)? != magic_before {
restore(path, snapshot);
restore(path, snapshot)?;
return Ok(classify_outcome(false, before, before, None));

@@ -105,9 +105,25 @@ }

}
match err.raw_os_error() {
Some(1) | Some(13) | Some(30) => Some(SkipReason::PermissionDenied), // EPERM/EACCES/EROFS
Some(16) | Some(26) => Some(SkipReason::Busy), // EBUSY/ETXTBSY
Some(27) => Some(SkipReason::TooLarge), // EFBIG
classify_errno(err.raw_os_error()?)
}
// Per-platform errno classification. Windows `raw_os_error()` is the Win32 space,
// which does NOT coincide with POSIX (e.g. 32 = SHARING_VIOLATION on Windows but
// EPIPE on unix), so the two maps are mutually exclusive by cfg.
#[cfg(not(windows))]
fn classify_errno(code: i32) -> Option<SkipReason> {
match code {
1 | 13 | 30 => Some(SkipReason::PermissionDenied), // EPERM/EACCES/EROFS
16 | 26 => Some(SkipReason::Busy), // EBUSY/ETXTBSY
27 => Some(SkipReason::TooLarge), // EFBIG
_ => None,
}
}
#[cfg(windows)]
fn classify_errno(code: i32) -> Option<SkipReason> {
match code {
5 | 19 => Some(SkipReason::PermissionDenied), // ACCESS_DENIED / WRITE_PROTECT
32 | 33 => Some(SkipReason::Busy), // SHARING_VIOLATION / LOCK_VIOLATION
_ => None,
}
}

@@ -153,8 +169,4 @@ /// One-pass guarded write of `content` to `path` as an OS-compressed file. Reached

let after = verify::on_disk_bytes(path)?;
let read_back = std::fs::read(path).map_err(|source| Error::Io {
context: "read-back",
source,
})?;
if read_back != content {
restore(path, content);
if !verify::readback_matches(path, content)? {
restore(path, content)?;
return Ok(Outcome::Skipped {

@@ -174,20 +186,30 @@ reason: SkipReason::IntegrityRevert,

/// Best-effort atomic restore of the pre-apply bytes (sibling temp + rename).
fn restore(path: &Path, bytes: &[u8]) {
/// Atomic restore of the pre-apply bytes (sibling temp + rename). Returns `Err`
/// when the rollback itself fails (e.g. `ENOSPC`, a read-only dir) — the caller
/// MUST surface that as a hard error rather than a benign `Skipped`, else a
/// corrupted file is left on disk while the outcome reads as non-fatal.
fn restore(path: &Path, bytes: &[u8]) -> Result<(), Error> {
use std::io::Write;
let Some(dir) = path.parent() else {
return;
};
let dir = path.parent().ok_or_else(|| Error::Io {
context: "rollback restore: path has no parent",
source: std::io::Error::from(std::io::ErrorKind::InvalidInput),
})?;
let tmp = dir.join(format!(".decmpfs-restore-{}.tmp", std::process::id()));
let wrote = std::fs::File::create(&tmp).and_then(|mut file| {
file.write_all(bytes)?;
file.sync_all()
});
if wrote.is_ok() && std::fs::rename(&tmp, path).is_ok() {
return;
}
let _ = std::fs::remove_file(&tmp);
let wrote = std::fs::File::create(&tmp)
.and_then(|mut file| {
file.write_all(bytes)?;
file.sync_all()
})
.and_then(|()| std::fs::rename(&tmp, path));
wrote.map_err(|source| {
let _ = std::fs::remove_file(&tmp);
Error::Io {
context: "rollback restore",
source,
}
})
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {

@@ -212,3 +234,3 @@ use super::*;

detect: Support::Supported,
apply_errno: Some(2),
apply_error: Some(std::io::ErrorKind::NotFound),
};

@@ -226,7 +248,11 @@ let out = apply_guarded(&backend, &path);

);
for errno in [1, 13, 30] {
#[cfg(not(windows))]
let os_errors = [1, 13, 30]; // EPERM / EACCES / EROFS
#[cfg(windows)]
let os_errors = [5, 19]; // ACCESS_DENIED / WRITE_PROTECT
for code in os_errors {
assert_eq!(
classify_skip(&std::io::Error::from_raw_os_error(errno)),
classify_skip(&std::io::Error::from_raw_os_error(code)),
Some(SkipReason::PermissionDenied),
"errno {errno}"
"OS error {code}"
);

@@ -238,7 +264,11 @@ }

fn busy_errors_become_skipped() {
for errno in [16, 26] {
#[cfg(not(windows))]
let os_errors = [16, 26]; // EBUSY / ETXTBSY
#[cfg(windows)]
let os_errors = [32, 33]; // SHARING_VIOLATION / LOCK_VIOLATION
for code in os_errors {
assert_eq!(
classify_skip(&std::io::Error::from_raw_os_error(errno)),
classify_skip(&std::io::Error::from_raw_os_error(code)),
Some(SkipReason::Busy),
"errno {errno}"
"OS error {code}"
);

@@ -248,2 +278,3 @@ }

#[cfg(not(windows))]
#[test]

@@ -295,3 +326,3 @@ fn efbig_becomes_too_large() {

std::fs::write(&path, b"corrupted-by-a-broken-backend").unwrap();
restore(&path, b"the original loadable bytes");
restore(&path, b"the original loadable bytes").unwrap();
assert_eq!(

@@ -329,3 +360,3 @@ std::fs::read(&path).unwrap(),

detect: Support::Supported,
apply_errno: None,
apply_error: None,
};

@@ -344,5 +375,6 @@ let out = compress_bytes_guarded(&backend, &path, content).unwrap();

#[test]
fn restore_is_a_noop_when_the_path_has_no_parent() {
// "/" has no parent → restore returns early without touching anything.
restore(std::path::Path::new("/"), b"x");
fn restore_errors_when_the_path_has_no_parent() {
// "/" has no parent → the rollback can't write a sibling temp, so it must
// surface an Err (a silent no-op would report a corrupt file as benign).
assert!(restore(std::path::Path::new("/"), b"x").is_err());
}

@@ -409,3 +441,6 @@

std::fs::create_dir_all(&target).unwrap();
restore(&target, b"bytes");
assert!(
restore(&target, b"bytes").is_err(),
"rename-over-dir must Err"
);
let tmp = dir.join(format!(".decmpfs-restore-{}.tmp", std::process::id()));

@@ -412,0 +447,0 @@ assert!(!tmp.exists(), "temp left behind");

+7
-1
src/unsupported.rs

@@ -14,3 +14,3 @@ //! Fallback for any OS without a backend — always Unsupported{PlatformBuild}.

pub(crate) fn apply_inplace(_path: &Path) -> Result<(), Error> {
pub(crate) fn apply_inplace(_path: &Path, _snapshot: &[u8]) -> Result<(), Error> {
Ok(())

@@ -36,1 +36,7 @@ }

}
/// No clone primitive on an unsupported platform — the caller takes the plain
/// byte-copy path.
pub(crate) fn clone_file(_src: &Path, _dest: &Path) -> Result<bool, Error> {
Ok(false)
}
+88
-0
src/verify.rs

@@ -74,7 +74,95 @@ //! Platform-agnostic measurement + loadability surface.

/// Stream-compare the file at `path` against `expected`, letting the kernel
/// decompress transparently, through a fixed reusable 64 KiB buffer — never
/// materializing a second full copy of the file (the read-back oracle runs on
/// every compressed write, so a `std::fs::read` here would heap-allocate a whole
/// extra copy of every addon). Returns `false` on the first differing byte or any
/// length divergence (a short OR long read-back), and short-circuits on mismatch.
pub(crate) fn readback_matches(path: &Path, expected: &[u8]) -> Result<bool, Error> {
let mut file = std::fs::File::open(path).map_err(|source| Error::Io {
context: "read-back",
source,
})?;
let mut buf = [0u8; 64 * 1024];
let mut off = 0usize;
loop {
let n = file.read(&mut buf).map_err(|source| Error::Io {
context: "read-back",
source,
})?;
if n == 0 {
break;
}
// A read-back that overruns `expected`, or a chunk that differs, is a
// mismatch — bail without reading the rest.
if off + n > expected.len() || buf[..n] != expected[off..off + n] {
return Ok(false);
}
off += n;
}
// Equal only if the on-disk length matched exactly (a truncated read-back is a
// mismatch the loop can't otherwise catch).
Ok(off == expected.len())
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
fn scratch(tag: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("decmpfs-verify-{tag}-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
dir
}
#[test]
fn readback_matches_identical_content() {
let dir = scratch("rb-eq");
let path = dir.join("f");
// Larger than one 64 KiB buffer to exercise the multi-chunk loop.
let content = vec![0xABu8; 200 * 1024];
std::fs::write(&path, &content).unwrap();
assert!(readback_matches(&path, &content).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn readback_detects_a_differing_byte() {
let dir = scratch("rb-diff");
let path = dir.join("f");
let content = vec![0x11u8; 100 * 1024];
let mut on_disk = content.clone();
on_disk[80 * 1024] = 0x22;
std::fs::write(&path, &on_disk).unwrap();
assert!(!readback_matches(&path, &content).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn readback_detects_a_short_file() {
let dir = scratch("rb-short");
let path = dir.join("f");
std::fs::write(&path, vec![0x33u8; 4096]).unwrap();
// Expected is longer than what's on disk.
assert!(!readback_matches(&path, &vec![0x33u8; 8192]).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn readback_detects_a_long_file() {
let dir = scratch("rb-long");
let path = dir.join("f");
std::fs::write(&path, vec![0x44u8; 8192]).unwrap();
// Expected is shorter than what's on disk (read-back overruns).
assert!(!readback_matches(&path, &vec![0x44u8; 4096]).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn readback_errors_on_a_missing_path() {
assert!(readback_matches(std::path::Path::new("/no/such/rb/x"), b"x").is_err());
}
#[test]
fn measures_allocation_and_reads_magic() {

@@ -81,0 +169,0 @@ let dir = std::env::temp_dir().join(format!("decmpfs-verify-{}", std::process::id()));

+195
-23
src/windows.rs

@@ -24,3 +24,3 @@ //! Windows backend — NTFS LZNT1 via `FSCTL_SET_COMPRESSION`.

use windows_sys::Win32::Storage::FileSystem::{
CreateFileW, GetFileAttributesW, GetVolumeInformationByHandleW,
CreateFileW, GetFileAttributesW, GetVolumeInformationByHandleW, MoveFileExW,
};

@@ -35,3 +35,7 @@ use windows_sys::Win32::System::IO::DeviceIoControl;

const OPEN_EXISTING: u32 = 3;
const CREATE_ALWAYS: u32 = 2;
const CREATE_NEW: u32 = 1;
// MoveFileExW flags: replace an existing target atomically, and don't return
// until the rename is flushed to disk.
const MOVEFILE_REPLACE_EXISTING: u32 = 0x0000_0001;
const MOVEFILE_WRITE_THROUGH: u32 = 0x0000_0008;
const FSCTL_SET_COMPRESSION: u32 = 0x0009_C040;

@@ -42,2 +46,6 @@ const COMPRESSION_FORMAT_DEFAULT: u16 = 1; // LZNT1

const INVALID_FILE_ATTRIBUTES: u32 = u32::MAX;
// Required to obtain a handle to a DIRECTORY (CreateFileW fails on a dir without
// it). `detect()` probes the parent directory of a not-yet-created target on a
// fresh install, so every open must set it; harmless on a regular file.
const FILE_FLAG_BACKUP_SEMANTICS: u32 = 0x0200_0000;

@@ -76,3 +84,3 @@ fn wide(path: &Path) -> Vec<u16> {

disposition,
0,
FILE_FLAG_BACKUP_SEMANTICS,
std::ptr::null_mut(),

@@ -143,3 +151,6 @@ )

pub(crate) fn apply_inplace(path: &Path) -> Result<(), Error> {
// `_snapshot` is unused on Windows: FSCTL_SET_COMPRESSION flags the EXISTING
// file in place (the kernel recompresses), so there's no temp+rewrite that would
// reuse the caller's bytes.
pub(crate) fn apply_inplace(path: &Path, _snapshot: &[u8]) -> Result<(), Error> {
let handle = open(path, GENERIC_READ | GENERIC_WRITE)?;

@@ -149,6 +160,53 @@ set_compression(handle.0)

/// Write `content` to `path` as a fresh NTFS-compressed file in ONE pass: create
/// the file, FSCTL_SET_COMPRESSION on the EMPTY handle (so writes compress on the
/// way in — never a write-then-recompress), then write the bytes through the same
/// handle. `mode` is unused on Windows (no POSIX bits). Shared by `compress_bytes`.
/// A collision-resistant sibling temp path. PID + wall-clock nanos + a
/// process-local counter so two writers on the SAME destination never share a
/// temp; paired with CREATE_NEW a collision errors rather than clobbering.
/// Mirrors the macOS/Linux backends' scheme.
fn unique_tmp(dir: &Path, name: &str) -> std::path::PathBuf {
static TMP_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
let seq = TMP_SEQ.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let nanos = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
dir.join(format!(
".{name}.decmpfs-{}-{nanos}-{seq}.tmp",
std::process::id()
))
}
/// Atomically publish `tmp` over `dest` (replace + write-through).
fn move_file_replace(tmp: &Path, dest: &Path) -> Result<(), Error> {
let wtmp = wide(tmp);
let wdest = wide(dest);
let ok = unsafe {
MoveFileExW(
wtmp.as_ptr(),
wdest.as_ptr(),
MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH,
)
};
if ok == 0 {
return Err(io("MoveFileExW"));
}
Ok(())
}
pub(crate) fn prepare_stream(file: &std::fs::File) -> Result<(), Error> {
use std::os::windows::io::AsRawHandle;
set_compression(file.as_raw_handle() as HANDLE)
}
pub(crate) fn publish_stream(tmp: &Path, dest: &Path) -> Result<(), Error> {
move_file_replace(tmp, dest)
}
/// Write `content` to `path` as a fresh NTFS-compressed file in ONE pass, then
/// publish it atomically: create a sibling temp with CREATE_NEW,
/// FSCTL_SET_COMPRESSION on the EMPTY handle (so writes compress on the way in —
/// never a write-then-recompress), write + flush, then MoveFileEx over `path`.
/// The temp+rename matches the macOS/Linux backends and fixes the old
/// CREATE_ALWAYS-in-place write, which truncated `path` before streaming — a
/// crash or a concurrent open saw a partial `.node`. `mode` is unused on Windows
/// (no POSIX bits). Shared by `compress_bytes`.
pub(crate) fn apply_bytes(

@@ -159,17 +217,38 @@ path: &Path,

) -> Result<(), Error> {
let handle = open_with(path, GENERIC_READ | GENERIC_WRITE, CREATE_ALWAYS)?;
set_compression(handle.0)?;
// Write through the same handle the compression flag was set on, so every block
// lands compressed. WriteFile via a std File would need a second open; instead
// borrow the raw handle into a File for the write, then forget it so Drop on the
// Handle (not the File) does the single CloseHandle.
use std::os::windows::io::FromRawHandle;
let mut file = unsafe { std::fs::File::from_raw_handle(handle.0 as _) };
let res = file.write_all(content).and_then(|()| file.flush());
// Don't let File's Drop close the handle — the Handle wrapper owns it.
std::mem::forget(file);
res.map_err(|source| Error::Io {
context: "write",
source,
})
let dir = path.parent().ok_or_else(|| io("no parent dir"))?;
let name = path.file_name().map_or_else(
|| std::borrow::Cow::Borrowed("addon"),
|n| n.to_string_lossy(),
);
let tmp = unique_tmp(dir, &name);
let result = (|| {
let handle = open_with(&tmp, GENERIC_READ | GENERIC_WRITE, CREATE_NEW)?;
set_compression(handle.0)?;
// Borrow the raw handle into a File for the write, then forget it so the
// Handle wrapper (not File) does the single CloseHandle.
use std::os::windows::io::FromRawHandle;
let mut file = unsafe { std::fs::File::from_raw_handle(handle.0 as _) };
let res = file
.write_all(content)
.and_then(|()| file.flush())
.and_then(|()| file.sync_all());
std::mem::forget(file);
res.map_err(|source| Error::Io {
context: "write temp",
source,
})
})();
match result {
Ok(()) => move_file_replace(&tmp, path).inspect_err(|_| {
// A failed publish (e.g. the dest is loaded → SHARING_VIOLATION) leaves the
// original file untouched; drop the orphan temp.
let _ = std::fs::remove_file(&tmp);
}),
Err(err) => {
let _ = std::fs::remove_file(&tmp);
Err(err)
}
}
}

@@ -183,1 +262,94 @@

}
/// NTFS has no reflink/clone primitive (ReFS block-cloning is out of scope) —
/// always report "cannot clone" so the caller takes the byte-copy path, which
/// re-applies `FSCTL_SET_COMPRESSION` at the destination.
pub(crate) fn clone_file(_src: &Path, _dest: &Path) -> Result<bool, Error> {
Ok(false)
}
#[cfg(test)]
#[cfg_attr(coverage_nightly, coverage(off))]
mod tests {
use super::*;
fn scratch(tag: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("decmpfs-win-{tag}-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
dir
}
fn on_ntfs(dir: &Path) -> bool {
let probe = dir.join(".ntfs-probe");
std::fs::write(&probe, b"x").unwrap();
let yes = matches!(detect(&probe), Ok(Support::Supported));
std::fs::remove_file(&probe).ok();
yes
}
#[test]
fn unique_tmp_never_collides_and_is_well_formed() {
let dir = std::env::temp_dir();
let a = unique_tmp(&dir, "addon.node");
let b = unique_tmp(&dir, "addon.node");
assert_ne!(a, b, "successive temps must differ (the seq counter)");
for p in [&a, &b] {
let f = p.file_name().unwrap().to_string_lossy();
assert!(
f.starts_with(".addon.node.decmpfs-"),
"unexpected temp name: {f}"
);
assert!(f.ends_with(".tmp"), "unexpected temp name: {f}");
}
}
#[test]
fn clone_file_always_declines_on_ntfs() {
let dir = scratch("clone");
assert!(!clone_file(&dir.join("a"), &dir.join("b")).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn detect_is_ok_and_a_fresh_file_is_not_compressed() {
let dir = scratch("detect");
let path = dir.join("f.bin");
std::fs::write(&path, b"MZ plain").unwrap();
assert!(detect(&path).is_ok());
assert!(!is_already_compressed(&path).unwrap());
std::fs::remove_dir_all(&dir).ok();
}
#[test]
fn apply_bytes_round_trips_atomically_and_races_converge_on_ntfs() {
let dir = scratch("rt");
if !on_ntfs(&dir) {
std::fs::remove_dir_all(&dir).ok();
return;
}
let content = vec![0xCDu8; 128 * 1024];
let dest = dir.join("addon.node");
// Eight writers racing on the SAME dest must all converge to identical
// bytes — the CREATE_NEW unique-temp + MoveFileEx replace prevents the
// partial/interleaved write the old CREATE_ALWAYS-in-place path allowed.
std::thread::scope(|s| {
for _ in 0..8 {
let dest = dest.clone();
let content = content.clone();
s.spawn(move || {
let _ = apply_bytes(&dest, &content, None);
});
}
});
assert_eq!(
std::fs::read(&dest).unwrap(),
content,
"bytes survive the race"
);
assert!(
is_already_compressed(&dest).unwrap(),
"landed NTFS-compressed"
);
std::fs::remove_dir_all(&dir).ok();
}
}
-110
.github/workflows/provenance.yml
name: 📦 Publish to npm
# Dependencies: none — self-contained.
#
# The publish step is INLINED (not delegated to a reusable workflow) because npm
# OIDC trusted publishing keys on the workflow file that contains the publish step:
# `job_workflow_ref` must be a stable
# decmpfs/decmpfs/.github/workflows/provenance.yml@<ref> for the trusted-publisher
# config to match. (socket-addon delegates to socket-registry's reusable, but that
# is Socket infra; decmpfs is independent, so it follows socket-lib's inlined
# pattern.)
#
# These are addons (a `.node` dlopen'd into Node), so each platform builds on its
# own matrix host, then the publish job publishes every @decmpfs/<triple> package
# plus the main `decmpfs` package with provenance.
on:
workflow_dispatch:
inputs:
dry-run:
description: 'Dry run (default: true)'
required: false
default: true
type: boolean
dist-tag:
description: 'npm dist-tag (latest, next, beta, …)'
required: false
default: latest
type: string
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false # Don't cancel publishing.
jobs:
build:
name: Build ${{ matrix.triple }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
- { triple: darwin-arm64, runner: macos-14 }
- { triple: darwin-x64, runner: macos-13 }
- { triple: linux-x64-gnu, runner: ubuntu-latest }
- { triple: linux-arm64-gnu, runner: ubuntu-24.04-arm }
- { triple: win32-x64-msvc, runner: windows-latest }
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 (2026-06-29)
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0 (2026-06-29)
with:
node-version: 24
- name: Build addon
working-directory: node
run: npm run build
- name: Test addon
working-directory: node
run: npm test
- name: Assemble platform package
working-directory: node
run: node scripts/make-npm-dirs.mts
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 (2026-06-29)
with:
name: ${{ matrix.triple }}
path: node/npm/${{ matrix.triple }}/
if-no-files-found: error
publish:
name: Publish to npm
needs: build
runs-on: ubuntu-latest
timeout-minutes: 15
# Real publishes run in the `release` environment (reviewer gate + a tighter
# OIDC trusted-publisher match); dry runs skip it for friction-free testing.
environment: ${{ inputs.dry-run == 'true' && '' || 'release' }}
permissions:
# Needed for npm provenance via OIDC trusted publishing.
id-token: write
contents: read
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 (2026-06-29)
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0 (2026-06-29)
with:
node-version: 24
registry-url: https://registry.npmjs.org/
- name: Use a trusted-publishing-capable npm
run: npm install --global npm@latest
- name: Download platform packages
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 (2026-06-29)
with:
path: node/npm
- name: Publish
working-directory: node
env:
DRY_RUN: ${{ inputs.dry-run }}
DIST_TAG: ${{ inputs.dist-tag }}
run: |
ARGS="--access public --tag ${DIST_TAG:-latest}"
if [ "$DRY_RUN" = "true" ]; then
ARGS="$ARGS --dry-run"
fi
# Platform packages first, then the main package that depends on them.
for dir in npm/*/; do
npm publish "$dir" $ARGS
done
npm publish . $ARGS
-1
.gitignore
/target
-19
assets/coverage-score.svg
<svg xmlns="http://www.w3.org/2000/svg" width="96" height="20" role="img" aria-label="coverage: 97%">
<title>coverage: 97%</title>
<linearGradient id="g" x2="0" y2="100%">
<stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
<stop offset="1" stop-opacity=".1"/>
</linearGradient>
<clipPath id="r"><rect width="96" height="20" rx="3" fill="#fff"/></clipPath>
<g clip-path="url(#r)">
<rect width="60" height="20" fill="#555"/>
<rect x="60" width="36" height="20" fill="#4c1"/>
<rect width="96" height="20" fill="url(#g)"/>
</g>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" text-rendering="geometricPrecision" font-size="110">
<text x="300" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="500">coverage</text>
<text x="300" y="140" transform="scale(.1)" textLength="500">coverage</text>
<text x="780" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="240">97%</text>
<text x="780" y="140" transform="scale(.1)" textLength="240">97%</text>
</g>
</svg>
-19
assets/socket-score.svg
<svg xmlns="http://www.w3.org/2000/svg" width="100" height="20" role="img" aria-label="socket: score">
<title>socket: score</title>
<linearGradient id="g" x2="0" y2="100%">
<stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
<stop offset="1" stop-opacity=".1"/>
</linearGradient>
<clipPath id="r"><rect width="100" height="20" rx="3" fill="#fff"/></clipPath>
<g clip-path="url(#r)">
<rect width="52" height="20" fill="#555"/>
<rect x="52" width="48" height="20" fill="#5b51d8"/>
<rect width="100" height="20" fill="url(#g)"/>
</g>
<g fill="#fff" text-anchor="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" text-rendering="geometricPrecision" font-size="110">
<text x="270" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="420">socket</text>
<text x="270" y="140" transform="scale(.1)" textLength="420">socket</text>
<text x="750" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="350">score</text>
<text x="750" y="140" transform="scale(.1)" textLength="350">score</text>
</g>
</svg>
-21
LICENSE
MIT License
Copyright (c) 2026 John-David Dalton
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-58
README.md
# decmpfs
![coverage score](assets/coverage-score.svg) [![Socket score](assets/socket-score.svg)](https://socket.dev/cargo/package/decmpfs)
Apply the operating system's **transparent per-file filesystem compression** to a
file — smaller on disk, byte-identical on read, decompressed by the kernel at
near-native speed. macOS APFS (decmpfs/LZVN), Linux btrfs (zstd→lzo→zlib), Windows
NTFS (LZNT1).
The core does the bytes-to-compressed-file write in **one pass** — it never writes a
file then reads it back to recompress.
```rust
use decmpfs::{compress_bytes, compress_file, Gate};
// Write `content` straight to an OS-compressed file (single pass).
let outcome = compress_bytes(path, &content, &Gate::any())?;
// Or compress a file that already exists, in place.
let outcome = compress_file(path)?;
```
## Outcome, never a surprise error
Every call returns an `Outcome` — `Compressed`, `NoGain` (incompressible /
sub-cluster), `AlreadyCompressed`, `Unsupported` (the FS has no per-file
compression — ext4, xfs, ZFS, ReFS, FAT, tmpfs, network mounts), or `Skipped`
(permission, lock, gate). An unsupported filesystem or a permission issue is a
non-fatal `Outcome`, not an `Err`; `Err` is reserved for genuine I/O failures.
## Gate
`Gate` decides which files to compress by glob and/or size:
```rust
let gate = Gate::new(Some("**/*.node"), Some(">= 1MB"))?;
```
## Codecs (speed over ratio — a file is written once, read on load)
- **macOS:** LZVN (the fastest decmpfs codec to decompress).
- **Linux btrfs:** zstd, falling back to lzo then zlib on older kernels.
- **Windows NTFS:** LZNT1 (survives a reinstall's open-for-write, unlike WOF).
## Features
The core is dependency-light (`libc` / `windows-sys` only). The optional `addon`
feature pulls `zstd` + `sha2` to unwrap a napi `--compress` hybrid `.node` back to
the raw addon before compressing.
## Node
An N-API binding (`writeDecmpfsFile` / `writeDecmpfsFileSync`, fs.writeFile-shaped,
atomic by default) lives in [`node/`](node/).
## License
MIT
Cargo.toml.orig

Sorry, the diff of this file is not supported yet