[JanZachmann]

Summary

• write_partition compares the partition file with the image in 4 KiB blocks, only where one of them holds data, and writes only the blocks that differ, instead of running fallocate -d. It also writes a block that is data in the partition file but a hole in the image, e.g. a zero block of a file that e2cp or mcopy wrote into free space. fallocate, dd and sync are removed from the Docker image.
• The image is copied with sparse::copy_file, and a partition is read out of the image with the same sparse copy instead of dd conv=sparse: it finds the data with SEEK_DATA/SEEK_HOLE, keeps the holes and copies each data range in a loop. Decompressing an image keeps zero blocks as holes, reads a file made of several streams (e.g. from pbzip2) and ignores zero padding after the last one, and fails on a truncated or corrupt stream. A compressed image is decompressed straight into the temp dir, without a copy first. A symlinked image, e.g. the Yocto deploy link, is detected by its target, so a compressed one is no longer read as uncompressed. Images named .gz and .bz2 are decompressed too; before, the output file had the name of the source, so creating it emptied the source. Compressing fails when the end of the stream cannot be written.
• omnect-cli writes the bmap of -b itself: it maps the ranges of the input bmap, or the whole image without one, plus every block the image file stores as data, with a sha256 per range. No zeros are written into the image, so it stays sparse, and bmaptool is no longer needed at runtime: -b now works in the Docker image, and bmap-tools is removed from the Debian dependencies and the Docker image. The bmap is copied next to the image after the image was written back.
• The input bmap is the file given with the new --input-bmap argument (requires -b), else <image>.bmap or the bmap of the uncompressed image next to it (e.g. x.wic.bmap for x.wic.xz, as the Yocto build creates it). It is only read with -b. The command fails if the input bmap is for an image of another size, if a range does not match its sha256 checksum, if the bmap has no range checksums, if the file is larger than 64 MiB or not UTF-8, or if bmaptool would find the input image itself as its bmap (an image named e.g. x.bmap). The new --no-input-bmap (requires -b) skips it, and the bmap then maps the whole image. The input bmap of an uncompressed image is checked before the image is copied.
• The created bmap is named <output image>.bmap and written next to the output image, e.g. x.wic.bmap for x.wic and x.wic.xz.bmap for x.wic.xz, where bmaptool finds it. So with -p it no longer replaces the bmap of the input image, which stays unchanged. An image changed in place keeps the name of its bmap: the new bmap replaces the one bmaptool finds next to it, e.g. x.wic.bmap for -i x.wic.xz -p xz, so no bmap of the old image stays.
• The output image is the input image, for a compressed image without the compression extension, plus the -p extension, as before. The new -o/--output-image writes it to another path, and the input image stays unchanged. With -p, the -o name must end in an extension of the compression that bmaptool reads (.xz, .bz2, .gzip or .gz); without -p it must not have a compression extension. An existing output image must be a regular file, so -o /dev/sdX or a directory is refused before the command runs. A command now fails instead of overwriting an existing file that is not the input image (a path given with -o is overwritten), writing a bmap that bmaptool would also use for an image that stays, i.e. <name> or <name>.<compression> next to it without another bmap that comes first, also named after the target of a symlinked bmap (e.g. x.wic.bmap for -i x.wic.xz without -p, also if no bmap exists yet and also with --input-bmap or --no-input-bmap, or for an in-place -i x.wic.xz -p xz while x.wic is next to it), or writing a bmap over the input or the output image (e.g. an image named x.bmap), or writing the output image over the input bmap, or to a path that bmaptool reads as the bmap of another image (e.g. -i x.wic -o x.wic.bmap), or writing an image without -b while <output image>.bmap is next to it, or for a compressed output image the bmap of the uncompressed image, which bmaptool also uses. That bmap would no longer match. So for several commands on one image, use -b on each.
• The output image and the bmap are written to a new file in the same directory and renamed over the old one, so a failed write, e.g. on a full disk, no longer destroys an image that is changed in place. A symlink is followed, and the old file keeps its permissions. A symlink whose target does not exist yet is followed too. The directory needs write access and space for one more image, which is checked before the command runs; an existing output image in a directory the user cannot write is staged in TMPDIR and copied into it, so an in-place run works as before, and the new file belongs to the user who runs omnect-cli. The image is replaced before its bmap: if the image step fails, the old image keeps its bmap, and if the bmap step fails, the old bmap is removed, and the error names it if it stays. An output image that is a mount point, e.g. a file mounted into a container, cannot be replaced by a rename, so its new content is staged in TMPDIR and copied into it; a rename that fails with EBUSY, EXDEV or EPERM (a sticky dir like /tmp and a file of another user) also falls back to this copy; if that copy fails, the staged file is kept with its new bmap, the old bmap is removed, and the error names both. An old bmap that cannot be removed is named in the error too. With -b such a mounted image fails, because the bmap would be written next to the mount point and not next to the image. All temp dirs are created with mode 0700, because they hold images and keys. Ctrl-C, SIGTERM or SIGHUP removes every temp dir of a command before the process stops; a signal that is ignored, e.g. SIGHUP under nohup, is not handled. The handler starts with each image command, because as PID 1, e.g. in the Docker image, the process ignores a stop signal without a handler. As PID 1 the process then exits with 128 plus the signal number, because it ignores a signal it raises itself. An image command refuses a TMPDIR that contains : before it starts, because e2tools split their file arguments there. A container started with --rm removes a file kept in its TMPDIR, so the README asks for a TMPDIR mounted from the host for a mounted image.
• copy-from-image no longer writes the image back. It only reads, so an error can no longer truncate the source, and for x.wic.xz it no longer leaves x.wic next to it. It reads an uncompressed image directly, without a temp copy, so it refuses an output file of -f that is the input image, also through a hard link, a symlink or an output dir, by device and inode.
• The partition files are written to their own temp dir. Before, they were named <num>.img next to the image, so an image named e.g. 1.img was overwritten by its partition 1. The temp dirs follow TMPDIR.
• set-device-certificate and set-edge-ca-certificate write the new certificate and key to a temp dir instead of under fixed names next to the input image, so two runs on one base image no longer share them, a read-only image directory works, and no private key is left there.
• Partitions of a GPT with 4096-byte sectors are read at the right offset; before, the offsets were always counted in 512-byte sectors. A partition that does not fit into the image, e.g. from a corrupt GPT, is refused, so its offset cannot overflow.
• Known limitations, tracked as follow-up tasks: the rename replaces only the name it writes, so other hard links of the output image keep the old image. On a kernel before 5.8, which does not report mount points in statx, a mounted image file with -b is not detected, and the bmap is written next to the mount point. The created bmap maps every block that the image file stores as data, so for an image that is not sparse it maps the whole image, also with an input bmap. For a partition that does not start on a 4 KiB boundary, a block at the edge of each data run is written back, so the image is less sparse. In both cases the image content is correct. An uncompressed output image is copied twice, into TMPDIR and into the staged file. A stop signal between the image and the bmap step keeps the old bmap next to the new image. The output name checks do not know the archive extensions of bmaptool (.tar.*, .zip), nor .zst, .lz4 and .lzo without -p. For a symlinked image, the bmap is looked up only next to the link name, so the bmap next to the target is not found and does not match after an in-place change. A bmap with fewer extensions than an image, e.g. x.bmap for x.wic.xz, is not checked as the bmap of an image that stays, because only <name> and <name>.<compression> count as images of <name>.bmap.
• copy-from-image accepts an output file without a directory, e.g. out.txt, and checks all output dirs before it reads the image.
• actix-web and actix-http are updated for RUSTSEC-2026-0446, RUSTSEC-2026-0447 and RUSTSEC-2026-0448.
• Breaking: --generate-bmap-file is renamed to --output-bmap, so the two options read as a pair. The short option -b stays. With -p the bmap is <output image>.bmap, e.g. x.wic.xz.bmap where releases before wrote x.wic.bmap, unless the image is changed in place and a bmap exists next to it. bmaptool finds the new name for x.wic.xz by itself, but a script that passes --bmap x.wic.bmap must use the new name. The name conflicts above fail where they overwrote files before, or wrote a bmap that bmaptool then used for the wrong image: -i x.wic.xz -b without -o or -p now fails, because the new x.wic.bmap would not match x.wic.xz. -p bzip2 writes <image>.bz2 instead of <image>.bzip2, because bmaptool does not read .bzip2. An input named .bzip2 is still read, but with -p the result is written next to it as .bz2, so a second run on the same input fails because that file exists. -o needs .bz2.
• roxmltree is added to read the bmap, rustix for SEEK_DATA/SEEK_HOLE and statx, signal-hook to remove temp dirs on a stop signal (its signal-hook-registry was already in the tree through tokio), and libc for sigaction (already in the tree); libfs is removed. The test bmap testfiles/image.wic.bmap is now in git (*.bmap is in .gitignore), with checksums that match the test images. The version is bumped to 0.29.0, and ctutils to 0.4.5 for RUSTSEC-2026-0385.

Reason

A sparse image, e.g. one that xz decompressed into a file, or one after fallocate -d, has holes wherever the image is zero, and a bmap skips them. Some of these zeros must still be written: a bmap-based flash otherwise keeps what an earlier installation left there. An old ext4 journal can be replayed after a power cut and apply an old transaction to the new filesystem; old GPT entries break the entry array CRC; old FAT sectors and directory entries show up as used clusters or stale files. On an rpi4-64 image, mapping only the ext4 journals still drops 188 blocks of the Yocto bmap: GPT primary and backup entries, 69 blocks of boot, and 16–24 blocks on every ext4 partition.

The Yocto bmap maps all of these ranges, because the build image holds real data there. Keeping its ranges mapped covers every format without knowing about file systems. From a sparse image alone, omnect-cli cannot tell such zero blocks from free space, so without an input bmap it maps the whole image. bmaptool copy reads a mapped range with plain reads, so a hole in such a range is written as zeros: the zeros only have to be in the bmap, not allocated in the image file. This replaces the image-side approach discussed in omnect/meta-omnect#701.

With default-features = false, libfs made byte copies, so every image came back fully allocated, and the decompress wrote zero blocks as data. fallocate -d hid both. The sparse copy uses SEEK_DATA/SEEK_HOLE directly: libfs::next_sparse_segments seeks both files to the same offset, so it cannot read a partition out of the image, and libfs::copy_file copies each data segment with a single copy_file_range call, which moves at most 2 GiB, and ignores a short copy.

All omnect partitions have a fixed size, so builds of one machine have the same image size, and the bmap of another build (e.g. of an older download next to a new image) passes a size check. The sha256 of each range tells the builds apart; bmaptool copy runs the same check on flash.

omnect-cli derives every output name from the input name and finds the input bmap by name, so names can collide. omnect-cli runs in CI scripts, where nobody reads a warning, so each collision fails with an error that names the option to resolve it. A counter in the output name would not work: a script and the later bmaptool copy need a fixed name.

Writing only changed blocks also fixes a gap of the old dd conv=notrunc,sparse: a block that changed to all zeros was skipped and kept its old content; now it is written because it differs. In the same way, dd conv=sparse turned every zero block of a partition into a hole, so a zero block of a copied file that landed in free space stayed a hole in the image, and a bmap flash kept the old content there.

[mlilien]

Thanks, resolved all but one thread (see my reply there). A few more points inline, nothing blocking.

[mlilien]

Thanks, all threads are resolved. One regression from db58abe inline (first comment); the rest are smaller hint and doc issues.

[mlilien]

Thanks, all threads are resolved. One data-loss case for mount-point outputs inline (first comment); the rest are smaller hint and ordering issues.

[mlilien]

Re-review of 691af1a: the open threads are fixed or tracked, so I resolved them. One new point, inline. I'd fix that one before the merge; the rest is fine as follow-ups.