Skip to content

pgalloc: skip host holes in disk-backed MemoryFiles - #16781

Open
copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl1011162159
Open

copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl1011162159

Conversation

@copybara-service

Copy link
Copy Markdown

pgalloc: skip host holes in disk-backed MemoryFiles

Checkpoint skips host holes only for memfd-backed MemoryFiles. Disk-backed ones (the --overlay2 filestores) kept the byte scan, because the sentry writes them through its mapping and FUSE answers SEEK_DATA from its server, which may not have seen pages that the kernel hasn't written back yet.

This enables the hole skip for disk-backed files when fstatfs says the file is on tmpfs, ext2/3/4, XFS, or btrfs. Their SEEK_DATA reports pages that are still dirty in the page cache, and the ext2 and ext3 drivers report the whole file as data. Everything else keeps the byte scan, including FUSE and overlayfs, whose fstatfs doesn't identify the upper filesystem. With the usual overlayfs container rootfs and --overlay2=root:self nothing changes; this helps root:dir= filestores and rootfs directories on those filesystems.

Ranges preallocated with fallocate read as holes on ext4, XFS, and btrfs, so checkpoint now leaves them allocated instead of punching them. The saved image does not change. This also drops a stale comment in createPrivateMemoryFile claiming that seccomp blocks fstatfs.

BenchmarkSaveToDiskBacked saves a 256 MiB disk-backed MemoryFile with 1% or all of its pages written, backed by a memfd (tmpfs) or by a file in the test's temporary directory (ext4 here), on a c8i.2xlarge with Linux 7.0:

goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) 6975P-C
                                               │    before     │                after                 │
                                               │    sec/op     │   sec/op     vs base                 │
SaveToDiskBacked/file=memfd/written=1pct-8       88950.3µ ± 0%   938.8µ ± 2%   -98.94% (p=0.000 n=16)
SaveToDiskBacked/file=memfd/written=100pct-8       2.561m ± 1%   5.694m ± 3%  +122.32% (p=0.000 n=16)
SaveToDiskBacked/file=tempdir/written=1pct-8     34994.0µ ± 3%   914.8µ ± 2%   -97.39% (p=0.000 n=16)
SaveToDiskBacked/file=tempdir/written=100pct-8     2.522m ± 1%   2.553m ± 2%         ~ (p=0.128 n=16)
geomean                                            11.91m        1.880m        -84.21%

With every page written, the SEEK_DATA and SEEK_HOLE calls find no holes and add about 3 ms per 256 MiB on tmpfs, the same cost memfd-backed MemoryFiles pay. SaveTo only scans pages that are not yet known to be committed, so this mostly affects pages written since the last checkpoint.
Auto-fixed-by: [🤖] @EtiennePerot
FUTURE_COPYBARA_INTEGRATE_REVIEW=#16500 from shayonj:shayon/pgalloc-diskbacked-holes 0552812

Checkpoint skips host holes only for memfd-backed `MemoryFile`s. Disk-backed ones (the `--overlay2` filestores) kept the byte scan, because the sentry writes them through its mapping and FUSE answers `SEEK_DATA` from its server, which may not have seen pages that the kernel hasn't written back yet.

This enables the hole skip for disk-backed files when `fstatfs` says the file is on tmpfs, ext2/3/4, XFS, or btrfs. Their `SEEK_DATA` reports pages that are still dirty in the page cache, and the ext2 and ext3 drivers report the whole file as data. Everything else keeps the byte scan, including FUSE and overlayfs, whose `fstatfs` doesn't identify the upper filesystem. With the usual overlayfs container rootfs and `--overlay2=root:self` nothing changes; this helps `root:dir=` filestores and rootfs directories on those filesystems.

Ranges preallocated with `fallocate` read as holes on ext4, XFS, and btrfs, so checkpoint now leaves them allocated instead of punching them. The saved image does not change. This also drops a stale comment in `createPrivateMemoryFile` claiming that seccomp blocks `fstatfs`.

`BenchmarkSaveToDiskBacked` saves a 256 MiB disk-backed `MemoryFile` with 1% or all of its pages written, backed by a memfd (tmpfs) or by a file in the test's temporary directory (ext4 here), on a c8i.2xlarge with Linux 7.0:

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) 6975P-C
                                               │    before     │                after                 │
                                               │    sec/op     │   sec/op     vs base                 │
SaveToDiskBacked/file=memfd/written=1pct-8       88950.3µ ± 0%   938.8µ ± 2%   -98.94% (p=0.000 n=16)
SaveToDiskBacked/file=memfd/written=100pct-8       2.561m ± 1%   5.694m ± 3%  +122.32% (p=0.000 n=16)
SaveToDiskBacked/file=tempdir/written=1pct-8     34994.0µ ± 3%   914.8µ ± 2%   -97.39% (p=0.000 n=16)
SaveToDiskBacked/file=tempdir/written=100pct-8     2.522m ± 1%   2.553m ± 2%         ~ (p=0.128 n=16)
geomean                                            11.91m        1.880m        -84.21%
```

With every page written, the `SEEK_DATA` and `SEEK_HOLE` calls find no holes and add about 3 ms per 256 MiB on tmpfs, the same cost memfd-backed `MemoryFile`s pay. `SaveTo` only scans pages that are not yet known to be committed, so this mostly affects pages written since the last checkpoint.
Auto-fixed-by: [🤖] @EtiennePerot
FUTURE_COPYBARA_INTEGRATE_REVIEW=#16500 from shayonj:shayon/pgalloc-diskbacked-holes 0552812
PiperOrigin-RevId: 1011162159
@copybara-service copybara-service Bot added the exported Issue was exported automatically label Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exported Issue was exported automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant