Repository navigation
pgalloc: skip host holes in disk-backed MemoryFiles - #16781
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
Checkpoint skips host holes only for memfd-backed `MemoryFile`s. Disk-backed ones (the `--overlay2` filestores) kept the byte scan, because the sentry writes them through its mapping and FUSE answers `SEEK_DATA` from its server, which may not have seen pages that the kernel hasn't written back yet.
This enables the hole skip for disk-backed files when `fstatfs` says the file is on tmpfs, ext2/3/4, XFS, or btrfs. Their `SEEK_DATA` reports pages that are still dirty in the page cache, and the ext2 and ext3 drivers report the whole file as data. Everything else keeps the byte scan, including FUSE and overlayfs, whose `fstatfs` doesn't identify the upper filesystem. With the usual overlayfs container rootfs and `--overlay2=root:self` nothing changes; this helps `root:dir=` filestores and rootfs directories on those filesystems.
Ranges preallocated with `fallocate` read as holes on ext4, XFS, and btrfs, so checkpoint now leaves them allocated instead of punching them. The saved image does not change. This also drops a stale comment in `createPrivateMemoryFile` claiming that seccomp blocks `fstatfs`.
`BenchmarkSaveToDiskBacked` saves a 256 MiB disk-backed `MemoryFile` with 1% or all of its pages written, backed by a memfd (tmpfs) or by a file in the test's temporary directory (ext4 here), on a c8i.2xlarge with Linux 7.0:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) 6975P-C
│ before │ after │
│ sec/op │ sec/op vs base │
SaveToDiskBacked/file=memfd/written=1pct-8 88950.3µ ± 0% 938.8µ ± 2% -98.94% (p=0.000 n=16)
SaveToDiskBacked/file=memfd/written=100pct-8 2.561m ± 1% 5.694m ± 3% +122.32% (p=0.000 n=16)
SaveToDiskBacked/file=tempdir/written=1pct-8 34994.0µ ± 3% 914.8µ ± 2% -97.39% (p=0.000 n=16)
SaveToDiskBacked/file=tempdir/written=100pct-8 2.522m ± 1% 2.553m ± 2% ~ (p=0.128 n=16)
geomean 11.91m 1.880m -84.21%
```
With every page written, the `SEEK_DATA` and `SEEK_HOLE` calls find no holes and add about 3 ms per 256 MiB on tmpfs, the same cost memfd-backed `MemoryFile`s pay. `SaveTo` only scans pages that are not yet known to be committed, so this mostly affects pages written since the last checkpoint.
Auto-fixed-by: [🤖] @EtiennePerot
FUTURE_COPYBARA_INTEGRATE_REVIEW=#16500 from shayonj:shayon/pgalloc-diskbacked-holes 0552812
PiperOrigin-RevId: 1011162159
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pgalloc: skip host holes in disk-backed MemoryFiles
Checkpoint skips host holes only for memfd-backed
MemoryFiles. Disk-backed ones (the--overlay2filestores) kept the byte scan, because the sentry writes them through its mapping and FUSE answersSEEK_DATAfrom its server, which may not have seen pages that the kernel hasn't written back yet.This enables the hole skip for disk-backed files when
fstatfssays the file is on tmpfs, ext2/3/4, XFS, or btrfs. TheirSEEK_DATAreports pages that are still dirty in the page cache, and the ext2 and ext3 drivers report the whole file as data. Everything else keeps the byte scan, including FUSE and overlayfs, whosefstatfsdoesn't identify the upper filesystem. With the usual overlayfs container rootfs and--overlay2=root:selfnothing changes; this helpsroot:dir=filestores and rootfs directories on those filesystems.Ranges preallocated with
fallocateread as holes on ext4, XFS, and btrfs, so checkpoint now leaves them allocated instead of punching them. The saved image does not change. This also drops a stale comment increatePrivateMemoryFileclaiming that seccomp blocksfstatfs.BenchmarkSaveToDiskBackedsaves a 256 MiB disk-backedMemoryFilewith 1% or all of its pages written, backed by a memfd (tmpfs) or by a file in the test's temporary directory (ext4 here), on a c8i.2xlarge with Linux 7.0:With every page written, the
SEEK_DATAandSEEK_HOLEcalls find no holes and add about 3 ms per 256 MiB on tmpfs, the same cost memfd-backedMemoryFiles pay.SaveToonly scans pages that are not yet known to be committed, so this mostly affects pages written since the last checkpoint.Auto-fixed-by: [🤖] @EtiennePerot
FUTURE_COPYBARA_INTEGRATE_REVIEW=#16500 from shayonj:shayon/pgalloc-diskbacked-holes 0552812