Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1993 +/- ##
==========================================
+ Coverage 62.64% 62.69% +0.04%
==========================================
Files 165 165
Lines 20792 20805 +13
Branches 3621 3623 +2
==========================================
+ Hits 13026 13044 +18
+ Misses 6886 6878 -8
- Partials 880 883 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
a8ff4bc to
974a77f
Compare
|
@JeanChristopheMorinPerso I would like to see this adopted in the rez, as I think for such a small change it does empower anyone to take ownership of the cache copy method without the need of too much refactor of rez's core code. |
|
This simple change would unlock Rez adoption for a use case I'm looking at as well. Is there an opportunity to move this forward? I'm happy to pick it up if that would help - rebasing and adding unit tests. It looks like a great enabler for remote payload backends like S3 or other object stores. Super minimal, simple to use, follows existing patterns of providing overridable methods for plugin extensions. |
maxnbk
left a comment
There was a problem hiding this comment.
It would be a good idea.
I've done some initial review today (sorry, I'm definitely trying to get some reviews on some PRs that have been waiting for them for a long time, but the backlog is deep).
My review findings are:
- Overall, it's a good forward step towards better package-caching utility and pluginification. There are some minor issues I would like to see addressed however, either as a matter of safety/consistency, or as a matter of being explicit about the architecture moving forward.
cache_variantis not abstract. There would be a silent no-op for any non-filesystem repo. The base class profiles a default implementation (thecopytree), which works for filesystem repos wherevariant.rootis a local path, but since we want to push forward remote/artifact repos (which this PR is helping to move the goalpost on),variant.rootmay not be a local filesystem path, or may not exist at all. A remote repo that doesn't overridecache_variantwould silently getcopytreecalled on a non-existent path, producing a confusingFileNotFoundErrorinstead of a clear "not supported" message. I would therefore recommend that we makecache_variantraiseNotImplementedErrorby default (similar to howget_package_payload_pathdoes), forcing each repo plugin to explicitly opt-in. If not this, then document that it assumes a local filesystem path, and have the filesystem repo override it explicitly (moving the implementation, not duplicating it).- The original
variant_rootwas gotten withgetattrwhich returnsNoneif the attribute doesn't exist.variant.rootis an@cached_propertyonVariantResourcethat callsself._root(). If_root()returnsNone,variant.rootwould raiseAttributeErroror returnNone. Basically, the failure mode is different thangetattrwith a default. In practice this probably doesn't matter becauseadd_variantalready validatesvariant_rootbefore reaching the copy, but thecache_variantmethod on the base class doesn't have that protection. It's a public method that could be called independently. My suggestion is thatcache_variantis at the wrong abstraction level. The method is onPackageRepository(the repo), but it receives both thevariantand thelocation(cache destination). The repo knows how to read from itself, but the cache knows where to write. The current signature conflates the source and destination concerns. Cleaner split might bePackageRepository.get_variant_payload(variant) -> Iterator[bytes]or-> path, where the repo provides the payload,PackageCachehandles writing to the cache location. Alternatively, if the goal is for the repo to control the full copy,cache_variant(self, variant, cache_rootpath)which is the current approach, but document that the repo owns the full source->dest transfer. It's not that this approach is bad, it's just pragmatic, but worth documenting clearly. - Not a bug in this PR, but a constraint that
cache_variantimplementations need to be aware of: The originalcopytreeuses defaults, so,symlinks=False,_copy_function=copy2. This is fine for filesystem repos, but the cachesget_variant_sizeexplicitly follows symlinks to compute size. If a custom repo overridescache_variantand usessymlinks=True, the cached size would mismatch the actual cached payload. Additionally, the.copying-*sentinel mechanism and the_while_copyingthread assume the copy takes a non-trivial amount of time. A remote repo that does a streaming download might finish instantly or take much longer, but the timeout/stall detection is calibrated for filesystem copy speeds. (Technically this is a problem today, but I am trying to be proactive with how we make these adjustments...)
Some concrete suggestions/nitpicks:
- A docstring on the new method would be nice.
- If
cache_variantwas overridden by all plugins thenimport shutilis dead import code for those plugins. Not a real problem but the import could be moved to the method level or the default should delegate to a utility function. - The tests still pass because effectively the same code is being exercised. However, tests could verify a custom repo plugin can override
cache_variant, verify that the base class default works without an override, and verify error handling whencache_variantfails.
I would only really consider the "default implementation safety" items as important to address, the docstring as an easy add, and the rest are good followups.
|
Hi @maxnbk |
392247f to
c43ad3d
Compare
|
@maxnbk
Points on design
What's not considered in regards to remote repos in this PR
Please let me know your thoughts, and if you think this is the right direction for rez to take. |
|
The pkg cache related changes here are very similar, but I think the implementation here is neater! 👍 Maybe we could merge this one and I'll form another PR just the build_process reliance on fs? |
|
Apologies for the slow review, I really am trying to catch up on rez reviews and get 3.5.0 out the door. I like the direction, I think things are headed in a good place. I have a couple small notes that are pretty easily addressable. Most of it is documentation-level and simple/straightforward or nitpicks. These are ranked in the order of my perceived importance or ease of dealing-with.
Let me know if any of that is unclear. |
|
@maxnbk No worries, thank you for the feed back, |
|
I had to drop this from the "Next" milestone since I think it still needs work and we desperately wanted to cut a 3.5.0 release. That said, I'm already trying to assemble the next releases and potentially include this in the next hopefully-not-very-far-release, so please do continue on it when you can and I'd like to see where this can take us. |
Signed-off-by: george.ridal <george.ridal@findesign.com.au>
- The api on the repo is now more generic to a copy action rather then a cache action My thinking is that it could be used else where in future, ie rez-cp, a repo nows how to write it payload to a filesystem. - PkgCache now calls variant.resource._cache() as a private method on the resource, this in turn passes itself and the path to copy_variant_payload on its own repo - This has the option of allowing a plugin to override both _cache on the resource and copy_variant_payload on repo Signed-off-by: george.ridal <george.ridal@findesign.com.au>
Signed-off-by: george.ridal <george.ridal@findesign.com.au>
Rebased to main branch Signed-off-by: george.ridal <george.ridal@findesign.com.au>
Signed-off-by: george.ridal <george.ridal@findesign.com.au>
- method is now called from variant class - naming is more generic, focus on copy action instead of cache action - variant.copy_payload could be used else where, rez-cp, rez-mv etc - tests on variant interface to confirm code path to repo.copy_variant_payload Signed-off-by: george.ridal <george.ridal@findesign.com.au>
Signed-off-by: george.ridal <george.ridal@findesign.com.au>
Signed-off-by: george.ridal <george.ridal@findesign.com.au>
suggestion - Variant.copy_payload docstring is a copy-paste from install() (src/rez/packages.py:488-493 @ feb7a50). It reads: "If the package already exists, this variant will be correctly merged into the package. If the variant already exists in this package, the existing variant is returned." That describes installation semantics; it has nothing to do with copying. Misleading docs on a new public API. Fix: replace with one accurate sentence e.g. ("Copy this variant's payload to the filesystem path path via the owning repository."). suggestion - the ownership contract isn't documented. Base docstring (package_repository.py:530-537) says only "Copy a variants payload from repository to a directory location". By your own wording, it should state: the repository owns the full source->destination transfer, and implementations should copy with shutil.copytree defaults (symlinks=False, copy2) so that the cached size stays consistent with get_variant_size following symlinks. suggestion - the bool return is unenforceable. Base docstring: "Should return True on success" - fs impl returns True; but the only caller, package_cache.py:410, ignores the return - a plugin returning False is a silent no-op. Either make the contract raise-on-failure (drop the bool; matches the abstract-by-default direction) or check the return in package_cache. nit - pin the abstractness with a test. a simple memory-repo test (assertRaises(NotImplementedError)) is the missing piece. (Briefly checked that the memory repo raises since it inherits the base-class default.) nit - style: if not variant_resource.repository_type == self.name(): (filesystem.py:1037) reads better as !=. Cosmetic. Signed-off-by: george.ridal <george.ridal@findesign.com.au>
6a98cca to
2e07065
Compare
Signed-off-by: George van Ridal <72674252+vanridal@users.noreply.github.com>
|
Hi @maxnbk Ive addressed your feed back. Hope it can make the next release. |
…e-fs-calls-to-repo-plugin
This PR is an attempt to decouple the package cache operation to the repository plugins
This can allow users to customize how the package cache touches the filesystem on a per repository bases either via the cache_variant() call, or on the variant resource
Does this help with the artifact repo direction, could some remote repos use the current cache system via moving these functions to the repository plugin?
I personally would like a way to customize pkg cache operation via a custom filesystem plugin to optimize cache speeds