Repository navigation
Conversation
Signed-off-by: hiroto.toyoda <hiroto.toyoda@dena.com>
ndeloof
left a comment
There was a problem hiding this comment.
Thanks for the fix, and for the clear reproduction in #14476. I checked it against the code: execStopContainer calls ContainerStop directly, and none of the OpStopContainer nodes carry the service, so the plan never reaches the pre_stop hooks that v5.1.4 ran through stopContainer. Putting the hook in execStopContainer covers every stop of the plan (scale down, restart: true dependents, network and volume recreation, recreate) without touching the seven sites in reconcile.go, which is the right place.
I read the diff only. I did not run the tests, and CI has not run yet (only DCO shows), so none of this is verified by a run.
On your two design questions: running pre_stop on the container a recreate replaces is a legitimate change, even if v5.1.4 did not do it, and I would keep it. A failing hook failing the plan is consistent with stop and down.
Changes I would ask for
-
A container that is gone or stopped by the time of its stop now fails
up.stopContainerindown.goignoresNotFoundandConflictcoming from the hook, which is what #12986 added for exactly this case.runPreStopreturns the error as is. Thectr.State != container.StateRunningcheck works on the summary observed when the plan was built, which can be stale: in a recreate, the old container is only stopped after the new one has been created. If it exits in between,ExecCreatefails with a conflict and the whole plan fails, whereContainerStopon a stopped container used to succeed. I would ignoreNotFoundandConflictfrom the hook asdowndoes, and go on toContainerStop. -
The hook loop now exists twice. The condition (
serviceknown, not a relay container) and the loop overPreStopare instopContainerand inrunPreStop. A shared helper would keep the error policy of point 1 in one place.
To check
- The temporary container of a recreate whose hook fails. The recreate plan is create (temporary name) → stop old → remove old → rename. If the hook fails, the old container stays running, as the description says, but the new one stays too, created and not started, with the labels of the service. I could not tell what the next
updoes with it, for instance whether it counts as a second replica. Since a failing hook makes this easy to reach, could you try a recreate with a failingpre_stop, then a secondup, and look atdocker ps -a? A test for it would help.
Smaller points
- The description says that skipping containers that are not running is "the same condition
stopContaineruses". It is not:stopContainerdoes not look at the state and relies on theConflicthandling of point 1. - The
isRelayContainerbranch has no test, and the recreate case is only covered by the e2e scenario. A unit case on a plan with a recreate would be quick to add. - The "Testing done" checkboxes are all unchecked and the "AI tool" section is empty. Please fill them in.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
What this PR does
Since the reconciliation plan (v5.2.0), the containers that
upandscalestop on their own (scale down, the container a recreate replaces, arestart: truedependent, a recreated network or volume) were stopped without running theirpre_stophooks. Onlystopanddownstill ran them. The plan's stop operation now runs the service'spre_stophooks first, asstopanddowndo.Related issue
Fixes #14476
Changes made
pkg/compose/executor_ops.go:execStopContainercalls a newrunPreStopbeforeContainerStop. It runs the hooks of the project's service on a running container withrunHook, the same helperexecRunPostStartuses.stopContaineruses forstopanddown.pkg/compose/executor_test.go:TestExecutePlanStopContainerRunsPreStopcovers four cases: the hook runs before the stop; a failing hook fails the plan without stopping; no hook for a container that is not running; no hook for a container of no service of the project.pkg/e2e/hooks_test.go,testdata/TestPreStopHookWhenUpStopsAContainer/compose.yaml: a scenario checking that the hook runs for a scaled-down replica, arestart: truedependent, and a recreated container. Each service writes to its own file in a volume.Testing done
make testmake e2e-composeormake e2e-compose-standalonemake lintmake fmtAI tool used (if applicable)
AI Tool:
Additional context
Two design decisions I was unsure about:
pre_stoptoo (it never did before).stopanddown. On a recreate, this leaves the old container running.Also, the output of these hooks is not streamed, because they run before
runStartsets the hook-log listener.(not mandatory) A picture of a cute animal, if possible in relation to what you did