Skip to content

Python: [Bug]: Stateless LocalShellTool timeout hangs on background jobs #9482

Closed

Description

@digit50

Observed Behavior

With LocalShellTool(mode="stateless"), the timeout does not bound how long a call takes when the command leaves a background process that still holds stdout or stderr. The result says timed_out=True, but the call only returns once that background process exits on its own. With timeout=0.5, sleep 6 & returns after 6.0 s, and (sleep 6 &); sleep 30 also returns after 6.0 s. With a long-lived job, for example python -m http.server &, the tool call and the agent run never return.

Expected Behavior

The call returns shortly after the timeout, with timed_out=True, and every process the command started, including background and double-forked jobs, is killed. The ShellExecutor.run docstring says the timeout bounds execution without leaking processes.

Steps to Reproduce

  1. Install agent-framework-core==1.22.0 and agent-framework-tools on Linux.
  2. Save the script below as repro.py and run python repro.py < /dev/null.
  3. Each case reports timed_out=True but returns after about 6 s instead of about 0.5 s.

Minimal Reproduction

import asyncio
import os
import signal
import tempfile
import time

from agent_framework.tools import LocalShellTool

TIMEOUT = 0.5


async def main() -> None:
    pidfile = os.path.join(tempfile.mkdtemp(), "bg.pid")
    tool = LocalShellTool(mode="stateless", timeout=TIMEOUT)
    cases = [
        ("background job", f"sleep 6 & echo $! > {pidfile}"),
        ("double-forked job", f"(sleep 6 & echo $! > {pidfile}); sleep 30"),
    ]
    for name, command in cases:
        started = time.monotonic()
        result = await tool.run(command)
        elapsed = time.monotonic() - started
        pid = int(open(pidfile).read().strip())
        try:
            os.kill(pid, 0)
            alive = True
        except ProcessLookupError:
            alive = False
        print(f"{name}: {command.replace(pidfile, '<pidfile>')!r} with timeout={TIMEOUT}")
        print(f"  observed: returned after {elapsed:.1f} s, timed_out={result.timed_out}, "
              f"exit_code={result.exit_code}, background sleep still alive after return: {alive}")
        print(f"  expected: returned after about {TIMEOUT} s, background sleep killed")
        if alive:
            os.kill(pid, signal.SIGKILL)


asyncio.run(main())

Output:

background job: 'sleep 6 & echo $! > <pidfile>' with timeout=0.5
  observed: returned after 6.0 s, timed_out=True, exit_code=0, background sleep still alive after return: False
  expected: returned after about 0.5 s, background sleep killed
double-forked job: '(sleep 6 & echo $! > <pidfile>); sleep 30' with timeout=0.5
  observed: returned after 6.0 s, timed_out=True, exit_code=-15, background sleep still alive after return: False
  expected: returned after about 0.5 s, background sleep killed

The background sleep is no longer alive when the call returns only because the call waited for it to finish by itself.

Error Messages and Stack Traces

No error is raised. The call blocks until the background process exits.

Package Versions

agent-framework-core: 1.22.0 (python-1.22.0), agent-framework-tools: 1.0.0b261015, psutil: 7.2.2

Python Version

Python 3.12.3

Operating System

Linux

Regression

Unknown

Additional Context

Cause, at main 44fe223:

  • run_stateless starts the shell in its own session (start_new_session=True, python/packages/tools/agent_framework_tools/shell/_executor.py line 31). On timeout it calls kill_process_tree(proc) and then drains with await proc.communicate() (line 78), which has no timeout and waits until every holder of the pipes has exited.
  • kill_process_tree (python/packages/tools/agent_framework_tools/shell/_killtree.py lines 56-79) returns immediately when proc.returncode is not None (line 56), so in the cmd & case, where the shell has already exited, nothing is signalled at all. Otherwise it kills only what psutil finds through parent.children(recursive=True) (line 76). A double-forked job is reparented away from the shell, so it is not a descendant either. Both jobs are still in the shell's process group and session.
  • The stdlib fallback _kill_via_stdlib (lines 99-130) signals the process group and would catch the double-forked job, but it is not used because psutil is a required dependency, and it would not help the cmd & case because of the early return at line 56.
  • test_shell_killtree.py mocks children() and has no case with a reparented process.

Proposed fix: on POSIX, signal the whole process group with os.killpg(proc.pid, ...) even when the shell has already exited (the group id equals the pid under start_new_session, and os.getpgid(proc.pid) fails once the shell has been reaped), keeping the psutil descendant walk as an extra step. Then bound the drain at _executor.py line 78 with asyncio.wait_for, and add a test that runs sleep N & with a short timeout and checks the elapsed time.

I can open a PR for this.

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I personally verified this behavior and the reproduction details are authentic.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 18, 2026
  2. added
    reproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflow
    on Oct 18, 2026
  3. github-actions commented on Oct 18, 2026

    @github-actions
    Contributor

    🤖 Automated triage reproduction notes (agent-authored — trust but verify)

    Agent analysis

    The bug occurs in python/packages/tools/agent_framework_tools/shell/_executor.py::run_stateless, with cleanup delegated to _killtree.py::kill_process_tree, when a POSIX background process inherits the captured pipes and the shell exits before timeout cleanup. Minimal repro: configure stateless LocalShellTool with timeout=0.1, run sleep 2 &, and assert the call returns within one second; it currently returns after approximately two seconds with timed_out=True.

    • Failing test: python/packages/tools/tests/test_local_shell_tool.py::test_stateless_timeout_bounds_background_job
    • Files examined: python/packages/tools/agent_framework_tools/shell/_executor.py, python/packages/tools/agent_framework_tools/shell/_killtree.py, python/packages/tools/agent_framework_tools/shell/_tool.py, python/packages/tools/tests/test_local_shell_tool.py, python/packages/tools/tests/test_shell_killtree.py, python/packages/tools/pyproject.toml
    • Tests run: test_local_shell_tool.py -k timeout (6 passed), test_stateless_timeout_bounds_background_job (failed)
    • Reported version: agent-framework-tools 1.0.0b261015
    • Current version: agent-framework-tools 1.0.0b261015
  4. added
    skillsUsage: [Issues, PRs], Target: skills related features
    and removed
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 18, 2026
  5. eavanvalkenburg commented on Nov 2, 2026

    @eavanvalkenburg
    Member

    /plan Reconcile the remaining #9482 cleanup defect with existing PR #9483 on current main, without starting competing implementation. The historical indefinite inherited-pipe wait is not the current baseline: bounded capture/drain and transport closure now exist separately. The remaining source-backed gap is kill_process_tree returning after shell exit and psutil losing reparented jobs still in the owned POSIX process group. Prior maintainer approval on #9483 predates the separate capture changes; preserve that review context rather than treating it as absent agreement. Verify which current PR changes remain necessary, owned-group assumptions and PID/group lifetime, TERM/grace/KILL behavior, zombie-only groups, escaped-session limitations, stdlib/psutil parity and Windows behavior. Require deterministic real-process checks with explicit owned-PID cleanup, bounded elapsed-time assertions and no leaked pipe handles. Do not promise killing arbitrary detached descendants or repeat the old unbounded-hang claim without current reproduction. Recommend the smallest remaining scope and any platform/compatibility decisions; this is planning, not a new implementation assignment or a PR closure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: PythonreproducedUsage: [Issues], Target: all issues that can be reproduced by the triage workflowskillsUsage: [Issues, PRs], Target: skills related features

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions