Skip to content

[Bug]: Daytona runs get stuck in provisioning because SSH token expiry exceeds the API limit #4494

Open

Description

@peterschmidt85

Steps to reproduce

  1. Configure the daytona backend with a valid API key and available H200 capacity.

  2. Create a fleet with dstack apply -f fleet.dstack.yml -y:

    type: fleet
    name: daytona
    nodes: 0..1
    backends: [daytona]
  3. Submit this task with dstack apply -f task.dstack.yml -y -d:

    type: task
    name: daytona-ssh
    fleets: [daytona]
    image: pytorch/pytorch:2.11.0-cuda12.8-cudnn9-runtime
    commands:
      - nvidia-smi
    resources:
      cpu: 4
      memory: 16GB
      disk: 100GB
      gpu: H200:1
    spot_policy: on-demand
    retry: false

Actual behaviour

Daytona creates and starts the sandbox, but the run stays in provisioning. Each attempt to create SSH access fails with HTTP 400. The task's commands never run.

Expected behaviour

dstack establishes SSH access and the run reaches running.

dstack version

Reproduced from source at c9528229c, configured to use runner 0.22.8. The affected Daytona backend code is identical on master at d616872d6.

Server logs

Got exception when updating instance pr286-e2e-0 provisioning data
DaytonaAPIError: Daytona API returned HTTP 400: expiresInMinutes must be between 1 and 1440

Additional information

Root cause: dstack sets SSH_ACCESS_MINUTES to 3650 * 24 * 60 and passes it when creating SSH access. This requests 5,256,000 minutes (ten years). Daytona's current OpenAPI documents a maximum of 1,440 minutes (24 hours) for POST /api/sandbox/{sandboxIdOrName}/ssh-access.

Possible fix: Restore support for long SSH-token expiries in Daytona, or handle credential expiry and renewal in dstack. Reducing the request to 24 hours would unblock startup, but would leave reconnection after expiry unresolved.

Reproduced through the Python client and an isolated dstack server on a real H200 while reviewing dstackai/gpuhunt#286. That PR changes GPU discovery, not SSH setup; this issue affects the existing dstack integration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions