"No such file or directory" error on sandbox create from another container inside cluster

[sjoerdvanBommel]

Agent Diagnostic

After debugging with the debug-openshell-cluster skill:

1. Sandboxes ARE created successfully: Running kubectl get pods -n openshell shows all sandboxes in Running state
2. The error happens AFTER creation: Server logs show CreateSandbox request completed successfully
3. CLI waits for something that fails: After the sandbox pod starts, the CLI attempts some operation that fails with ENOENT
4. openshell sandbox exec works: Commands can be executed in created sandboxes using openshell sandbox exec --name xyz pwd

Description

Summary

openshell sandbox create fails with "No such file or directory (os error 2)" even when no command is specified or when using simple commands like ls. The sandbox is successfully created and reaches "Ready" phase, but the command execution step consistently fails.

Expected Behavior

Sandbox should be created successfully and the command should drop into an interactive shell

Actual Behavior

Created sandbox: test-sandbox

  [0.0s] Requesting compute...
[32.0s] Sandbox allocated
[32.4s] Image pulled
[32.4s] Pulling image ghcr.io/nvidia/openshell-community/sandboxes/base:latest
[32.9s] Image pulled (1.0 GB)
[34.0s] Image pulled
[34.0s] Pulling image ghcr.io/nvidia/openshell-community/sandboxes/base:latest
[55.4s] Image pulled (1.0 GB)
Error:   × No such file or directory (os error 2)

Exit code: 1

Additional Test Cases

All of the following commands produce the same error:

Test 1: With explicit command

openshell sandbox create --name test-ls -- ls -la

Result: Same "No such file or directory" error

Test 2: With shell command

openshell sandbox create --name test-sh -- /bin/sh -c 'echo "hello"'

Result: Same "No such file or directory" error

Test 3: With simple echo

openshell sandbox create --name test-echo -- echo Created

Result: Same "No such file or directory" error

Hypothesis

The error appears to be in the CLI client's execution path after the sandbox is created. Possible causes:

1. Missing executable: The CLI might be trying to execute an SSH binary, rsync, or another dependency that doesn't exist in the client container
2. Protocol issue: The HTTP2/gRPC connection issue in the logs suggests a communication problem during command execution
3. Path resolution: The "os error 2" (ENOENT) suggests the CLI is looking for a file/binary that doesn't exist

Environment-Specific Behavior

The issue occurs when running openshell CLI from:

• Node.js execFile() in a container
• Shell (sh -c) in a container
• But works when run directly from kubectl exec interactive session

This suggests the CLI is trying to access a file/resource that exists in interactive sessions but not in non-interactive/programmatic contexts.

Workaround

Option 1: Ignore the error (Sandbox is actually created)

try {
  await execFileAsync('openshell', ['sandbox', 'create', '--name', name]);
} catch (error) {
  // Sandbox is created despite the error
  // Verify with: openshell sandbox list
  console.log('Sandbox likely created despite error');
}

Option 2: Create and verify separately

# Create (will fail but sandbox is created)
openshell sandbox create --name xyz 2>&1 | grep "Created sandbox" && echo "Success"

# Verify it exists
openshell sandbox list | grep xyz

# Use it
openshell sandbox exec --name xyz -- <command>

Option 3: Use the API directly
Skip the CLI and call the OpenShell gRPC API directly to avoid the CLI's post-creation logic.

However, all workarounds are suboptimal for automation and programmatic usage.

Reproduction Steps

1. Install OpenShell CLI in a container in your cluster:

curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh

2. Register the gateway in that container:

openshell gateway add http://openshell.openshell.svc.cluster.local:8080 --name openshell-local
openshell gateway select openshell-local

3. Attempt to create a sandbox from that container:

openshell sandbox create --name test-sandbox

Environment

• Platform: Kubernetes (KinD cluster)
• Gateway Type: Remote (plain HTTP)
• Base Image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest
• Client Environment: Docker container running Debian Bookworm Slim with Node.js 24
• OS: Apple M4 Pro (26.5 (25F71))
• Reproduced on: 2026-06-10
• Related Components: CLI, Kubernetes driver, Remote gateway

Logs

## Server Logs (Relevant Sections)


[2026-06-10T09:00:06.550487Z]  INFO openshell_server::grpc::sandbox: minted sandbox JWT sandbox_id=30c6071d-5cf4-4800-b547-b263479c4a5a
[2026-06-10T09:00:06.558861Z]  INFO openshell_driver_kubernetes::driver: Sandbox created in Kubernetes successfully sandbox_id=30c6071d-5cf4-4800-b547-b263479c4a5a sandbox_name=debug-test
[2026-06-10T09:00:06.558904Z]  INFO openshell_server::grpc::sandbox: CreateSandbox request completed successfully sandbox_id=30c6071d-5cf4-4800-b547-b263479c4a5a sandbox_name=debug-test


Later:

[2026-06-10T09:00:34.011422Z]  WARN openshell_server::supervisor_session: supervisor session: stream error sandbox_id=09d5f87c-ebdc-4751-ad99-e3733930d02b session_id=1acf9eca-2180-41ef-937a-e2c0d464336f error=status: Unknown, message: "h2 protocol error: error reading a body from connection"

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why

[sjoerdvanBommel]

After further investigation of the OpenShell CLI source code (using Claude), I think I've identified the cause of this issue.

What's Happening

1. Sandbox creation succeeds - The gRPC CreateSandbox request completes successfully and the sandbox enters Ready state
2. CLI tries to auto-connect - After creation, the CLI automatically attempts to SSH into the sandbox (code path: run.rs:2195-2205)
3. SSH ProxyCommand fails - The SSH connection uses a ProxyCommand that calls openshell ssh-proxy, which fails with "No such file or directory (os error 2)"

Technical Details

The issue occurs in the OpenShell CLI's SSH connection logic:

File: crates/openshell-cli/src/ssh.rs

When openshell sandbox create finishes provisioning:

• If no command is provided (line run.rs:2195), it calls sandbox_connect()
• This builds an SSH ProxyCommand: /usr/bin/openshell ssh-proxy --gateway ... --sandbox-id ... --token ... (forward.rs:514-528)
• SSH attempts to execute this ProxyCommand
• The ProxyCommand execution fails with ENOENT (file not found)

Why It Fails in Containers

The ProxyCommand string is executed by SSH in a subshell. When running from:

• Interactive terminal: Works fine, SSH can execute the openshell binary
• Container via Node.js execFileAsync: Fails because:
  • The process has no controlling TTY
  • SSH's ProxyCommand execution context can't find or execute the binary
  • Likely a PATH, permissions, or shell context issue

Key code locations:

• run.rs:2195-2205 - Auto-connect logic after sandbox creation
• ssh.rs:66-131 - ssh_session_config() builds the ProxyCommand
• ssh.rs:244-269 - sandbox_connect_with_mode() executes SSH
• forward.rs:514-529 - build_proxy_command() creates the command string

Attempted Workarounds

I tried several approaches:
2. ❌ Passing a dummy command (-- true)
3. ⚠️ Catching the error - Works but ugly, error still occurs

Proposed Solutions

Option 1: Add --no-connect / --detach flag (Recommended)

Add a CLI flag to skip the post-creation SSH connection:

// In run.rs around line 2195
if command.is_empty() && !no_connect {
    let connect_result = if persist {
        sandbox_connect(&effective_server, &sandbox_name, &effective_tls).await
    } else {
        // ...
    }
}

Option 2: Fix ProxyCommand execution

Investigate why SSH's ProxyCommand fails in container environments. This might be a shell escaping, PATH, or permissions issue.

Option 3: Use gRPC API directly

For programmatic usage, bypass the CLI and call the gRPC CreateSandbox RPC directly. This avoids the auto-connect behavior entirely.

[TaylorMutch]

📋 triage-agent

Triage Assessment

Classification: bug-confirmed

Summary

The report describes a real post-provisioning CLI failure. CreateSandbox can succeed and leave the sandbox Ready, then sandbox create fails locally while trying to start the post-create SSH session. The same mechanism is used for sandbox create -- <cmd> when the sandbox is kept, while openshell sandbox exec uses the gRPC exec path.

Investigation

• The agent diagnostic is substantive and there was no prior triage comment.
• crates/openshell-cli/src/main.rs defaults create to keep the sandbox alive unless --no-keep is passed, and there is no --detach or --no-connect flag.
• crates/openshell-cli/src/run.rs calls sandbox_connect for empty create commands and sandbox_exec for trailing create commands after provisioning.
• Those functions in crates/openshell-cli/src/ssh.rs build an external OpenSSH command with Command::new("ssh"). In a slim client container or non-interactive runtime where ssh is absent from PATH, Rust returns No such file or directory (os error 2).
• The ProxyCommand uses std::env::current_exe() and build_proxy_command, so the most direct ENOENT is likely the local ssh executable rather than a failed sandbox or Kubernetes provisioning problem.
• openshell sandbox exec is separate: main.rs routes it to sandbox_exec_grpc, which matches the reporter observation that exec works after the sandbox exists.

Recommendation

Keep this as a CLI compatibility bug. The fix should either preflight and report missing OpenSSH before creating a sandbox, provide a create-only --detach or --no-connect mode, or move post-create command execution onto the gRPC exec path for automation. Add coverage for sandbox create in a client environment without ssh, and verify that create -- <cmd> does not require the external SSH client when no interactive session is needed.

Applied labels: area:cli, area:sandbox, area:cluster, topic:compatibility, os:linux.

[sjoerdvanBommel]

Thanks for the triage assessment. Somehow I didn't think of the fact that the container might not have OpenSSH client installed. Installing it solved at least my issue, but these improvements still sound valuable 👍