Cloud Dispatch
workflow ~238 tokens updated 2026-08-04
Delegate Claude Code agent work to Hetzner Cloud VMs. Provides golden image pipeline, VM lifecycle management, secret injection, workspace provisioning, and /dispatch command for one-command multi-agent dispatch.
Tags
README
cloud-dispatch
Delegate Claude Code agent work to Hetzner Cloud VMs. Boot ephemeral agents from a pre-baked golden image in 30-90 seconds, dispatch GitHub issues across them in parallel, and collect PRs when done.
Up to 12 agents (3 VMs x 4 agents each) can run simultaneously, each working an independent GitHub issue from its own isolated user account.
Overview
The dispatch pipeline runs entirely from your local machine via SSH:
- Create cx22 VMs from a pre-baked Hetzner snapshot (30-90 seconds boot)
- Inject credentials at session time via tmpfs (never in git or cloud-init)
- Provision a git clone per agent slot with an issue assignment
- Launch Claude Code via SSH into a named tmux session
- Monitor via
/dispatch-statusoragent-status.sh --all - Collect PR URLs and commit hashes when work is done
- Destroy VMs to stop billing
Prerequisites
| Tool | Install | Purpose |
|---|---|---|
hcloud |
brew install hcloud |
Hetzner Cloud CLI |
terraform |
brew install terraform |
Infrastructure provisioning (optional) |
packer |
brew install packer |
Build golden images |
gh |
brew install gh |
GitHub CLI |
jq |
brew install jq |
JSON parsing |
You also need:
- A Hetzner Cloud account with an API token
- An Anthropic API key for agents, or an active Claude Max subscription
- A GitHub fine-grained PAT with
contents:writeandpull_requests:writescoped to the target repo ssh-agentrunning withSSH_AUTH_SOCKset
Set your Hetzner token:
export HCLOUD_TOKEN=your-token-here
# Or use named contexts:
hcloud context create my-project
Quick Start
1. Build the Golden Image (one-time setup)
cd modules/cloud-dispatch/packer
packer init agent-image.pkr.hcl
HCLOUD_TOKEN=$HCLOUD_TOKEN packer build agent-image.pkr.hcl
Build takes 5-10 minutes. Creates a Hetzner snapshot labeled purpose=ccgm-agent.
2. (Optional) Apply Terraform Infrastructure
cd modules/cloud-dispatch/terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your SSH public key and firewall settings
terraform init
terraform apply
This creates the named SSH key and firewall rules in Hetzner that the VM scripts expect.
3. Dispatch Issues
In Claude Code, run:
/dispatch owner/repo --issues 42,43,44
Or run the pipeline directly:
# Create VMs
bash lib/vm-create.sh 1
# Initialize session credentials
bash lib/secrets-init.sh
# Inject GitHub token into all agents
GITHUB_TOKEN=$(gh auth token)
bash lib/secrets-inject-all.sh --github-token "$GITHUB_TOKEN"
# Clone repo and assign issues
bash lib/workspace-setup-all.sh "https://github.com/owner/repo.git" --issues "42,43"
# Launch agents
bash lib/agent-launch-all.sh --max-turns 200
4. Monitor Progress
/dispatch-status
Or directly:
bash lib/agent-status.sh --all
5. Collect Results and Clean Up
/dispatch-stop
Or directly:
bash lib/workspace-collect.sh --all
bash lib/agent-stop.sh --all
bash lib/secrets-cleanup.sh
bash lib/vm-destroy.sh --all --force
Commands
/dispatch
Orchestrates the full pipeline: creates VMs if needed, injects credentials, provisions workspaces, and launches agents.
/dispatch owner/repo --issues 42,43,44
/dispatch owner/repo --issues 42,43,44 --vm-count 2 --max-turns 100
Arguments extracted from the message:
REPO- GitHub repo inowner/repoformatISSUES- Comma-separated issue numbersVM_COUNT- Number of VMs to create (default: 3, or ceil(issues/4), whichever is smaller)MAX_TURNS- Max turns per agent (default: 200)
/dispatch-status
Checks the status of all running agents and reports PR URLs, last log line, and elapsed time.
/dispatch-status
/dispatch-stop
Stops all running agents and optionally destroys VMs.
/dispatch-stop
/dispatch-stop --destroy # also destroys VMs after stopping
/vm-manage
Low-level VM management. Accepts sub-commands:
| Sub-command | Description |
|---|---|
create [N] |
Create N VMs from golden image (default: 3) |
destroy [--all | name] |
Destroy one or all VMs |
status |
List all VMs with state and IP |
health |
Run health checks on all VMs |
ssh <name> |
Open SSH session into a named VM |
Architecture
System Overview
MacBook (orchestrator)
|
+-- /dispatch owner/repo --issues 42,43,44
|
+-- vm-create.sh # Boot cx22 VMs from golden snapshot
+-- secrets-init.sh # Generate session SSH keypair
+-- secrets-inject-all.sh # GitHub PAT -> /run/secrets/agent-N/ on each VM
+-- workspace-setup-all.sh# git clone + issue assignment per agent slot
+-- agent-launch-all.sh # SSH -> tmux -> claude --dangerously-skip-permissions
|
+-- VM 1 (fsn1): agent-0 (issue 42), agent-1 (issue 43)
+-- VM 2 (nbg1): agent-2 (issue 44), agent-3 (idle)
+-- VM 3 (hel1): idle
VMs are spread round-robin across three Hetzner datacenters (fsn1, nbg1, hel1) for availability.
VM Layout
Each VM runs Ubuntu 22.04 LTS with 4 isolated agent user accounts:
/
+-- home/
| +-- agent-0/ # agent user 0
| | +-- workspace/ # git clone lives here
| | +-- assignment.json # issue metadata
| | +-- status # AGENT_RUNNING / AGENT_STOPPED / AGENT_TIMEOUT
| | +-- run.log # stdout from claude process
| +-- agent-1/ ...
| +-- agent-2/ ...
| +-- agent-3/ ...
+-- opt/
| +-- ccgm/
| +-- auto-shutdown.sh # cron: idle shutdown after 15m, forced after 8h
+-- run/
| +-- secrets/ # tmpfs - credentials live here only at runtime
| +-- agent-0/
| | +-- github_token
| | +-- claude_auth
| +-- agent-1/ ...
+-- var/
+-- lib/ccgm/ # auto-shutdown state (last-active, vm-start timestamps)
+-- log/ccgm-shutdown.log
Security Model
- Credentials not in git or cloud-init: GitHub PAT and Anthropic key are injected at session time via SSH, written only to
tmpfsat/run/secrets/agent-N/ - Ephemeral session keys:
secrets-init.shgenerates a per-session ed25519 keypair, loads it into ssh-agent, and registers the public key with Hetzner.secrets-cleanup.shrevokes it. - Agent isolation: Each agent runs as a dedicated Linux user with no sudo access,
chmod 700home directory, andHISTFILE=/dev/null - Network egress allowlist: iptables rules allow only
github.com(TCP 443+22),api.anthropic.com(TCP 443), andregistry.npmjs.org(TCP 443) - Hetzner metadata API blocked: The metadata endpoint (169.254.169.254) is blocked for non-root processes
- Fine-grained PAT scoping: Scope the GitHub token to the specific repo being worked on
Network
The Terraform config creates a Hetzner firewall (ccgm-dispatch-firewall) that:
- Allows inbound SSH (port 22) from any source
- Blocks all other inbound traffic
On each VM, iptables rules (set by the golden image security hardening script) control egress:
- Allowlisted outbound:
github.com,api.anthropic.com,registry.npmjs.org - Everything else blocked for non-root users
VM Sizing
| VM Type | vCPU | RAM | Price/hr | Recommended agents |
|---|---|---|---|---|
| cx22 | 2 | 4 GB | ~$0.006 | 1-2 |
| cx32 | 4 | 8 GB | ~$0.011 | 4 |
| ccx63 | 48 | 192 GB | ~$0.58 | 4 (memory-rich) |
Price/hr figures come from the canonical HOURLY_RATES table in lib/budget-track.sh (the single source of truth for module pricing).
Default is ccx63 (set in lib/common.sh as CCGM_SERVER_TYPE). Override with:
CCGM_SERVER_TYPE=cx22 bash lib/vm-create.sh 3
Cost Reference
Cost depends on VM type, session length, and whether you're using a Claude API key or Max subscription. VM costs only (Claude API/Max subscription is separate):
| VM type | VMs | Hours/day | VM cost/month estimate |
|---|---|---|---|
| cx22 | 3 | 8 | ~$4 |
| cx32 | 3 | 8 | ~$8 |
| ccx63 | 3 | 8 | ~$418 |
VMs are billed per second on Hetzner. Destroy them when done to stop charges. The auto-shutdown cron job on each VM shuts it down after 15 minutes of inactivity or 8 hours of wall-clock time (whichever comes first).
Script Reference
VM Lifecycle
| Script | Usage | Description |
|---|---|---|
lib/vm-create.sh |
vm-create.sh [count] [--type TYPE] |
Boot N VMs from golden snapshot |
lib/vm-destroy.sh |
vm-destroy.sh --all [--force] |
Destroy VMs; prompts unless --force |
lib/vm-status.sh |
vm-status.sh |
List all ccgm-agent-* VMs with IP and state |
lib/vm-health.sh |
vm-health.sh --all |
SSH reachability + agent user + disk + memory |
lib/vm-ssh.sh |
vm-ssh.sh <name> |
Interactive SSH into a named VM |
Secret Management
| Script | Usage | Description |
|---|---|---|
lib/secrets-init.sh |
secrets-init.sh |
Generate session keypair, register with Hetzner |
lib/secrets-inject.sh |
secrets-inject.sh <ip> <agent-index> --github-token TOKEN |
Inject credentials to one agent slot |
lib/secrets-inject-all.sh |
secrets-inject-all.sh --github-token TOKEN |
Inject to all agents on all VMs |
lib/secrets-cleanup.sh |
secrets-cleanup.sh |
Revoke session key from Hetzner, clear tmpfs |
lib/secrets-rotate.sh |
secrets-rotate.sh --github-token TOKEN |
Rotate credentials without destroying VMs |
Workspace Management
| Script | Usage | Description |
|---|---|---|
lib/workspace-setup.sh |
workspace-setup.sh <ip> <agent-index> <repo-url> |
Clone repo on one agent slot |
lib/workspace-setup-all.sh |
workspace-setup-all.sh <repo-url> --issues "42,43" |
Clone and assign across all VMs |
lib/workspace-assign.sh |
workspace-assign.sh <ip> <agent-index> <issue-num> <title> |
Write assignment.json to one slot |
lib/workspace-collect.sh |
workspace-collect.sh --all [--json] |
Pull PR URLs and git state from all agents |
lib/workspace-cleanup.sh |
workspace-cleanup.sh --all |
Remove workspace dirs from VMs |
lib/ccgm-headless-install.sh |
ccgm-headless-install.sh <ccgm-repo-path> <preset> <target-user-home> |
Non-interactive CCGM installer for a cloud agent user; runs on the VM, invoked internally by workspace-setup.sh |
Agent Management
| Script | Usage | Description |
|---|---|---|
lib/agent-launch.sh |
agent-launch.sh <ip> <agent-index> [--max-turns N] |
Launch one agent via SSH + tmux |
lib/agent-launch-all.sh |
agent-launch-all.sh [--max-turns N] [--jitter N] [--dry-run] |
Launch all assigned agents with jitter |
lib/agent-status.sh |
agent-status.sh --all |
Report status, issue, last log, PR URL per agent |
lib/agent-stop.sh |
agent-stop.sh --all |
Kill tmux sessions and write AGENT_STOPPED |
lib/agent-collect.sh |
agent-collect.sh --all [--json] |
Collect agent results (status, PR, log tail) |
lib/recovery.sh |
recovery.sh check | recovery.sh retry <ip> <agent-index> | recovery.sh retry-all |
Scan agents for failures (rate-limited, crashed, timeout, error) and re-dispatch failed ones |
Cost Reporting
| Script | Usage | Description |
|---|---|---|
lib/cost-report.sh |
cost-report.sh session [--json] | cost-report.sh monthly [--json] |
Cost report for the current/last session or the current calendar month, from /tmp/ccgm-budget*.json |
VM Auto-Shutdown
| Script | Location | Description |
|---|---|---|
lib/auto-shutdown.sh |
Installed on VM at /opt/ccgm/auto-shutdown.sh |
Cron-based shutdown: idle after 15m, forced after 8h |
Configure via environment variables on the VM (/etc/ccgm/env):
| Variable | Default | Description |
|---|---|---|
MAX_HOURS |
8 |
Wall-clock hours before forced shutdown |
IDLE_MINUTES |
15 |
Minutes of no active tmux sessions before idle shutdown |
Golden Image
Building
cd modules/cloud-dispatch/packer
packer init agent-image.pkr.hcl
HCLOUD_TOKEN=$HCLOUD_TOKEN packer build agent-image.pkr.hcl
Build takes 5-10 minutes. Creates a Hetzner snapshot with label purpose=ccgm-agent. The vm-create.sh script selects the most recently created snapshot with this label.
Building with Custom Versions
HCLOUD_TOKEN=$HCLOUD_TOKEN packer build \
-var "image_version=1.1.0" \
-var "node_version=22" \
-var "claude_code_version=1.5.0" \
packer/agent-image.pkr.hcl
When to Rebuild
Rebuild the golden image when:
- A new major version of Claude Code is released
- Node.js LTS version changes
- Security patches are needed (or allow unattended-upgrades to handle them)
- Every 4-8 weeks as general maintenance
What Is Baked In
| Component | Notes |
|---|---|
| OS | Ubuntu 22.04 LTS |
| Node.js | 22 LTS (via NodeSource) |
| pnpm | Latest stable |
| Claude Code CLI | Pinned version in agent-image.pkr.hcl |
| Playwright + Chromium | Pinned version |
| git, tmux, jq, curl, python3 | System packages |
| Agent users | agent-0 through agent-3 created |
/opt/ccgm/auto-shutdown.sh |
Installed and registered in cron |
| iptables egress rules | Applied via security-hardening.sh |
Packer Files
| File | Description |
|---|---|
packer/agent-image.pkr.hcl |
Main Packer template |
packer/scripts/install-tools.sh |
Node.js, pnpm, Claude Code, Playwright |
packer/scripts/setup.sh |
Agent users, /opt/ccgm, SSH config |
packer/scripts/security-hardening.sh |
iptables, sshd hardening, unattended-upgrades |
packer/scripts/validate.sh |
Post-build verification |
Terraform Infrastructure
The Terraform config in terraform/ creates Hetzner resources that the scripts depend on:
- SSH key (
ccgm-dispatch-key) - a placeholder used when creating VMs. The actual session key is generated bysecrets-init.sh. - Firewall (
ccgm-dispatch-firewall) - inbound SSH only, all other inbound blocked
These resources are optional if you create them manually in the Hetzner dashboard, but Terraform makes it reproducible.
cd modules/cloud-dispatch/terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars
terraform init && terraform apply
Jitter
The --jitter option in agent-launch-all.sh (default: 90 seconds) staggers agent starts to avoid simultaneous Claude API requests hitting the rate limiter. With 4 agents and a 90s max jitter, total launch time is up to 4.5 minutes.
Increase jitter if you see rate limit errors. Set to 0 to disable (not recommended with more than 2 agents).
E2E Testing
A manual end-to-end test script is provided at tests/e2e-dispatch.sh. It validates the full pipeline against real Hetzner infrastructure.
# Preview without executing (no cost)
bash tests/e2e-dispatch.sh --dry-run
# Full run (~$2-3 cost, auto-cleans up)
bash tests/e2e-dispatch.sh
# Leave VMs running after test (for manual inspection)
bash tests/e2e-dispatch.sh --skip-cleanup
Prerequisites: HCLOUD_TOKEN set, golden image built, gh authenticated, ssh-agent running.
Troubleshooting
"hcloud not authenticated"
Set HCLOUD_TOKEN in your environment or run hcloud context create.
"No snapshot found matching label purpose=ccgm-agent"
Build the golden image first:
cd packer && packer build agent-image.pkr.hcl
VM boots but health check fails
SSH may not be ready yet. Wait 60 seconds and retry /vm-manage health. Check whether the VM console in the Hetzner dashboard shows a boot error.
Agent exits immediately after launch
Check that secrets were injected:
bash lib/vm-ssh.sh VM_NAME "ls /run/secrets/agent-0/"
If the directory is empty, rerun secrets-inject-all.sh.
"SSH_AUTH_SOCK is not set"
secrets-init.sh requires a running ssh-agent. Start one:
eval "$(ssh-agent -s)"
Then rerun secrets-init.sh.
Rate limit errors across agents
Increase the --jitter value when launching:
bash lib/agent-launch-all.sh --jitter 120
The default is 90 seconds. At 120 seconds, 4 agents take up to 6 minutes to start.
"No running ccgm-agent-* VMs found"
Run vm-status.sh to see VM states. VMs may have auto-shutdown. Create new ones with vm-create.sh.
Secrets-cleanup fails to find session file
If /tmp/ccgm-session.json was removed (e.g., VM reboot), manually delete the session key from Hetzner:
hcloud ssh-key list
hcloud ssh-key delete ccgm-session-TIMESTAMP
Manual Installation
If you are not using the CCGM installer, copy the files manually. The lib/ scripts install into
their own ~/.claude/lib/cloud-dispatch/ subdirectory — the slash commands invoke them by that
hardcoded path (bash ~/.claude/lib/cloud-dispatch/<script>.sh), and a flat ~/.claude/lib/ would
collide with other modules' files (e.g. common.sh).
mkdir -p ~/.claude/commands ~/.claude/rules ~/.claude/lib/cloud-dispatch
cp commands/*.md ~/.claude/commands/
cp rules/cloud-dispatch.md ~/.claude/rules/cloud-dispatch.md
cp lib/*.sh ~/.claude/lib/cloud-dispatch/
chmod +x ~/.claude/lib/cloud-dispatch/*.sh
Then:
- Set
HCLOUD_TOKENin your environment - Build the golden image (see Quick Start)
The packer/, terraform/, and tests/ directories stay in the repo — they are build and
infrastructure sources, not installed files. Run them from a checkout.
Will install
| Path | Action | Target | Type |
|---|---|---|---|
commands/dispatch.md | → | commands/dispatch.md | command |
commands/dispatch-status.md | → | commands/dispatch-status.md | command |
commands/dispatch-stop.md | → | commands/dispatch-stop.md | command |
commands/vm-manage.md | → | commands/vm-manage.md | command |
rules/cloud-dispatch.md | → | rules/cloud-dispatch.md | rule |
lib/agent-collect.sh | → | lib/cloud-dispatch/agent-collect.sh | lib |
lib/agent-launch.sh | → | lib/cloud-dispatch/agent-launch.sh | lib |
lib/agent-launch-all.sh | → | lib/cloud-dispatch/agent-launch-all.sh | lib |
lib/agent-status.sh | → | lib/cloud-dispatch/agent-status.sh | lib |
lib/agent-stop.sh | → | lib/cloud-dispatch/agent-stop.sh | lib |
lib/auto-shutdown.sh | → | lib/cloud-dispatch/auto-shutdown.sh | lib |
lib/budget-track.sh | → | lib/cloud-dispatch/budget-track.sh | lib |
lib/ccgm-headless-install.sh | → | lib/cloud-dispatch/ccgm-headless-install.sh | lib |
lib/cost-report.sh | → | lib/cloud-dispatch/cost-report.sh | lib |
lib/recovery.sh | → | lib/cloud-dispatch/recovery.sh | lib |
lib/common.sh | → | lib/cloud-dispatch/common.sh | lib |
lib/vm-create.sh | → | lib/cloud-dispatch/vm-create.sh | lib |
lib/vm-destroy.sh | → | lib/cloud-dispatch/vm-destroy.sh | lib |
lib/vm-health.sh | → | lib/cloud-dispatch/vm-health.sh | lib |
lib/vm-ssh.sh | → | lib/cloud-dispatch/vm-ssh.sh | lib |
lib/vm-status.sh | → | lib/cloud-dispatch/vm-status.sh | lib |
lib/secrets-cleanup.sh | → | lib/cloud-dispatch/secrets-cleanup.sh | lib |
lib/secrets-init.sh | → | lib/cloud-dispatch/secrets-init.sh | lib |
lib/secrets-inject-all.sh | → | lib/cloud-dispatch/secrets-inject-all.sh | lib |
lib/secrets-inject.sh | → | lib/cloud-dispatch/secrets-inject.sh | lib |
lib/secrets-rotate.sh | → | lib/cloud-dispatch/secrets-rotate.sh | lib |
lib/workspace-assign.sh | → | lib/cloud-dispatch/workspace-assign.sh | lib |
lib/workspace-cleanup.sh | → | lib/cloud-dispatch/workspace-cleanup.sh | lib |
lib/workspace-collect.sh | → | lib/cloud-dispatch/workspace-collect.sh | lib |
lib/workspace-setup-all.sh | → | lib/cloud-dispatch/workspace-setup-all.sh | lib |
lib/workspace-setup.sh | → | lib/cloud-dispatch/workspace-setup.sh | lib |
Dependencies
No dependencies.
Required by
No other module depends on this one.
Asks during install
-
Set HCLOUD_TOKEN in your shell environment (e.g. export HCLOUD_TOKEN=... in ~/.zshrc). See https://console.hetzner.cloud -> Security -> API Tokens.
Default:
Included in presets
Install this module
Agent prompt
Recommended for agent users -- hands the whole install off to your assistant.
Fetch https://cd23a9be.ccgm-site.pages.dev/modules/cloud-dispatch.md and install this module into my Claude Code setup.
Native plugin marketplace
One command via the native plugin marketplace -- additive, does not merge settings.json.
claude plugin install cloud-dispatch@ccgm
The marketplace path is additive, not a replacement: it installs commands, agents, and skills as native plugin components, but it does not perform the bash installer's deep settings.json merge, and it does not write the always-loaded global CLAUDE.md context. Rules are only injected via an opt-in SessionStart hook rather than being auto-loaded. Use the bash installer when those pieces matter to you.
Files
rule (1)
rules/cloud-dispatch.md
# Cloud Dispatch Rules Rules that apply when working with cloud-dispatched agents. ## VM Agent Behavior When running as an agent on a cloud VM: - Always create a feature branch before making changes - Commit frequently (every significant change) for git-based recovery - Create a PR when work is complete - Never push directly to main - Use --max-turns to prevent runaway execution ## Dispatch Workflow When dispatching work to cloud VMs: - Verify VMs are healthy before dispatching - Use jittered starts to avoid rate limit thundering herd - Monitor agent status periodically - Collect results and clean up VMs when done - Track costs and stay within budget ## Security - Never embed secrets in cloud-init or commit them to git - Use fine-grained GitHub PATs scoped to the target repo - Session SSH keys are ephemeral - generated per session, revoked on cleanup - Agent isolation: each agent runs as a separate Linux user with no sudo access
command (4)
commands/dispatch.md
---
description: Delegate work to cloud VMs - dispatch GitHub issues to autonomous Claude Code agents running on Hetzner Cloud
allowed-tools: Bash
---
# /dispatch - Delegate work to cloud VMs
Dispatch GitHub issues to autonomous Claude Code agents running on Hetzner Cloud VMs.
## Usage
The user will provide:
- Which repo to work on
- Which issues to dispatch (by number)
- Optionally: number of VMs, max turns, time limit
## Execution Steps
Follow these steps in order. Use the Bash tool to run the shell scripts.
### Step 1: Parse Arguments
Extract from the user's message:
- `REPO`: GitHub repo (owner/repo format, or just repo name to resolve from ~/code/)
- `ISSUES`: Comma-separated issue numbers
- `VM_COUNT`: Number of VMs (default: 3, or issues / 4 rounded up, whichever is smaller)
- `MAX_TURNS`: Max turns per agent (default: 200)
- `MAX_HOURS`: Max hours before auto-shutdown (default: 4)
### Step 2: Validate Prerequisites
```bash
# Check required tools
command -v hcloud >/dev/null 2>&1 || { echo "ERROR: hcloud CLI not installed. Run: brew install hcloud"; exit 1; }
command -v gh >/dev/null 2>&1 || { echo "ERROR: gh CLI not installed. Run: brew install gh"; exit 1; }
# Check Hetzner auth
hcloud server-type list >/dev/null 2>&1 || { echo "ERROR: hcloud not authenticated. Set HCLOUD_TOKEN env var or run: hcloud context create"; exit 1; }
# Check GitHub auth
gh auth status >/dev/null 2>&1 || { echo "ERROR: gh not authenticated. Run: gh auth login"; exit 1; }
```
### Step 3: Check VM Status
```bash
source ~/.claude/lib/cloud-dispatch/common.sh
bash ~/.claude/lib/cloud-dispatch/vm-status.sh
```
If no VMs are running, create them:
```bash
bash ~/.claude/lib/cloud-dispatch/vm-create.sh $VM_COUNT
```
If VMs exist, health-check them:
```bash
bash ~/.claude/lib/cloud-dispatch/vm-health.sh --all
```
### Step 4: Initialize Session Secrets
```bash
bash ~/.claude/lib/cloud-dispatch/secrets-init.sh
```
Ask the user for their GitHub token if not already configured:
"I need a GitHub fine-grained PAT with `contents:write` and `pull_requests:write` scoped to the target repo. Provide it now, or press Enter to use the token from `gh auth token`."
If the user provides a token, set `GITHUB_TOKEN` to that value. Otherwise:
```bash
GITHUB_TOKEN=$(gh auth token)
```
Then inject secrets to all VMs:
```bash
bash ~/.claude/lib/cloud-dispatch/secrets-inject-all.sh --github-token "$GITHUB_TOKEN"
```
### Step 5: Set Up Workspaces
```bash
bash ~/.claude/lib/cloud-dispatch/workspace-setup-all.sh "https://github.com/$REPO.git" --issues "$ISSUES"
```
### Step 6: Launch Agents
```bash
bash ~/.claude/lib/cloud-dispatch/agent-launch-all.sh --max-turns $MAX_TURNS --jitter 75
```
> **Concurrency — avoid the 429 throttle.** The org-level rate limit applies across *all* VMs at once, not per machine — every remote agent's requests bill against the same ceiling. Keep `VM_COUNT` modest and the launch `--jitter` on (it staggers VM starts so they don't burst together). If a run reports `Server is temporarily limiting requests · Rate limited`, that is the server throttle, not a usage cap — reduce `VM_COUNT` and re-dispatch the affected issues rather than relaunching the whole fleet. See `~/.claude/rules/concurrency-and-rate-limits.md`.
### Step 7: Report
Print a summary of what was dispatched:
- Number of agents launched
- Which issues were assigned to which VM/agent
- How to check status: "Run /dispatch-status to check progress"
- How to stop: "Run /dispatch-stop to terminate all agents"
- Estimated cost: roughly $0.006/hour per cx22 VM (3 VMs = $0.018/hour). For the default ccx63, ~$0.58/hour per VM. See the canonical rate table in `~/.claude/lib/cloud-dispatch/budget-track.sh`.
commands/dispatch-status.md
--- description: Check the status of dispatched agents across all cloud VMs allowed-tools: Bash --- # /dispatch-status - Check agent status Check the status of dispatched agents across all cloud VMs. ## Execution ### Step 1: Check Agent Status ```bash bash ~/.claude/lib/cloud-dispatch/agent-status.sh --all ``` ### Step 2: Collect Results Pull PR URLs and completed work from all VMs: ```bash bash ~/.claude/lib/cloud-dispatch/workspace-collect.sh --all ``` ### Step 3: Present Summary Format and present the results showing each agent's: - VM name and agent slot (e.g. ccgm-agent-1 / agent-0) - Assigned issue number - Status: running / completed / failed / idle - PR URL (if a PR was opened) - Last git commit message (if available) If all agents have completed, remind the user they can run `/dispatch-stop` to clean up.
commands/dispatch-stop.md
--- description: Stop all dispatched agents and optionally destroy cloud VMs allowed-tools: Bash --- # /dispatch-stop - Stop dispatched agents Stop all running agents and optionally destroy VMs. ## Execution ### Step 1: Stop Agents ```bash bash ~/.claude/lib/cloud-dispatch/agent-stop.sh --all ``` ### Step 2: Collect Results Pull any final results, PRs, or uncommitted work before cleanup: ```bash bash ~/.claude/lib/cloud-dispatch/workspace-collect.sh --all ``` ### Step 3: Ask About Cleanup Ask the user: "Agents stopped. What do you want to do with the VMs? 1. Keep running (reuse for another dispatch later - saves ~2 min boot time) 2. Destroy VMs (clean shutdown, stops billing)" If the user chooses destroy: ```bash # Revoke session SSH keys and clear secrets from tmpfs bash ~/.claude/lib/cloud-dispatch/secrets-cleanup.sh # Destroy all dispatch VMs bash ~/.claude/lib/cloud-dispatch/vm-destroy.sh --all --force ``` Confirm destruction with a final status: ```bash bash ~/.claude/lib/cloud-dispatch/vm-status.sh ```
commands/vm-manage.md
--- description: Manage Hetzner Cloud VMs for agent dispatch - create, destroy, status, health checks, and SSH access allowed-tools: Bash --- # /vm-manage - Manage cloud VMs Manage Hetzner Cloud VMs for agent dispatch. ## Usage The user will specify an action: - `create [N]` - Create N VMs (default 3) - `destroy [--all | name]` - Destroy one or all VMs - `status` - List all VMs and their current state - `health` - Run health checks on all VMs - `ssh <name>` - Open an SSH session into a VM ## Execution Parse the action from the user's message and run the corresponding script: ```bash # For create: bash ~/.claude/lib/cloud-dispatch/vm-create.sh $N # For destroy (all): bash ~/.claude/lib/cloud-dispatch/vm-destroy.sh --all # For destroy (specific VM): bash ~/.claude/lib/cloud-dispatch/vm-destroy.sh $VM_NAME # For status: bash ~/.claude/lib/cloud-dispatch/vm-status.sh # For health: bash ~/.claude/lib/cloud-dispatch/vm-health.sh --all # For ssh: bash ~/.claude/lib/cloud-dispatch/vm-ssh.sh $VM_NAME ``` After running each command, display the output clearly. For `status`, format it as a table showing VM name, IP, state, and uptime.
lib (26)
lib/agent-collect.sh
#!/usr/bin/env bash
# agent-collect.sh — Collect run results from one or all Claude Code agents.
#
# Usage:
# agent-collect.sh --all [--json] [--log-lines N]
# agent-collect.sh <vm-ip> <agent-index> [--json] [--log-lines N]
#
# Output:
# Formatted summary table (or JSON with --json) containing:
# - Agent status (from ~/status)
# - PR URL (from run.log or gh CLI)
# - Branch name and last commit
# - Exit status
# - Last N lines of run.log (default: 20)
#
# Requires:
# - hcloud CLI (when --all is used)
# - SSH key in ~/.ssh/ccgm-dispatch-session or $SSH_KEY_PATH
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --all [--json] [--log-lines N]" >&2
echo " $0 <vm-ip> <agent-index> [--json] [--log-lines N]" >&2
exit 1
fi
COLLECT_ALL=false
VM_IP=""
AGENT_INDEX=""
JSON_OUTPUT=false
LOG_LINES=20
while [[ $# -gt 0 ]]; do
case "$1" in
--all)
COLLECT_ALL=true
shift
;;
--json)
JSON_OUTPUT=true
shift
;;
--log-lines)
LOG_LINES="$2"
shift 2
;;
-*)
log_error "Unknown option: $1"
exit 1
;;
*)
if [[ -z "${VM_IP}" ]]; then
VM_IP="$1"
elif [[ -z "${AGENT_INDEX}" ]]; then
AGENT_INDEX="$1"
else
log_error "Unexpected argument: $1"
exit 1
fi
shift
;;
esac
done
if [[ "${COLLECT_ALL}" == "false" ]]; then
if [[ -z "${VM_IP}" || -z "${AGENT_INDEX}" ]]; then
log_error "Provide --all or both <vm-ip> and <agent-index>"
exit 1
fi
if ! [[ "${AGENT_INDEX}" =~ ^[0-3]$ ]]; then
log_error "agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})"
exit 1
fi
fi
if ! [[ "${LOG_LINES}" =~ ^[0-9]+$ ]]; then
log_error "--log-lines must be a positive integer (got: ${LOG_LINES})"
exit 1
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
if [[ "${COLLECT_ALL}" == "true" ]]; then
require_cmd hcloud
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set."
exit 1
fi
fi
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
ssh_root_quiet() {
local ip="$1"; shift
# shellcheck disable=SC2206
read -ra _opts <<< "$(ssh_opts)"
# SC2029: expansion on client side is intentional
# shellcheck disable=SC2029
ssh "${_opts[@]}" "root@${ip}" "$@" 2>/dev/null
}
json_escape() {
printf '%s' "$1" | python3 -c "import json,sys; print(json.dumps(sys.stdin.read()))" 2>/dev/null \
|| printf '"%s"' "$1"
}
collect_one() {
local ip="$1"
local vm_name="$2"
local idx="$3"
local agent_user="agent-${idx}"
local agent_home="/home/${agent_user}"
local assignment_file="${agent_home}/assignment.json"
local status_file="${agent_home}/status"
local run_log="${agent_home}/run.log"
# --- assignment ---
local issue_number="" issue_title="" repo="" branch=""
if ssh_root_quiet "${ip}" "test -f '${assignment_file}'"; then
local raw_assignment
raw_assignment=$(ssh_root_quiet "${ip}" "cat '${assignment_file}'" || echo "{}")
issue_number=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_number',''))" 2>/dev/null || true)
issue_title=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_title',''))" 2>/dev/null || true)
repo=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('repo',''))" 2>/dev/null || true)
branch=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('branch',''))" 2>/dev/null || true)
fi
# --- status file ---
local agent_status="unknown"
if ssh_root_quiet "${ip}" "test -f '${status_file}'"; then
agent_status=$(ssh_root_quiet "${ip}" "cat '${status_file}'" 2>/dev/null | tr -d '[:space:]' || echo "unknown")
fi
# --- git state ---
local current_branch="" last_commit=""
if [[ -n "${repo}" ]]; then
local repo_name
repo_name=$(basename "${repo}")
local clone_dir="${agent_home}/workspace/${repo_name}"
if ssh_root_quiet "${ip}" "test -d '${clone_dir}/.git'"; then
current_branch=$(ssh_root_quiet "${ip}" \
"su - ${agent_user} -c 'git -C ${clone_dir} rev-parse --abbrev-ref HEAD'" \
2>/dev/null || echo "")
local sha msg
sha=$(ssh_root_quiet "${ip}" \
"su - ${agent_user} -c 'git -C ${clone_dir} rev-parse --short HEAD'" \
2>/dev/null || echo "")
msg=$(ssh_root_quiet "${ip}" \
"su - ${agent_user} -c 'git -C ${clone_dir} log -1 --pretty=%s'" \
2>/dev/null || echo "")
[[ -n "${sha}" ]] && last_commit="${sha} ${msg}"
fi
fi
# --- PR URL ---
local pr_url=""
if [[ -n "${branch}" && -n "${repo}" ]]; then
pr_url=$(ssh_root_quiet "${ip}" \
"su - ${agent_user} -c 'gh pr list --repo ${repo} --head ${branch} --json url --jq .[0].url 2>/dev/null'" \
2>/dev/null || true)
fi
if [[ -z "${pr_url}" ]]; then
pr_url=$(ssh_root_quiet "${ip}" \
"grep -oE 'https://github.com/[^/]+/[^/]+/pull/[0-9]+' '${run_log}' 2>/dev/null | tail -1" \
2>/dev/null || true)
fi
# --- log tail ---
local log_tail=""
if ssh_root_quiet "${ip}" "test -f '${run_log}'"; then
log_tail=$(ssh_root_quiet "${ip}" "tail -n ${LOG_LINES} '${run_log}'" 2>/dev/null || true)
fi
# ---------------------------------------------------------------------------
# Output
# ---------------------------------------------------------------------------
if [[ "${JSON_OUTPUT}" == "true" ]]; then
printf '{\n'
printf ' "vm": %s,\n' "$(json_escape "${vm_name}")"
printf ' "agent": %s,\n' "$(json_escape "${agent_user}")"
printf ' "issue_number": %s,\n' "${issue_number:-null}"
printf ' "issue_title": %s,\n' "$(json_escape "${issue_title}")"
printf ' "repo": %s,\n' "$(json_escape "${repo}")"
printf ' "branch": %s,\n' "$(json_escape "${branch}")"
printf ' "current_branch": %s,\n' "$(json_escape "${current_branch}")"
printf ' "last_commit": %s,\n' "$(json_escape "${last_commit}")"
printf ' "pr_url": %s,\n' "$(json_escape "${pr_url}")"
printf ' "status": %s,\n' "$(json_escape "${agent_status}")"
printf ' "log_tail": %s\n' "$(json_escape "${log_tail}")"
printf '}'
else
echo "--- ${vm_name} / ${agent_user} ---"
printf "Status: %s\n" "${agent_status}"
if [[ -n "${issue_number}" ]]; then
printf "Issue: #%s - %s\n" "${issue_number}" "${issue_title}"
else
printf "Issue: (no assignment)\n"
fi
printf "Repo: %s\n" "${repo:-(none)}"
printf "Branch: %s\n" "${branch:-${current_branch:-(none)}}"
if [[ -n "${last_commit}" ]]; then
printf "Last commit: %s\n" "${last_commit}"
fi
if [[ -n "${pr_url}" ]]; then
printf "PR: %s\n" "${pr_url}"
fi
if [[ -n "${log_tail}" ]]; then
echo ""
echo "Log (last ${LOG_LINES} lines):"
while IFS= read -r line; do
printf ' %s\n' "${line}"
done <<< "${log_tail}"
fi
echo ""
fi
}
# ---------------------------------------------------------------------------
# Single agent
# ---------------------------------------------------------------------------
if [[ "${COLLECT_ALL}" == "false" ]]; then
VM_NAME="${VM_IP}"
if command -v hcloud &>/dev/null && [[ -n "${HCLOUD_TOKEN:-}" ]]; then
VM_NAME=$(hcloud server list --output columns=name,ipv4 2>/dev/null \
| awk -v ip="${VM_IP}" '$2==ip {print $1}' || echo "${VM_IP}")
fi
collect_one "${VM_IP}" "${VM_NAME}" "${AGENT_INDEX}"
exit 0
fi
# ---------------------------------------------------------------------------
# All agents across all running VMs
# ---------------------------------------------------------------------------
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status \
| awk '$2=="running" && /ccgm-agent/ {print $1}' \
| sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
log_warn "No running ccgm-agent-* VMs found."
exit 0
fi
if [[ "${JSON_OUTPUT}" == "true" ]]; then
echo "["
first_entry=true
fi
for vm_name in "${VM_NAMES[@]}"; do
ip=$(hcloud server describe "${vm_name}" --output format='{{.PublicNet.IPv4.IP}}')
for idx in $(seq 0 $(( CCGM_AGENTS_PER_VM - 1 ))); do
if [[ "${JSON_OUTPUT}" == "true" && "${first_entry}" != "true" ]]; then
echo ","
fi
collect_one "${ip}" "${vm_name}" "${idx}"
first_entry=false
done
done
if [[ "${JSON_OUTPUT}" == "true" ]]; then
echo ""
echo "]"
fi
lib/agent-launch.sh
#!/usr/bin/env bash
# agent-launch.sh — Launch a single Claude Code agent in a tmux session on a VM.
#
# Usage:
# agent-launch.sh <vm-ip> <agent-index> [--max-turns N] [--prompt PROMPT]
#
# Arguments:
# vm-ip Public IP of the Hetzner Cloud VM
# agent-index 0-3, selects agent-N user on the VM
# --max-turns Maximum turns before Claude stops (default: 200)
# --prompt Override default Claude prompt
#
# The agent's assignment is read from /home/agent-N/assignment.json on the VM.
# The tmux session is named agent-N and output is appended to ~/run.log.
# On completion, ~/status is set to AGENT_DONE (or AGENT_ERROR on failure).
#
# Requires:
# - SSH key in ~/.ssh/ccgm-dispatch-session or $SSH_KEY_PATH
# - assignment.json already written (run workspace-assign.sh first)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -lt 2 ]]; then
echo "Usage: $0 <vm-ip> <agent-index> [--max-turns N] [--prompt PROMPT]" >&2
exit 1
fi
VM_IP="$1"
AGENT_INDEX="$2"
shift 2
MAX_TURNS=200
CUSTOM_PROMPT=""
while [[ $# -gt 0 ]]; do
case "$1" in
--max-turns)
MAX_TURNS="$2"
shift 2
;;
--prompt)
CUSTOM_PROMPT="$2"
shift 2
;;
*)
log_error "Unknown argument: $1"
exit 1
;;
esac
done
if ! [[ "${AGENT_INDEX}" =~ ^[0-3]$ ]]; then
log_error "agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})"
exit 1
fi
if ! [[ "${MAX_TURNS}" =~ ^[0-9]+$ ]]; then
log_error "--max-turns must be a positive integer (got: ${MAX_TURNS})"
exit 1
fi
AGENT_USER="agent-${AGENT_INDEX}"
AGENT_HOME="/home/${AGENT_USER}"
ASSIGNMENT_FILE="${AGENT_HOME}/assignment.json"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
ssh_root() {
# shellcheck disable=SC2206
read -ra _opts <<< "$(ssh_opts)"
# SC2029: expansion on client side is intentional (remote command args are built locally)
# shellcheck disable=SC2029
ssh "${_opts[@]}" "root@${VM_IP}" "$@"
}
# ---------------------------------------------------------------------------
# Step 1: Verify assignment.json exists
# ---------------------------------------------------------------------------
log_info "Checking assignment for ${AGENT_USER} on ${VM_IP}"
if ! ssh_root "test -f '${ASSIGNMENT_FILE}'" 2>/dev/null; then
log_error "No assignment.json found at ${ASSIGNMENT_FILE}. Run workspace-assign.sh first."
exit 1
fi
# ---------------------------------------------------------------------------
# Step 2: Read assignment fields
# ---------------------------------------------------------------------------
ASSIGNMENT=$(ssh_root "cat '${ASSIGNMENT_FILE}'" 2>/dev/null)
ISSUE_NUMBER=$(echo "${ASSIGNMENT}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_number',''))" 2>/dev/null || true)
ISSUE_TITLE=$(echo "${ASSIGNMENT}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_title',''))" 2>/dev/null || true)
REPO=$(echo "${ASSIGNMENT}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('repo',''))" 2>/dev/null || true)
BRANCH=$(echo "${ASSIGNMENT}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('branch',''))" 2>/dev/null || true)
if [[ -z "${ISSUE_NUMBER}" || -z "${REPO}" ]]; then
log_error "assignment.json is missing required fields (issue_number, repo)."
exit 1
fi
REPO_NAME=$(basename "${REPO}")
WORKSPACE_DIR="${AGENT_HOME}/workspace/${REPO_NAME}"
# ---------------------------------------------------------------------------
# Step 3: Build the Claude prompt
# ---------------------------------------------------------------------------
if [[ -n "${CUSTOM_PROMPT}" ]]; then
PROMPT="${CUSTOM_PROMPT}"
else
PROMPT="You are an autonomous Claude Code agent. Work on GitHub issue #${ISSUE_NUMBER}: ${ISSUE_TITLE}. \
Create a branch named ${BRANCH}, implement the changes with tests, and create a PR that closes #${ISSUE_NUMBER}. \
Follow the repo's CLAUDE.md instructions. Commit with message format: #${ISSUE_NUMBER}: description."
fi
# ---------------------------------------------------------------------------
# Step 4: Kill any existing session with the same name
# ---------------------------------------------------------------------------
log_info "Launching tmux session '${AGENT_USER}' on ${VM_IP}"
# Silently kill a stale session if present
ssh_root "su - ${AGENT_USER} -c 'tmux kill-session -t ${AGENT_USER} 2>/dev/null || true'" 2>/dev/null || true
# ---------------------------------------------------------------------------
# Step 5: Build the inner shell command
#
# The command runs as agent-N inside tmux. It:
# 1. Sources the injected secrets env file
# 2. cd's into the workspace clone
# 3. Runs claude in print mode
# 4. Writes AGENT_DONE (or AGENT_ERROR) to ~/status
# ---------------------------------------------------------------------------
# Single-quote the prompt for safe embedding in the outer double-quoted string.
# We use printf %q which escapes for bash, then wrap in single quotes for the
# su -c argument.
ESCAPED_PROMPT=$(printf '%s' "${PROMPT}" | sed "s/'/'\\\\''/g")
INNER_CMD="source /run/secrets/${AGENT_USER}/env 2>/dev/null || true; \
cd '${WORKSPACE_DIR}'; \
claude -p '${ESCAPED_PROMPT}' --dangerously-skip-permissions --max-turns ${MAX_TURNS} \
>> ~/run.log 2>&1 \
&& echo AGENT_DONE > ~/status \
|| echo AGENT_ERROR > ~/status"
# Wrap inner command for su -c (needs single outer quotes)
SU_CMD="tmux new-session -d -s ${AGENT_USER} 'bash -lc $(printf '%q' "${INNER_CMD}")'"
ssh_root "su - ${AGENT_USER} -c $(printf '%q' "${SU_CMD}")"
# ---------------------------------------------------------------------------
# Step 6: Verify the session was created
# ---------------------------------------------------------------------------
if ssh_root "su - ${AGENT_USER} -c 'tmux has-session -t ${AGENT_USER} 2>/dev/null'" 2>/dev/null; then
log_success "Session '${AGENT_USER}' is running on ${VM_IP}"
echo " Agent: ${AGENT_USER}@${VM_IP}"
echo " Issue: #${ISSUE_NUMBER} - ${ISSUE_TITLE}"
echo " Branch: ${BRANCH}"
echo " Repo: ${REPO}"
else
log_error "tmux session '${AGENT_USER}' did not start on ${VM_IP}"
exit 1
fi
lib/agent-launch-all.sh
#!/usr/bin/env bash
# agent-launch-all.sh — Launch all assigned agents across all running VMs with jitter.
#
# Usage:
# agent-launch-all.sh [--jitter SECONDS] [--max-turns N] [--prompt PROMPT] [--dry-run]
#
# Options:
# --jitter N Random sleep max between agent launches in seconds (default: 90).
# Actual sleep is a random value between 60 and JITTER (min 60).
# Set to 0 to disable jitter.
# --max-turns N Maximum turns per agent (default: 200)
# --prompt P Claude prompt override (passed to agent-launch.sh)
# --dry-run Print what would be launched without executing
#
# Jitter is critical: it staggers Claude API calls so all agents don't hit the
# rate limiter simultaneously.
#
# Requires:
# - hcloud CLI installed and authenticated (HCLOUD_TOKEN set)
# - agent-launch.sh in the same directory as this script
# - SSH key in ~/.ssh/ccgm-dispatch-session or $SSH_KEY_PATH
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
MAX_TURNS=200
JITTER=90
CUSTOM_PROMPT=""
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case "$1" in
--jitter)
JITTER="$2"
shift 2
;;
--max-turns)
MAX_TURNS="$2"
shift 2
;;
--prompt)
CUSTOM_PROMPT="$2"
shift 2
;;
--dry-run)
DRY_RUN=true
shift
;;
*)
log_error "Unknown argument: $1"
exit 1
;;
esac
done
if ! [[ "${MAX_TURNS}" =~ ^[0-9]+$ ]]; then
log_error "--max-turns must be a positive integer (got: ${MAX_TURNS})"
exit 1
fi
if ! [[ "${JITTER}" =~ ^[0-9]+$ ]]; then
log_error "--jitter must be a non-negative integer (got: ${JITTER})"
exit 1
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd hcloud
require_cmd python3
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Export it before running this script."
exit 1
fi
LAUNCH_SCRIPT="${SCRIPT_DIR}/agent-launch.sh"
if [[ ! -x "${LAUNCH_SCRIPT}" ]]; then
log_error "agent-launch.sh not found or not executable at ${LAUNCH_SCRIPT}"
exit 1
fi
# ---------------------------------------------------------------------------
# Discover running agent VMs
# ---------------------------------------------------------------------------
log_info "Discovering running ccgm-agent-* VMs..."
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status \
| awk '$2=="running" && /ccgm-agent/ {print $1}' \
| sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
log_error "No running ccgm-agent-* VMs found."
exit 1
fi
log_info "Found ${#VM_NAMES[@]} VM(s): ${VM_NAMES[*]}"
# ---------------------------------------------------------------------------
# Build launch list: only slots that have an assignment.json
# ---------------------------------------------------------------------------
ssh_root() {
local ip="$1"; shift
# shellcheck disable=SC2206
read -ra _opts <<< "$(ssh_opts)"
# SC2029: expansion on client side is intentional
# shellcheck disable=SC2029
ssh "${_opts[@]}" "root@${ip}" "$@" 2>/dev/null
}
declare -a LAUNCH_VM_IPS
declare -a LAUNCH_VM_NAMES
declare -a LAUNCH_AGENT_INDEXES
log_info "Checking agent assignments..."
for vm_name in "${VM_NAMES[@]}"; do
vm_ip=$(hcloud server describe "${vm_name}" --output format='{{.PublicNet.IPv4.IP}}')
for agent_index in $(seq 0 $(( CCGM_AGENTS_PER_VM - 1 ))); do
agent_user="agent-${agent_index}"
assignment_file="/home/${agent_user}/assignment.json"
if ssh_root "${vm_ip}" "test -f '${assignment_file}'"; then
LAUNCH_VM_IPS+=("${vm_ip}")
LAUNCH_VM_NAMES+=("${vm_name}")
LAUNCH_AGENT_INDEXES+=("${agent_index}")
log_info " ${vm_name} / ${agent_user} - has assignment"
else
log_warn " ${vm_name} / ${agent_user} - no assignment, skipping"
fi
done
done
TOTAL="${#LAUNCH_VM_IPS[@]}"
if [[ "${TOTAL}" -eq 0 ]]; then
log_warn "No agents have assignments. Run workspace-assign.sh (or workspace-setup-all.sh) first."
exit 0
fi
log_info "Agents to launch: ${TOTAL}"
if [[ "${DRY_RUN}" == "true" ]]; then
echo ""
echo "Dry run - agents that would be launched:"
for i in "${!LAUNCH_VM_IPS[@]}"; do
printf " %s / agent-%s\n" "${LAUNCH_VM_NAMES[$i]}" "${LAUNCH_AGENT_INDEXES[$i]}"
done
echo ""
echo "(--dry-run: no changes made)"
exit 0
fi
# ---------------------------------------------------------------------------
# Launch agents with jitter between each
# ---------------------------------------------------------------------------
LAUNCHED=0
FAILED=0
START_TIME=$(date +%s)
for i in "${!LAUNCH_VM_IPS[@]}"; do
vm_ip="${LAUNCH_VM_IPS[$i]}"
vm_name="${LAUNCH_VM_NAMES[$i]}"
agent_index="${LAUNCH_AGENT_INDEXES[$i]}"
agent_user="agent-${agent_index}"
log_info "[$(( i + 1 ))/${TOTAL}] Launching ${vm_name} / ${agent_user}"
launch_args=("${vm_ip}" "${agent_index}" --max-turns "${MAX_TURNS}")
if [[ -n "${CUSTOM_PROMPT}" ]]; then
launch_args+=(--prompt "${CUSTOM_PROMPT}")
fi
if "${LAUNCH_SCRIPT}" "${launch_args[@]}"; then
LAUNCHED=$(( LAUNCHED + 1 ))
else
log_error "Launch failed for ${vm_name} / ${agent_user}"
FAILED=$(( FAILED + 1 ))
fi
# Jitter: sleep a random amount between 60s and JITTER (skip after last agent)
if [[ "${JITTER}" -gt 0 && $(( i + 1 )) -lt "${TOTAL}" ]]; then
if [[ "${JITTER}" -gt 60 ]]; then
sleep_secs=$(( 60 + RANDOM % (JITTER - 60 + 1) ))
else
sleep_secs="${JITTER}"
fi
log_info "Jitter: sleeping ${sleep_secs}s before next launch..."
sleep "${sleep_secs}"
fi
done
END_TIME=$(date +%s)
ELAPSED=$(( END_TIME - START_TIME ))
echo ""
log_info "Launch complete"
echo " Total agents launched: ${LAUNCHED}"
if [[ "${FAILED}" -gt 0 ]]; then
echo " Failed: ${FAILED}"
fi
echo " Total elapsed: ${ELAPSED}s"
[[ "${FAILED}" -eq 0 ]]
lib/agent-status.sh
#!/usr/bin/env bash
# agent-status.sh — Report the status of one or all Claude Code agents.
#
# Usage:
# agent-status.sh --all
# agent-status.sh <vm-ip> <agent-index>
#
# Output per agent:
# VM: ccgm-agent-fsn1-0 | Agent: agent-0 | Status: running | Issue: #42
# Last log: "Creating PR for branch 42-habit-streaks..."
# Duration: 23m
#
# Requires:
# - hcloud CLI (when --all is used)
# - SSH key in ~/.ssh/ccgm-dispatch-session or $SSH_KEY_PATH
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --all" >&2
echo " $0 <vm-ip> <agent-index>" >&2
exit 1
fi
STATUS_ALL=false
VM_IP=""
AGENT_INDEX=""
while [[ $# -gt 0 ]]; do
case "$1" in
--all)
STATUS_ALL=true
shift
;;
-*)
log_error "Unknown option: $1"
exit 1
;;
*)
if [[ -z "${VM_IP}" ]]; then
VM_IP="$1"
elif [[ -z "${AGENT_INDEX}" ]]; then
AGENT_INDEX="$1"
else
log_error "Unexpected argument: $1"
exit 1
fi
shift
;;
esac
done
if [[ "${STATUS_ALL}" == "false" ]]; then
if [[ -z "${VM_IP}" || -z "${AGENT_INDEX}" ]]; then
log_error "Provide --all or both <vm-ip> and <agent-index>"
exit 1
fi
if ! [[ "${AGENT_INDEX}" =~ ^[0-3]$ ]]; then
log_error "agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})"
exit 1
fi
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
if [[ "${STATUS_ALL}" == "true" ]]; then
require_cmd hcloud
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set."
exit 1
fi
fi
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
ssh_root_quiet() {
local ip="$1"; shift
# shellcheck disable=SC2206
read -ra _opts <<< "$(ssh_opts)"
# SC2029: expansion on client side is intentional
# shellcheck disable=SC2029
ssh "${_opts[@]}" "root@${ip}" "$@" 2>/dev/null
}
# status_one <vm-ip> <vm-name> <agent-index>
status_one() {
local ip="$1"
local name="$2"
local idx="$3"
local agent_user="agent-${idx}"
local agent_home="/home/${agent_user}"
local assignment_file="${agent_home}/assignment.json"
local status_file="${agent_home}/status"
local run_log="${agent_home}/run.log"
# --- assignment ---
local issue_number="" issue_title="" assigned_at=""
if ssh_root_quiet "${ip}" "test -f '${assignment_file}'"; then
local raw_assignment
raw_assignment=$(ssh_root_quiet "${ip}" "cat '${assignment_file}'" || echo "{}")
issue_number=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_number',''))" 2>/dev/null || true)
issue_title=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_title',''))" 2>/dev/null || true)
assigned_at=$(echo "${raw_assignment}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('assigned_at',''))" 2>/dev/null || true)
fi
# --- tmux / status ---
local agent_state="no-session"
if ssh_root_quiet "${ip}" "su - ${agent_user} -c 'tmux has-session -t ${agent_user} 2>/dev/null'"; then
agent_state="running"
elif ssh_root_quiet "${ip}" "test -f '${status_file}'"; then
agent_state=$(ssh_root_quiet "${ip}" "cat '${status_file}'" 2>/dev/null | tr -d '[:space:]' | tr '[:upper:]' '[:lower:]' || echo "unknown")
fi
# --- last log line ---
local last_log_line=""
if ssh_root_quiet "${ip}" "test -f '${run_log}'"; then
last_log_line=$(ssh_root_quiet "${ip}" "tail -1 '${run_log}'" 2>/dev/null | tr -d '\r' || true)
fi
# --- PR URL ---
local pr_url=""
if ssh_root_quiet "${ip}" "test -f '${run_log}'"; then
pr_url=$(ssh_root_quiet "${ip}" "grep -oE 'https://github.com/[^/]+/[^/]+/pull/[0-9]+' '${run_log}' | tail -1" 2>/dev/null || true)
fi
# --- duration ---
local duration=""
if [[ -n "${assigned_at}" ]]; then
duration=$(python3 - "${assigned_at}" <<'PYEOF'
import sys
from datetime import datetime, timezone
try:
assigned = datetime.fromisoformat(sys.argv[1].replace("Z", "+00:00"))
now = datetime.now(timezone.utc)
total = int((now - assigned).total_seconds())
h, rem = divmod(total, 3600)
m = rem // 60
if h > 0:
print(f"{h}h {m}m")
else:
print(f"{m}m")
except Exception:
print("")
PYEOF
)
fi
# --- output ---
printf "VM: %-24s | Agent: %-8s | Status: %-14s | Issue: %s\n" \
"${name}" "${agent_user}" "${agent_state}" "${issue_number:-(none)}"
if [[ -n "${issue_title}" ]]; then
printf " Title: %s\n" "${issue_title}"
fi
if [[ -n "${last_log_line}" ]]; then
printf " Last log: %s\n" "${last_log_line}"
fi
if [[ -n "${pr_url}" ]]; then
printf " PR: %s\n" "${pr_url}"
fi
if [[ -n "${duration}" ]]; then
printf " Duration: %s\n" "${duration}"
fi
echo ""
}
# ---------------------------------------------------------------------------
# Single agent
# ---------------------------------------------------------------------------
if [[ "${STATUS_ALL}" == "false" ]]; then
# Resolve VM name from IP (best effort)
VM_NAME="${VM_IP}"
if command -v hcloud &>/dev/null && [[ -n "${HCLOUD_TOKEN:-}" ]]; then
VM_NAME=$(hcloud server list --output columns=name,ipv4 2>/dev/null \
| awk -v ip="${VM_IP}" '$2==ip {print $1}' || echo "${VM_IP}")
fi
status_one "${VM_IP}" "${VM_NAME}" "${AGENT_INDEX}"
exit 0
fi
# ---------------------------------------------------------------------------
# All agents across all running VMs
# ---------------------------------------------------------------------------
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status \
| awk '$2=="running" && /ccgm-agent/ {print $1}' \
| sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
log_warn "No running ccgm-agent-* VMs found."
exit 0
fi
for vm_name in "${VM_NAMES[@]}"; do
ip=$(hcloud server describe "${vm_name}" --output format='{{.PublicNet.IPv4.IP}}')
for idx in $(seq 0 $(( CCGM_AGENTS_PER_VM - 1 ))); do
status_one "${ip}" "${vm_name}" "${idx}"
done
done
lib/agent-stop.sh
#!/usr/bin/env bash
# agent-stop.sh — Stop one or all running Claude Code agent tmux sessions.
#
# Usage:
# agent-stop.sh --all
# agent-stop.sh <vm-ip> <agent-index>
#
# For each target agent:
# 1. Kills the tmux session named agent-N on the VM
# 2. Writes AGENT_STOPPED to ~/status
#
# Requires:
# - hcloud CLI (when --all is used)
# - SSH key in ~/.ssh/ccgm-dispatch-session or $SSH_KEY_PATH
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --all" >&2
echo " $0 <vm-ip> <agent-index>" >&2
exit 1
fi
STOP_ALL=false
VM_IP=""
AGENT_INDEX=""
while [[ $# -gt 0 ]]; do
case "$1" in
--all)
STOP_ALL=true
shift
;;
-*)
log_error "Unknown option: $1"
exit 1
;;
*)
if [[ -z "${VM_IP}" ]]; then
VM_IP="$1"
elif [[ -z "${AGENT_INDEX}" ]]; then
AGENT_INDEX="$1"
else
log_error "Unexpected argument: $1"
exit 1
fi
shift
;;
esac
done
if [[ "${STOP_ALL}" == "false" ]]; then
if [[ -z "${VM_IP}" || -z "${AGENT_INDEX}" ]]; then
log_error "Provide --all or both <vm-ip> and <agent-index>"
exit 1
fi
if ! [[ "${AGENT_INDEX}" =~ ^[0-3]$ ]]; then
log_error "agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})"
exit 1
fi
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
if [[ "${STOP_ALL}" == "true" ]]; then
require_cmd hcloud
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set."
exit 1
fi
fi
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
ssh_root_quiet() {
local ip="$1"; shift
# shellcheck disable=SC2206
read -ra _opts <<< "$(ssh_opts)"
# SC2029: expansion on client side is intentional
# shellcheck disable=SC2029
ssh "${_opts[@]}" "root@${ip}" "$@" 2>/dev/null
}
stop_one() {
local ip="$1"
local idx="$2"
local agent_user="agent-${idx}"
local agent_home="/home/${agent_user}"
local status_file="${agent_home}/status"
log_info "Stopping ${agent_user} on ${ip}"
# Kill the tmux session (ignore errors if not running)
ssh_root_quiet "${ip}" "su - ${agent_user} -c 'tmux kill-session -t ${agent_user} 2>/dev/null || true'"
# Write stopped status
ssh_root_quiet "${ip}" "echo AGENT_STOPPED > '${status_file}' && chown '${agent_user}:${agent_user}' '${status_file}'"
log_success "Stopped ${agent_user} on ${ip}"
}
# ---------------------------------------------------------------------------
# Single agent
# ---------------------------------------------------------------------------
if [[ "${STOP_ALL}" == "false" ]]; then
stop_one "${VM_IP}" "${AGENT_INDEX}"
exit 0
fi
# ---------------------------------------------------------------------------
# All agents across all running VMs
# ---------------------------------------------------------------------------
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status \
| awk '$2=="running" && /ccgm-agent/ {print $1}' \
| sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
log_warn "No running ccgm-agent-* VMs found."
exit 0
fi
STOPPED=0
for vm_name in "${VM_NAMES[@]}"; do
ip=$(hcloud server describe "${vm_name}" --output format='{{.PublicNet.IPv4.IP}}')
for idx in $(seq 0 $(( CCGM_AGENTS_PER_VM - 1 ))); do
stop_one "${ip}" "${idx}"
STOPPED=$(( STOPPED + 1 ))
done
done
log_info "Stopped ${STOPPED} agent session(s)."
lib/auto-shutdown.sh
#!/usr/bin/env bash
# auto-shutdown.sh — Auto-shutdown script that runs ON the VM (not the orchestrator).
#
# Installed as a cron job on the VM (every 5 minutes):
# */5 * * * * /opt/ccgm/auto-shutdown.sh >> /var/log/ccgm-shutdown.log 2>&1
#
# Behavior:
# 1. If any agent tmux sessions are still active, record the time and exit.
# 2. If no sessions have been active for 15+ minutes, initiate shutdown.
# 3. If MAX_HOURS wall-clock time has been exceeded, kill all agents and shut down.
#
# Environment variables (read from /etc/ccgm/env if present):
# MAX_HOURS Maximum wall-clock hours before forced shutdown (default: 8)
# IDLE_MINUTES Minutes of no active sessions before idle shutdown (default: 15)
#
# State files (on the VM):
# /var/lib/ccgm/last-active Timestamp of last observed active session (epoch seconds)
# /var/lib/ccgm/vm-start Timestamp of VM start / script first run (epoch seconds)
# /var/log/ccgm-shutdown.log Shutdown reason and timing (written by this script)
#
# Install in the golden image or via workspace-setup.sh:
# install -m 0755 auto-shutdown.sh /opt/ccgm/auto-shutdown.sh
set -euo pipefail
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
MAX_HOURS="${MAX_HOURS:-8}"
IDLE_MINUTES="${IDLE_MINUTES:-15}"
STATE_DIR="/var/lib/ccgm"
LAST_ACTIVE_FILE="${STATE_DIR}/last-active"
VM_START_FILE="${STATE_DIR}/vm-start"
SHUTDOWN_LOG="/var/log/ccgm-shutdown.log"
ENV_FILE="/etc/ccgm/env"
# ---------------------------------------------------------------------------
# Logging (to shutdown log only - stdout is redirected there by cron)
# ---------------------------------------------------------------------------
log() {
echo "$(date '+%Y-%m-%dT%H:%M:%S') [auto-shutdown] $*"
}
# ---------------------------------------------------------------------------
# Load optional env overrides
# ---------------------------------------------------------------------------
if [[ -f "${ENV_FILE}" ]]; then
# shellcheck source=/dev/null
source "${ENV_FILE}"
fi
# ---------------------------------------------------------------------------
# Ensure state directory exists
# ---------------------------------------------------------------------------
mkdir -p "${STATE_DIR}"
NOW=$(date +%s)
# ---------------------------------------------------------------------------
# Record VM start time on first run
# ---------------------------------------------------------------------------
if [[ ! -f "${VM_START_FILE}" ]]; then
echo "${NOW}" > "${VM_START_FILE}"
log "VM start time recorded: ${NOW}"
fi
VM_START=$(cat "${VM_START_FILE}")
WALL_CLOCK_SECS=$(( NOW - VM_START ))
WALL_CLOCK_HOURS=$(( WALL_CLOCK_SECS / 3600 ))
# ---------------------------------------------------------------------------
# Check wall-clock cap first (hard limit)
# ---------------------------------------------------------------------------
MAX_SECS=$(( MAX_HOURS * 3600 ))
if [[ "${WALL_CLOCK_SECS}" -ge "${MAX_SECS}" ]]; then
log "SHUTDOWN: wall-clock limit reached (${WALL_CLOCK_HOURS}h >= ${MAX_HOURS}h limit)"
log "Killing all agent tmux sessions..."
# Kill all agent-N tmux sessions
for i in 0 1 2 3; do
agent_user="agent-${i}"
if id "${agent_user}" &>/dev/null; then
su - "${agent_user}" -c "tmux kill-server 2>/dev/null || true" 2>/dev/null || true
echo "AGENT_TIMEOUT" > "/home/${agent_user}/status" || true
chown "${agent_user}:${agent_user}" "/home/${agent_user}/status" 2>/dev/null || true
fi
done
log "SHUTDOWN: initiating poweroff (reason: max-hours)"
echo "shutdown" > "${SHUTDOWN_LOG}.reason"
/sbin/poweroff
exit 0
fi
# ---------------------------------------------------------------------------
# Check for active tmux sessions across all agent users
# ---------------------------------------------------------------------------
ACTIVE_SESSIONS=0
for i in 0 1 2 3; do
agent_user="agent-${i}"
if id "${agent_user}" &>/dev/null; then
if su - "${agent_user}" -c "tmux has-session 2>/dev/null"; then
ACTIVE_SESSIONS=$(( ACTIVE_SESSIONS + 1 ))
fi
fi
done
# ---------------------------------------------------------------------------
# Update last-active or check idle timeout
# ---------------------------------------------------------------------------
if [[ "${ACTIVE_SESSIONS}" -gt 0 ]]; then
echo "${NOW}" > "${LAST_ACTIVE_FILE}"
log "Active sessions: ${ACTIVE_SESSIONS} (wall-clock: ${WALL_CLOCK_HOURS}h, max: ${MAX_HOURS}h)"
exit 0
fi
# No active sessions - check idle duration
if [[ -f "${LAST_ACTIVE_FILE}" ]]; then
LAST_ACTIVE=$(cat "${LAST_ACTIVE_FILE}")
else
# Never been active - use VM start as baseline
LAST_ACTIVE="${VM_START}"
echo "${LAST_ACTIVE}" > "${LAST_ACTIVE_FILE}"
fi
IDLE_SECS=$(( NOW - LAST_ACTIVE ))
IDLE_MINS=$(( IDLE_SECS / 60 ))
IDLE_THRESHOLD_SECS=$(( IDLE_MINUTES * 60 ))
log "No active sessions. Idle for ${IDLE_MINS}m (threshold: ${IDLE_MINUTES}m, wall-clock: ${WALL_CLOCK_HOURS}h)"
if [[ "${IDLE_SECS}" -ge "${IDLE_THRESHOLD_SECS}" ]]; then
log "SHUTDOWN: idle threshold reached (${IDLE_MINS}m >= ${IDLE_MINUTES}m)"
log "SHUTDOWN: initiating poweroff (reason: idle-timeout)"
echo "idle-timeout" > "${SHUTDOWN_LOG}.reason"
/sbin/poweroff
exit 0
fi
log "Idle timer running: ${IDLE_MINS}m / ${IDLE_MINUTES}m until shutdown"
lib/budget-track.sh
#!/usr/bin/env bash
# budget-track.sh — Track VM costs for CCGM cloud-dispatch sessions.
#
# Usage:
# budget-track.sh start
# budget-track.sh stop
# budget-track.sh status
# budget-track.sh report
#
# Subcommands:
# start Record session start time and active VM details to BUDGET_FILE.
# stop Finalize session cost and append to MONTHLY_FILE.
# status Print current session runtime and running cost estimate.
# report Print full cost report for the current month.
#
# State files:
# /tmp/ccgm-budget.json Current session data
# /tmp/ccgm-budget-monthly.json Monthly session log (append-only array)
#
# CANONICAL HOURLY RATES (USD, approximate EUR->USD at 1.08).
# This HOURLY_RATES table is the single source of truth for VM pricing in
# cloud-dispatch. README.md and commands/dispatch.md must match these figures.
# ccx63: $0.58/hr (48 vCPU, 192 GB) dedicated
# ccx43: $0.28/hr (16 vCPU, 64 GB) dedicated
# ccx33: $0.20/hr ( 8 vCPU, 32 GB) dedicated
# cx42: $0.022/hr ( 8 vCPU, 16 GB) shared
# cx32: $0.011/hr ( 4 vCPU, 8 GB) shared
# cx22: $0.006/hr ( 2 vCPU, 4 GB) shared
#
# Requires: jq, hcloud (for start subcommand)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
BUDGET_FILE="${CCGM_BUDGET_FILE:-/tmp/ccgm-budget.json}"
MONTHLY_FILE="${CCGM_MONTHLY_FILE:-/tmp/ccgm-budget-monthly.json}"
MONTHLY_BUDGET_USD="${CCGM_MONTHLY_BUDGET:-2000}"
CLAUDE_MAX_USD="${CCGM_CLAUDE_MAX_USD:-200}"
# Hourly rates by server type (USD). Canonical source of truth for the module.
declare -A HOURLY_RATES=(
[ccx63]="0.58"
[ccx43]="0.28"
[ccx33]="0.20"
[cx42]="0.022"
[cx32]="0.011"
[cx22]="0.006"
)
DEFAULT_RATE="0.58"
# ---------------------------------------------------------------------------
# Usage
# ---------------------------------------------------------------------------
usage() {
echo "Usage: $0 <start|stop|status|report>" >&2
exit 1
}
if [[ $# -lt 1 ]]; then
usage
fi
SUBCOMMAND="$1"
shift
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
require_cmd jq
# hourly_rate <server-type> — print the USD/hr rate for a given server type
hourly_rate() {
local stype
# printf '%s', not echo: bash's builtin echo flag-parses a leading -n/-e,
# so a server-type of exactly "-n" would silently normalize to empty and
# fall through to DEFAULT_RATE for the wrong reason.
stype=$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')
echo "${HOURLY_RATES[$stype]:-${DEFAULT_RATE}}"
}
# now_iso — current UTC time in ISO 8601 format
now_iso() {
date -u +"%Y-%m-%dT%H:%M:%SZ"
}
# now_epoch — current time as Unix epoch seconds
now_epoch() {
date +%s
}
# iso_to_epoch <iso-string> — convert ISO 8601 UTC string to epoch seconds
iso_to_epoch() {
python3 -c "
from datetime import datetime, timezone
s = '$1'.replace('Z', '+00:00')
try:
dt = datetime.fromisoformat(s)
print(int(dt.timestamp()))
except Exception:
print(0)
"
}
# elapsed_human <start_epoch> — return human-readable elapsed time
elapsed_human() {
local start="$1"
local now
now=$(now_epoch)
local total=$(( now - start ))
local h=$(( total / 3600 ))
local m=$(( (total % 3600) / 60 ))
if [[ ${h} -gt 0 ]]; then
echo "${h}h ${m}m"
else
echo "${m}m"
fi
}
# session_id — generate a session ID from current timestamp
session_id() {
date -u +"%Y%m%d-%H%M%S"
}
# current_month — YYYY-MM
current_month() {
date -u +"%Y-%m"
}
# month_label — "Month YYYY" format for display
month_label() {
date -u +"%B %Y"
}
# ---------------------------------------------------------------------------
# Subcommand: start
# ---------------------------------------------------------------------------
cmd_start() {
require_cmd hcloud
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Cannot list VMs."
exit 1
fi
log_info "Scanning for running ccgm-agent-* VMs..."
# Collect VM details via hcloud
local vm_json
vm_json=$(hcloud server list --output json 2>/dev/null \
| python3 -c "
import json, sys
servers = json.load(sys.stdin)
result = []
for s in servers:
name = s.get('name', '')
if not name.startswith('ccgm-agent'):
continue
if s.get('status', '') != 'running':
continue
stype = ''
st = s.get('server_type', {})
if st:
stype = st.get('name', '')
created = s.get('created', '')
result.append({'name': name, 'type': stype, 'started_at': created})
print(json.dumps(result))
" 2>/dev/null || echo "[]")
local sid
sid=$(session_id)
local ts
ts=$(now_iso)
# Build per-VM entries with hourly rates using jq
local rate_map_json
rate_map_json=$(jq -n \
--argjson ccx63 "${HOURLY_RATES[ccx63]:-${DEFAULT_RATE}}" \
--argjson ccx43 "${HOURLY_RATES[ccx43]:-${DEFAULT_RATE}}" \
--argjson ccx33 "${HOURLY_RATES[ccx33]:-${DEFAULT_RATE}}" \
--argjson cx42 "${HOURLY_RATES[cx42]:-${DEFAULT_RATE}}" \
--argjson cx32 "${HOURLY_RATES[cx32]:-${DEFAULT_RATE}}" \
--argjson cx22 "${HOURLY_RATES[cx22]:-${DEFAULT_RATE}}" \
'{ccx63: $ccx63, ccx43: $ccx43, ccx33: $ccx33, cx42: $cx42, cx32: $cx32, cx22: $cx22}')
local default_rate="${DEFAULT_RATE}"
local vms_json
vms_json=$(echo "${vm_json}" | jq \
--argjson rates "${rate_map_json}" \
--argjson default_rate "${default_rate}" \
--arg fallback_ts "${ts}" \
'[.[] | {
name: .name,
type: (.type | ascii_downcase),
hourly_rate: ($rates[(.type | ascii_downcase)] // $default_rate),
started_at: (if .started_at == "" or .started_at == null then $fallback_ts else .started_at end)
}]' 2>/dev/null || echo "[]")
jq -n \
--arg sid "${sid}" \
--arg ts "${ts}" \
--argjson vms "${vms_json}" \
'{
session_id: $sid,
started_at: $ts,
vms: $vms,
vm_hours_total: 0,
estimated_cost_usd: 0
}' > "${BUDGET_FILE}"
local vm_count
vm_count=$(jq '.vms | length' "${BUDGET_FILE}")
log_success "Session ${sid} started. Tracking ${vm_count} VM(s)."
echo " Budget file: ${BUDGET_FILE}"
}
# ---------------------------------------------------------------------------
# Subcommand: stop
# ---------------------------------------------------------------------------
cmd_stop() {
if [[ ! -f "${BUDGET_FILE}" ]]; then
log_error "No active session found at ${BUDGET_FILE}. Run 'start' first."
exit 1
fi
local now_ts
now_ts=$(now_iso)
# Calculate total VM-hours and cost
local total_hours total_cost
total_hours=$(jq --arg now "${now_ts}" '
[.vms[] |
(($now | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime) -
(.started_at | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime)) / 3600
] | add // 0
' "${BUDGET_FILE}" 2>/dev/null || echo "0")
total_cost=$(jq --arg now "${now_ts}" '
[.vms[] |
((($now | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime) -
(.started_at | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime)) / 3600) * .hourly_rate
] | add // 0
' "${BUDGET_FILE}" 2>/dev/null || echo "0")
# Update session file with final values
local session_data
session_data=$(jq \
--arg stopped "${now_ts}" \
--argjson hours "${total_hours}" \
--argjson cost "${total_cost}" \
'. + {stopped_at: $stopped, vm_hours_total: $hours, estimated_cost_usd: $cost}' \
"${BUDGET_FILE}")
# Append to monthly log
local month
month=$(current_month)
if [[ ! -f "${MONTHLY_FILE}" ]]; then
echo '{"sessions": []}' > "${MONTHLY_FILE}"
fi
jq --argjson entry "${session_data}" --arg month "${month}" \
'.sessions += [$entry] | .last_updated = now | .month = $month' \
"${MONTHLY_FILE}" > "${MONTHLY_FILE}.tmp" && mv "${MONTHLY_FILE}.tmp" "${MONTHLY_FILE}"
# Print session summary
local sid
sid=$(jq -r '.session_id' "${BUDGET_FILE}")
printf "\nSession %s complete.\n" "${sid}"
printf " VM-hours: %.2f\n" "${total_hours}"
printf " Cost: \$%.2f\n" "${total_cost}"
printf " Saved to: %s\n\n" "${MONTHLY_FILE}"
# Clean up session file
rm -f "${BUDGET_FILE}"
}
# ---------------------------------------------------------------------------
# Subcommand: status
# ---------------------------------------------------------------------------
cmd_status() {
if [[ ! -f "${BUDGET_FILE}" ]]; then
log_warn "No active session. Run 'start' to begin tracking."
exit 0
fi
local started_at
started_at=$(jq -r '.started_at' "${BUDGET_FILE}")
local start_ep
start_ep=$(iso_to_epoch "${started_at}")
local elapsed
elapsed=$(elapsed_human "${start_ep}")
local vm_count
vm_count=$(jq '.vms | length' "${BUDGET_FILE}")
# Running cost: sum of (elapsed_hours * hourly_rate) per VM
local now_ts
now_ts=$(now_iso)
local running_cost
running_cost=$(jq --arg now "${now_ts}" '
[.vms[] |
((($now | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime) -
(.started_at | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime)) / 3600) * .hourly_rate
] | add // 0
' "${BUDGET_FILE}" 2>/dev/null || echo "0")
local hourly_rate_total
hourly_rate_total=$(jq '[.vms[].hourly_rate] | add // 0' "${BUDGET_FILE}")
# Monthly stats
local monthly_total="0"
if [[ -f "${MONTHLY_FILE}" ]]; then
monthly_total=$(jq '[.sessions[].estimated_cost_usd // 0] | add // 0' "${MONTHLY_FILE}" 2>/dev/null || echo "0")
fi
# Session-hours so far (for monthly projection)
local month_seconds_elapsed
month_seconds_elapsed=$(python3 -c "
from datetime import datetime, timezone
now = datetime.now(timezone.utc)
import calendar
days_in_month = calendar.monthrange(now.year, now.month)[1]
day = now.day + now.hour/24.0
print(int(day / days_in_month * 86400 * days_in_month))
" 2>/dev/null || echo "1")
local month_seconds_total
month_seconds_total=$(python3 -c "
from datetime import datetime, timezone
import calendar
now = datetime.now(timezone.utc)
days_in_month = calendar.monthrange(now.year, now.month)[1]
print(days_in_month * 86400)
" 2>/dev/null || echo "2592000")
local grand_total
grand_total=$(python3 -c "print(round(${monthly_total} + ${running_cost}, 2))" 2>/dev/null || echo "${monthly_total}")
local projected
projected=$(python3 -c "
elapsed = ${month_seconds_elapsed}
total = ${month_seconds_total}
spent = ${grand_total}
if elapsed > 0:
print(round(spent / elapsed * total, 2))
else:
print(0)
" 2>/dev/null || echo "0")
printf "Session: %s | VMs: %s | Rate: \$%.2f/hr | Session cost: \$%.2f\n" \
"${elapsed}" "${vm_count}" "${hourly_rate_total}" "${running_cost}"
printf "Monthly total: \$%.2f (est. \$%.2f/mo at current pace)\n" \
"${grand_total}" "${projected}"
}
# ---------------------------------------------------------------------------
# Subcommand: report
# ---------------------------------------------------------------------------
cmd_report() {
local month_label
month_label=$(month_label)
echo "=== CCGM Cloud Dispatch - Cost Report ==="
echo "Month: ${month_label}"
if [[ ! -f "${MONTHLY_FILE}" ]]; then
echo "No session data found for this month."
exit 0
fi
# Session stats
local session_count total_vm_hours total_vm_cost issues_completed
session_count=$(jq '.sessions | length' "${MONTHLY_FILE}")
total_vm_hours=$(jq '[.sessions[].vm_hours_total // 0] | add // 0' "${MONTHLY_FILE}")
total_vm_cost=$(jq '[.sessions[].estimated_cost_usd // 0] | add // 0' "${MONTHLY_FILE}")
# Count issues (vms * agents * sessions as a proxy - sum vm counts per session)
issues_completed=$(jq '[.sessions[] | (.vms | length)] | add // 0' "${MONTHLY_FILE}")
local total_cost
total_cost=$(python3 -c "print(round(${total_vm_cost} + ${CLAUDE_MAX_USD}, 2))" 2>/dev/null || echo "${total_vm_cost}")
local budget_remaining
budget_remaining=$(python3 -c "print(round(${MONTHLY_BUDGET_USD} - ${total_cost}, 2))" 2>/dev/null || echo "0")
# Projection
local month_seconds_total
month_seconds_total=$(python3 -c "
from datetime import datetime, timezone
import calendar
now = datetime.now(timezone.utc)
days_in_month = calendar.monthrange(now.year, now.month)[1]
print(days_in_month * 86400)
" 2>/dev/null || echo "2592000")
local month_seconds_elapsed
month_seconds_elapsed=$(python3 -c "
from datetime import datetime, timezone
import calendar
now = datetime.now(timezone.utc)
days_in_month = calendar.monthrange(now.year, now.month)[1]
elapsed = (now.day - 1) * 86400 + now.hour * 3600 + now.minute * 60 + now.second
print(max(1, elapsed))
" 2>/dev/null || echo "86400")
local projected
projected=$(python3 -c "
elapsed = ${month_seconds_elapsed}
total = ${month_seconds_total}
spent = ${total_cost}
print(round(spent / elapsed * total, 2))
" 2>/dev/null || echo "0")
printf "Sessions: %s\n" "${session_count}"
printf "Total VM-hours: %.1f\n" "${total_vm_hours}"
printf "Total VM cost: \$%.2f\n" "${total_vm_cost}"
printf "Claude Max subscription:\$%.2f\n" "${CLAUDE_MAX_USD}"
printf "Estimated total: \$%.2f\n" "${total_cost}"
printf "Budget remaining: \$%.2f (of \$%s)\n" "${budget_remaining}" "${MONTHLY_BUDGET_USD}"
printf "Projected month-end: \$%.2f\n" "${projected}"
# Cost per issue (if any completed)
if [[ "${issues_completed}" -gt 0 && $(python3 -c "print(1 if ${total_cost} > 0 else 0)") == "1" ]]; then
local cost_per_issue
cost_per_issue=$(python3 -c "print(round(${total_vm_cost} / ${issues_completed}, 2))" 2>/dev/null || echo "N/A")
printf "Cost per issue (VM): \$%s\n" "${cost_per_issue}"
fi
}
# ---------------------------------------------------------------------------
# Dispatch
# ---------------------------------------------------------------------------
case "${SUBCOMMAND}" in
start) cmd_start ;;
stop) cmd_stop ;;
status) cmd_status ;;
report) cmd_report ;;
*)
log_error "Unknown subcommand: ${SUBCOMMAND}"
usage
;;
esac
lib/ccgm-headless-install.sh
#!/usr/bin/env bash
# ccgm-headless-install.sh — Non-interactive CCGM installer for cloud agent users.
#
# Installs a CCGM preset into a target user's ~/.claude/ directory by copying
# rule and command files from the CCGM repo. No interactive prompts, no template
# expansion (cloud agents don't need user-specific config).
#
# Usage:
# ccgm-headless-install.sh <ccgm-repo-path> <preset-name> <target-user-home>
#
# Example:
# ccgm-headless-install.sh /opt/ccgm/repo cloud-agent /home/agent-0
#
# This script is designed to run ON the VM (not from the orchestrator).
set -euo pipefail
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <ccgm-repo-path> <preset-name> <target-user-home>" >&2
exit 1
fi
CCGM_REPO="$1"
PRESET_NAME="$2"
TARGET_HOME="$3"
CLAUDE_DIR="${TARGET_HOME}/.claude"
PRESET_FILE="${CCGM_REPO}/presets/${PRESET_NAME}.json"
if [[ ! -f "${PRESET_FILE}" ]]; then
echo "Error: preset not found: ${PRESET_FILE}" >&2
exit 1
fi
if [[ ! -d "${CCGM_REPO}/modules" ]]; then
echo "Error: CCGM modules directory not found: ${CCGM_REPO}/modules" >&2
exit 1
fi
# Parse preset JSON to get module list
if ! command -v jq &>/dev/null; then
echo "Error: jq is required but not installed" >&2
exit 1
fi
mapfile -t MODULES < <(jq -r '.[]' "${PRESET_FILE}")
echo "==> Installing CCGM preset '${PRESET_NAME}' to ${CLAUDE_DIR}"
echo " Modules: ${MODULES[*]}"
# Create target directories
mkdir -p "${CLAUDE_DIR}/rules"
mkdir -p "${CLAUDE_DIR}/commands"
INSTALLED_RULES=0
INSTALLED_COMMANDS=0
for module in "${MODULES[@]}"; do
MODULE_DIR="${CCGM_REPO}/modules/${module}"
MANIFEST="${MODULE_DIR}/module.json"
if [[ ! -f "${MANIFEST}" ]]; then
echo " WARN: module '${module}' not found, skipping"
continue
fi
echo " --> Installing module: ${module}"
# Extract file mappings from module.json
# Each file entry has: source path (key), target path, and type
while IFS=$'\t' read -r src target ftype; do
SRC_PATH="${MODULE_DIR}/${src}"
if [[ ! -f "${SRC_PATH}" ]]; then
echo " WARN: source file not found: ${SRC_PATH}"
continue
fi
case "${ftype}" in
rule)
cp "${SRC_PATH}" "${CLAUDE_DIR}/${target}"
INSTALLED_RULES=$((INSTALLED_RULES + 1))
;;
command)
cp "${SRC_PATH}" "${CLAUDE_DIR}/${target}"
INSTALLED_COMMANDS=$((INSTALLED_COMMANDS + 1))
;;
# Skip lib, hook, settings, and other non-rule/command types
# Cloud agents get lib scripts via the cloud-dispatch module's own paths
*)
;;
esac
done < <(jq -r '.files | to_entries[] | [.key, .value.target, .value.type] | @tsv' "${MANIFEST}")
done
echo "==> CCGM headless install complete"
echo " Rules installed: ${INSTALLED_RULES}"
echo " Commands installed: ${INSTALLED_COMMANDS}"
echo " Target: ${CLAUDE_DIR}"
lib/cost-report.sh
#!/usr/bin/env bash
# cost-report.sh — Generate cost reports for CCGM cloud-dispatch sessions.
#
# Usage:
# cost-report.sh session [--json]
# cost-report.sh monthly [--json]
#
# Subcommands:
# session Report for the current or last completed session.
# monthly Report for the current calendar month.
#
# Flags:
# --json Output machine-readable JSON instead of formatted text.
#
# Reads from:
# /tmp/ccgm-budget.json Current or last session data
# /tmp/ccgm-budget-monthly.json Monthly session log
#
# Requires: jq
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
BUDGET_FILE="${CCGM_BUDGET_FILE:-/tmp/ccgm-budget.json}"
MONTHLY_FILE="${CCGM_MONTHLY_FILE:-/tmp/ccgm-budget-monthly.json}"
MONTHLY_BUDGET_USD="${CCGM_MONTHLY_BUDGET:-2000}"
CLAUDE_MAX_USD="${CCGM_CLAUDE_MAX_USD:-200}"
# ---------------------------------------------------------------------------
# Usage
# ---------------------------------------------------------------------------
usage() {
echo "Usage: $0 <session|monthly> [--json]" >&2
exit 1
}
if [[ $# -lt 1 ]]; then
usage
fi
SUBCOMMAND="$1"
shift
JSON_OUTPUT=false
while [[ $# -gt 0 ]]; do
case "$1" in
--json)
JSON_OUTPUT=true
shift
;;
*)
log_error "Unknown argument: $1"
usage
;;
esac
done
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd jq
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
# month_label — "Month YYYY" format for display
month_label() {
date -u +"%B %Y"
}
# current_month — YYYY-MM
current_month() {
date -u +"%Y-%m"
}
# safe_jq <filter> <file> <default> — run jq filter and return default on failure
safe_jq() {
local filter="$1"
local file="$2"
local default="${3:-0}"
jq -r "${filter}" "${file}" 2>/dev/null || echo "${default}"
}
# py_calc <expr> — evaluate a Python arithmetic expression and print result
py_calc() {
python3 -c "print(round(${1}, 2))" 2>/dev/null || echo "0"
}
# duration_label <hours_float> — convert decimal hours to human-readable string
duration_label() {
python3 -c "
h = float('${1}')
hours = int(h)
mins = int((h - hours) * 60)
if hours > 0:
print(f'{hours}h {mins}m')
else:
print(f'{mins}m')
" 2>/dev/null || echo "${1}h"
}
# ---------------------------------------------------------------------------
# Compute session-level stats from a session JSON object
# Returns a JSON object with derived fields
# ---------------------------------------------------------------------------
session_stats() {
local session_json="$1"
# Extract base fields
local sid started_at stopped_at vm_count vm_hours_total cost
sid=$(echo "${session_json}" | jq -r '.session_id // "unknown"')
started_at=$(echo "${session_json}" | jq -r '.started_at // ""')
stopped_at=$(echo "${session_json}" | jq -r '.stopped_at // ""')
vm_count=$(echo "${session_json}" | jq '.vms | length')
vm_hours_total=$(echo "${session_json}" | jq '.vm_hours_total // 0')
cost=$(echo "${session_json}" | jq '.estimated_cost_usd // 0')
# Compute wall-clock duration
local duration_hours="0"
if [[ -n "${started_at}" && -n "${stopped_at}" ]]; then
duration_hours=$(python3 -c "
from datetime import datetime, timezone
def parse(s):
return datetime.fromisoformat(s.replace('Z', '+00:00'))
try:
s = parse('${started_at}')
e = parse('${stopped_at}')
print(round((e - s).total_seconds() / 3600, 3))
except Exception:
print(0)
" 2>/dev/null || echo "0")
fi
# VM breakdown
local vm_breakdown
vm_breakdown=$(echo "${session_json}" | jq -c '[.vms[] | {name: .name, type: .type, hourly_rate: .hourly_rate}]')
jq -n \
--arg sid "${sid}" \
--arg started_at "${started_at}" \
--arg stopped_at "${stopped_at}" \
--argjson vm_count "${vm_count}" \
--argjson vm_hours "${vm_hours_total}" \
--argjson cost "${cost}" \
--argjson duration "${duration_hours}" \
--argjson vms "${vm_breakdown}" \
'{
session_id: $sid,
started_at: $started_at,
stopped_at: $stopped_at,
vm_count: $vm_count,
vm_hours_total: $vm_hours,
estimated_cost_usd: $cost,
duration_hours: $duration,
vms: $vms
}'
}
# ---------------------------------------------------------------------------
# Subcommand: session
# ---------------------------------------------------------------------------
cmd_session() {
local session_json
if [[ -f "${BUDGET_FILE}" ]]; then
# Active session
session_json=$(cat "${BUDGET_FILE}")
local is_active=true
# For active sessions, compute running cost from current time
local now_ts
now_ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
local running_cost
running_cost=$(echo "${session_json}" | jq --arg now "${now_ts}" '
[.vms[] |
((($now | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime) -
(.started_at | gsub("Z$";"") | strptime("%Y-%m-%dT%H:%M:%S") | mktime)) / 3600) * .hourly_rate
] | add // 0
' 2>/dev/null || echo "0")
session_json=$(echo "${session_json}" | jq --argjson cost "${running_cost}" '.estimated_cost_usd = $cost')
elif [[ -f "${MONTHLY_FILE}" ]]; then
# Use last completed session
session_json=$(jq '.sessions[-1]' "${MONTHLY_FILE}" 2>/dev/null || echo "null")
if [[ "${session_json}" == "null" ]]; then
log_warn "No session data found."
exit 0
fi
local is_active=false
else
log_warn "No session data found. Run 'budget-track.sh start' to begin tracking."
exit 0
fi
local stats
stats=$(session_stats "${session_json}")
if [[ "${JSON_OUTPUT}" == "true" ]]; then
echo "${stats}"
return
fi
# Human-readable output
local sid vm_count vm_hours cost duration
sid=$(echo "${stats}" | jq -r '.session_id')
vm_count=$(echo "${stats}" | jq '.vm_count')
vm_hours=$(echo "${stats}" | jq '.vm_hours_total')
cost=$(echo "${stats}" | jq '.estimated_cost_usd')
duration=$(echo "${stats}" | jq '.duration_hours')
local duration_str
duration_str=$(duration_label "${duration}")
echo "=== CCGM Cloud Dispatch - Session Report ==="
printf "Session ID: %s\n" "${sid}"
if [[ "${is_active}" == "true" ]]; then
printf "Status: ACTIVE (running)\n"
else
printf "Status: Completed\n"
fi
printf "Duration: %s\n" "${duration_str}"
printf "VMs active: %s\n" "${vm_count}"
printf "VM-hours: %.2f\n" "${vm_hours}"
printf "Session cost: \$%.2f\n" "${cost}"
# VM breakdown
local vm_entries
vm_entries=$(echo "${stats}" | jq -r '.vms[] | " \(.name) (\(.type)): \$\(.hourly_rate)/hr"')
if [[ -n "${vm_entries}" ]]; then
echo ""
echo "VM Breakdown:"
echo "${vm_entries}"
fi
}
# ---------------------------------------------------------------------------
# Subcommand: monthly
# ---------------------------------------------------------------------------
cmd_monthly() {
if [[ ! -f "${MONTHLY_FILE}" ]]; then
log_warn "No monthly data found at ${MONTHLY_FILE}."
exit 0
fi
# Aggregate session stats
local session_count avg_duration_hours total_vm_hours total_vm_cost
session_count=$(jq '.sessions | length' "${MONTHLY_FILE}")
total_vm_hours=$(jq '[.sessions[].vm_hours_total // 0] | add // 0' "${MONTHLY_FILE}")
total_vm_cost=$(jq '[.sessions[].estimated_cost_usd // 0] | add // 0' "${MONTHLY_FILE}")
if [[ "${session_count}" -gt 0 ]]; then
avg_duration_hours=$(python3 -c "print(round(${total_vm_hours} / ${session_count}, 2))" 2>/dev/null || echo "0")
else
avg_duration_hours="0"
fi
# Count total issues dispatched (sum of VM counts across sessions as proxy)
local issues_dispatched
issues_dispatched=$(jq '[.sessions[] | (.vms | length)] | add // 0' "${MONTHLY_FILE}")
# Total cost including Claude subscription
local total_cost
total_cost=$(py_calc "${total_vm_cost} + ${CLAUDE_MAX_USD}")
# Budget utilization
local budget_remaining budget_pct_used
budget_remaining=$(py_calc "${MONTHLY_BUDGET_USD} - ${total_cost}")
budget_pct_used=$(python3 -c "print(round(${total_cost} / ${MONTHLY_BUDGET_USD} * 100, 1))" 2>/dev/null || echo "0")
# Cost per issue
local cost_per_issue="N/A"
if [[ "${issues_dispatched}" -gt 0 ]]; then
cost_per_issue=$(python3 -c "print(f'\${round(${total_vm_cost} / ${issues_dispatched}, 2)}')" 2>/dev/null || echo "N/A")
fi
# Month-end projection based on days elapsed
local month_days_total month_days_elapsed projected_cost
month_days_total=$(python3 -c "
from datetime import datetime, timezone
import calendar
now = datetime.now(timezone.utc)
print(calendar.monthrange(now.year, now.month)[1])
" 2>/dev/null || echo "30")
month_days_elapsed=$(python3 -c "
from datetime import datetime, timezone
now = datetime.now(timezone.utc)
print(max(1, now.day))
" 2>/dev/null || echo "1")
projected_cost=$(python3 -c "
print(round(${total_cost} / ${month_days_elapsed} * ${month_days_total}, 2))
" 2>/dev/null || echo "0")
# Per-session type breakdown (by server type)
local type_breakdown
type_breakdown=$(jq -r '
[.sessions[].vms[]] |
group_by(.type) |
map({type: .[0].type, count: length, rate: .[0].hourly_rate}) |
.[] |
" \(.type): \(.count) VM-slot(s) @ $\(.rate)/hr"
' "${MONTHLY_FILE}" 2>/dev/null || true)
if [[ "${JSON_OUTPUT}" == "true" ]]; then
jq -n \
--arg month "$(current_month)" \
--argjson sessions "${session_count}" \
--argjson issues "${issues_dispatched}" \
--argjson vm_hours "${total_vm_hours}" \
--argjson vm_cost "${total_vm_cost}" \
--argjson claude_cost "${CLAUDE_MAX_USD}" \
--argjson total "${total_cost}" \
--argjson remaining "${budget_remaining}" \
--argjson budget "${MONTHLY_BUDGET_USD}" \
--argjson projected "${projected_cost}" \
--argjson pct "${budget_pct_used}" \
'{
month: $month,
session_count: $sessions,
issues_dispatched: $issues,
vm_hours_total: $vm_hours,
vm_cost_usd: $vm_cost,
claude_subscription_usd: $claude_cost,
total_cost_usd: $total,
budget_usd: $budget,
budget_remaining_usd: $remaining,
budget_pct_used: $pct,
projected_month_end_usd: $projected
}'
return
fi
# Human-readable output
local month_lbl
month_lbl=$(month_label)
echo "=== CCGM Cloud Dispatch - Cost Report ==="
printf "Month: %s\n" "${month_lbl}"
printf "Sessions: %s\n" "${session_count}"
printf "Issues dispatched: %s\n" "${issues_dispatched}"
printf "Total VM-hours: %.1f\n" "${total_vm_hours}"
if [[ "${session_count}" -gt 0 ]]; then
local avg_str
avg_str=$(duration_label "${avg_duration_hours}")
printf "Avg session duration: %s\n" "${avg_str}"
fi
echo ""
printf "Total VM cost: \$%.2f\n" "${total_vm_cost}"
printf "Claude Max subscription:\$%.2f\n" "${CLAUDE_MAX_USD}"
printf "Estimated total: \$%.2f\n" "${total_cost}"
echo ""
printf "Budget: \$%s\n" "${MONTHLY_BUDGET_USD}"
printf "Budget used: %.1f%%\n" "${budget_pct_used}"
printf "Budget remaining: \$%.2f\n" "${budget_remaining}"
printf "Projected month-end: \$%.2f\n" "${projected_cost}"
if [[ "${cost_per_issue}" != "N/A" ]]; then
echo ""
printf "Cost per issue (VM): %s\n" "${cost_per_issue}"
fi
if [[ -n "${type_breakdown}" ]]; then
echo ""
echo "VM Type Usage:"
echo "${type_breakdown}"
fi
# Budget warning
if python3 -c "exit(0 if ${budget_pct_used} >= 80 else 1)" 2>/dev/null; then
echo ""
log_warn "Budget utilization is ${budget_pct_used}%. Consider reviewing dispatch frequency."
fi
}
# ---------------------------------------------------------------------------
# Dispatch
# ---------------------------------------------------------------------------
case "${SUBCOMMAND}" in
session) cmd_session ;;
monthly) cmd_monthly ;;
*)
log_error "Unknown subcommand: ${SUBCOMMAND}"
usage
;;
esac
lib/recovery.sh
#!/usr/bin/env bash
# recovery.sh — Failure recovery for CCGM cloud-dispatch agents.
#
# Usage:
# recovery.sh check
# recovery.sh retry <vm-ip> <agent-index>
# recovery.sh retry-all
#
# Subcommands:
# check Scan all agents for failures and classify them.
# retry <ip> <idx> Re-dispatch a specific failed agent.
# retry-all Retry all agents currently classified as failed.
#
# Failure classifications:
# rate-limited Agent hit Claude API rate limits
# crashed Agent process died unexpectedly (tmux gone, no status file)
# timeout Auto-shutdown killed the agent (AGENT_TIMEOUT status)
# error Agent completed but with errors (AGENT_ERROR status)
# success Agent completed successfully (AGENT_DONE status, PR created)
# running Agent is still running
# unknown Cannot determine state
#
# Requires: hcloud, jq, SSH key at ~/.ssh/ccgm-dispatch-session
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
RETRY_LOG="${CCGM_RETRY_LOG:-/tmp/ccgm-recovery.json}"
# ---------------------------------------------------------------------------
# Usage
# ---------------------------------------------------------------------------
usage() {
echo "Usage: $0 check" >&2
echo " $0 retry <vm-ip> <agent-index>" >&2
echo " $0 retry-all" >&2
exit 1
}
if [[ $# -lt 1 ]]; then
usage
fi
SUBCOMMAND="$1"
shift
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd jq
# ---------------------------------------------------------------------------
# SSH helpers
# ---------------------------------------------------------------------------
ssh_root_quiet() {
local ip="$1"; shift
# shellcheck disable=SC2206
read -ra _opts <<< "$(ssh_opts)"
# SC2029: expansion is intentional - building command on client side
# shellcheck disable=SC2029
ssh "${_opts[@]}" "root@${ip}" "$@" 2>/dev/null
}
ssh_agent_quiet() {
local ip="$1"
local agent_user="$2"
local cmd="$3"
# SC2029: expansion is intentional
# shellcheck disable=SC2029
ssh_root_quiet "${ip}" "su - ${agent_user} -c $(printf '%q' "${cmd}")" 2>/dev/null || true
}
# ---------------------------------------------------------------------------
# classify_agent <vm-ip> <vm-name> <agent-index>
# Prints a JSON object describing the agent's failure state.
# ---------------------------------------------------------------------------
classify_agent() {
local ip="$1"
local vm_name="$2"
local idx="$3"
local agent_user="agent-${idx}"
local agent_home="/home/${agent_user}"
local assignment_file="${agent_home}/assignment.json"
local status_file="${agent_home}/status"
local run_log="${agent_home}/run.log"
# --- assignment ---
local issue_number="" issue_title="" repo="" branch=""
if ssh_root_quiet "${ip}" "test -f '${assignment_file}'" 2>/dev/null; then
local raw
raw=$(ssh_root_quiet "${ip}" "cat '${assignment_file}'" || echo "{}")
issue_number=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_number',''))" 2>/dev/null || true)
issue_title=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_title',''))" 2>/dev/null || true)
repo=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('repo',''))" 2>/dev/null || true)
branch=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('branch',''))" 2>/dev/null || true)
fi
if [[ -z "${issue_number}" ]]; then
# No assignment - skip this agent slot
echo "null"
return
fi
# --- tmux session state ---
local tmux_running=false
if ssh_root_quiet "${ip}" "su - ${agent_user} -c 'tmux has-session -t ${agent_user} 2>/dev/null'" 2>/dev/null; then
tmux_running=true
fi
# --- status file ---
local status_content=""
if ssh_root_quiet "${ip}" "test -f '${status_file}'" 2>/dev/null; then
status_content=$(ssh_root_quiet "${ip}" "cat '${status_file}'" 2>/dev/null | tr -d '[:space:]' || true)
fi
# --- log analysis for rate limits ---
local rate_limited=false
if ssh_root_quiet "${ip}" "test -f '${run_log}'" 2>/dev/null; then
if ssh_root_quiet "${ip}" "grep -qiE 'rate.?limit|429|too many requests|overloaded' '${run_log}'" 2>/dev/null; then
rate_limited=true
fi
fi
# --- PR existence check ---
local has_pr=false
if [[ -n "${branch}" && -n "${repo}" ]]; then
local pr_url
pr_url=$(ssh_agent_quiet "${ip}" "${agent_user}" \
"gh pr list --repo ${repo} --head ${branch} --json url --jq '.[0].url' 2>/dev/null" || true)
[[ -n "${pr_url}" ]] && has_pr=true
fi
# --- git commits check ---
local has_commits=false
if [[ -n "${repo}" ]]; then
local repo_name
repo_name=$(basename "${repo}")
local clone_dir="${agent_home}/workspace/${repo_name}"
if ssh_root_quiet "${ip}" "test -d '${clone_dir}/.git'" 2>/dev/null; then
local commit_count
commit_count=$(ssh_agent_quiet "${ip}" "${agent_user}" \
"git -C '${clone_dir}' rev-list --count HEAD ^origin/main 2>/dev/null" || echo "0")
[[ "${commit_count:-0}" -gt 0 ]] && has_commits=true
fi
fi
# --- classify ---
local classification
if [[ "${tmux_running}" == "true" ]]; then
classification="running"
elif [[ "${status_content}" == "AGENT_DONE" && "${has_pr}" == "true" ]]; then
classification="success"
elif [[ "${status_content}" == "AGENT_TIMEOUT" ]]; then
classification="timeout"
elif [[ "${rate_limited}" == "true" ]]; then
classification="rate-limited"
elif [[ "${status_content}" == "AGENT_ERROR" ]]; then
classification="error"
elif [[ "${tmux_running}" == "false" && -z "${status_content}" ]]; then
classification="crashed"
else
classification="unknown"
fi
# Output JSON
jq -n \
--arg vm "${vm_name}" \
--arg ip "${ip}" \
--arg agent "${agent_user}" \
--argjson idx "${idx}" \
--arg issue "${issue_number}" \
--arg title "${issue_title}" \
--arg repo "${repo}" \
--arg branch "${branch}" \
--arg status "${status_content}" \
--arg class "${classification}" \
--argjson has_pr "${has_pr}" \
--argjson has_commits "${has_commits}" \
'{
vm: $vm,
ip: $ip,
agent: $agent,
agent_index: $idx,
issue_number: $issue,
issue_title: $title,
repo: $repo,
branch: $branch,
status: $status,
classification: $class,
has_pr: $has_pr,
has_commits: $has_commits
}'
}
# ---------------------------------------------------------------------------
# Subcommand: check
# ---------------------------------------------------------------------------
cmd_check() {
require_cmd hcloud
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set."
exit 1
fi
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status \
| awk '$2=="running" && /ccgm-agent/ {print $1}' \
| sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
log_warn "No running ccgm-agent-* VMs found."
exit 0
fi
local results=()
local total=0
local running=0
local success=0
local failed=0
for vm_name in "${VM_NAMES[@]}"; do
local ip
ip=$(hcloud server describe "${vm_name}" --output format='{{.PublicNet.IPv4.IP}}')
for idx in $(seq 0 $(( CCGM_AGENTS_PER_VM - 1 ))); do
log_info "Checking ${vm_name} agent-${idx}..."
local result
result=$(classify_agent "${ip}" "${vm_name}" "${idx}")
if [[ "${result}" == "null" ]]; then
continue
fi
results+=("${result}")
total=$(( total + 1 ))
local class
class=$(echo "${result}" | jq -r '.classification')
case "${class}" in
running) running=$(( running + 1 )) ;;
success) success=$(( success + 1 )) ;;
*) failed=$(( failed + 1 )) ;;
esac
done
done
# Save results to retry log
printf '[%s]' "$(IFS=','; echo "${results[*]}")" \
| jq '.' > "${RETRY_LOG}" 2>/dev/null || true
# Print summary
echo ""
echo "=== Agent Recovery Check ==="
printf "Total agents: %s | Running: %s | Success: %s | Failed: %s\n\n" \
"${total}" "${running}" "${success}" "${failed}"
# Print per-agent status
for result in "${results[@]}"; do
local vm agent issue class has_pr has_commits
vm=$(echo "${result}" | jq -r '.vm')
agent=$(echo "${result}" | jq -r '.agent')
issue=$(echo "${result}" | jq -r '.issue_number')
class=$(echo "${result}" | jq -r '.classification')
has_pr=$(echo "${result}" | jq -r '.has_pr')
has_commits=$(echo "${result}" | jq -r '.has_commits')
local status_icon
case "${class}" in
success) status_icon="${_COLOR_GREEN}[OK]${_COLOR_RESET} " ;;
running) status_icon="${_COLOR_CYAN}[RUN]${_COLOR_RESET} " ;;
rate-limited) status_icon="${_COLOR_YELLOW}[RATE]${_COLOR_RESET} " ;;
crashed) status_icon="${_COLOR_RED}[CRASH]${_COLOR_RESET}" ;;
timeout) status_icon="${_COLOR_YELLOW}[TIME]${_COLOR_RESET} " ;;
error) status_icon="${_COLOR_RED}[ERR]${_COLOR_RESET} " ;;
*) status_icon="[???] " ;;
esac
printf "%s %-24s %-8s #%-6s %s" \
"${status_icon}" "${vm}" "${agent}" "${issue}" "${class}"
[[ "${has_commits}" == "true" ]] && printf " [has-commits]"
[[ "${has_pr}" == "true" ]] && printf " [pr-created]"
echo ""
done
echo ""
if [[ ${failed} -gt 0 ]]; then
echo "Run '$0 retry-all' to re-dispatch all failed agents."
fi
}
# ---------------------------------------------------------------------------
# retry_one <vm-ip> <agent-index>
# Re-dispatch a single failed agent, resuming from last commit if possible.
# ---------------------------------------------------------------------------
retry_one() {
local ip="$1"
local idx="$2"
local agent_user="agent-${idx}"
local agent_home="/home/${agent_user}"
local assignment_file="${agent_home}/assignment.json"
log_info "Preparing retry for ${agent_user}@${ip}"
# Read assignment
if ! ssh_root_quiet "${ip}" "test -f '${assignment_file}'" 2>/dev/null; then
log_error "No assignment.json for ${agent_user}@${ip}. Cannot retry."
return 1
fi
local raw
raw=$(ssh_root_quiet "${ip}" "cat '${assignment_file}'" || echo "{}")
local issue_number repo branch
issue_number=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('issue_number',''))" 2>/dev/null || true)
repo=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('repo',''))" 2>/dev/null || true)
branch=$(echo "${raw}" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('branch',''))" 2>/dev/null || true)
if [[ -z "${issue_number}" || -z "${repo}" || -z "${branch}" ]]; then
log_error "Incomplete assignment.json for ${agent_user}@${ip}."
return 1
fi
local repo_name
repo_name=$(basename "${repo}")
local clone_dir="${agent_home}/workspace/${repo_name}"
# Check if the branch has commits beyond origin/main
local has_commits=false
local commit_count
commit_count=$(ssh_agent_quiet "${ip}" "${agent_user}" \
"git -C '${clone_dir}' rev-list --count HEAD ^origin/main 2>/dev/null" || echo "0")
[[ "${commit_count:-0}" -gt 0 ]] && has_commits=true
# Choose prompt based on commit state
local prompt
if [[ "${has_commits}" == "true" ]]; then
log_info "Branch has commits - resuming from checkpoint."
prompt="You are resuming work on issue #${issue_number}. \
Check the current state of the branch '${branch}' and continue from where the previous agent left off. \
Review any existing commits, assess what remains to be done, and complete the implementation. \
Create a PR when complete that closes #${issue_number}."
else
log_info "No commits found - starting fresh."
# Reset branch to origin/main
ssh_agent_quiet "${ip}" "${agent_user}" \
"git -C '${clone_dir}' checkout -B '${branch}' origin/main 2>/dev/null" || true
prompt="You are an autonomous Claude Code agent. Work on GitHub issue #${issue_number}. \
Create a branch named ${branch}, implement the changes with tests, and create a PR that closes #${issue_number}. \
Follow the repo's CLAUDE.md instructions. Commit with message format: #${issue_number}: description."
fi
# Clear previous status and log (append retry marker)
ssh_root_quiet "${ip}" "rm -f '${agent_home}/status'" || true
ssh_root_quiet "${ip}" \
"echo '--- RETRY $(date -u +"%Y-%m-%dT%H:%M:%SZ") ---' >> '${agent_home}/run.log'" || true
# Re-launch via agent-launch.sh
"${SCRIPT_DIR}/agent-launch.sh" "${ip}" "${idx}" --prompt "${prompt}"
}
# ---------------------------------------------------------------------------
# Subcommand: retry <vm-ip> <agent-index>
# ---------------------------------------------------------------------------
cmd_retry() {
if [[ $# -lt 2 ]]; then
echo "Usage: $0 retry <vm-ip> <agent-index>" >&2
exit 1
fi
local ip="$1"
local idx="$2"
if ! [[ "${idx}" =~ ^[0-3]$ ]]; then
log_error "agent-index must be 0, 1, 2, or 3 (got: ${idx})"
exit 1
fi
retry_one "${ip}" "${idx}"
}
# ---------------------------------------------------------------------------
# Subcommand: retry-all
# ---------------------------------------------------------------------------
cmd_retry_all() {
if [[ ! -f "${RETRY_LOG}" ]]; then
log_warn "No recovery data found at ${RETRY_LOG}. Run 'check' first."
exit 0
fi
local retry_count=0
local skip_count=0
# Read failed agents from the check results
while IFS= read -r entry; do
local class ip idx
class=$(echo "${entry}" | jq -r '.classification')
ip=$(echo "${entry}" | jq -r '.ip')
idx=$(echo "${entry}" | jq -r '.agent_index')
case "${class}" in
success|running)
skip_count=$(( skip_count + 1 ))
log_info "Skipping agent-${idx}@${ip} (${class})"
;;
*)
log_info "Retrying agent-${idx}@${ip} (${class})"
if retry_one "${ip}" "${idx}"; then
retry_count=$(( retry_count + 1 ))
else
log_warn "Retry failed for agent-${idx}@${ip}"
fi
;;
esac
done < <(jq -c '.[]' "${RETRY_LOG}" 2>/dev/null || true)
echo ""
log_success "Retry complete: ${retry_count} agent(s) re-dispatched, ${skip_count} skipped."
}
# ---------------------------------------------------------------------------
# Dispatch
# ---------------------------------------------------------------------------
case "${SUBCOMMAND}" in
check) cmd_check ;;
retry) cmd_retry "$@" ;;
retry-all) cmd_retry_all ;;
*)
log_error "Unknown subcommand: ${SUBCOMMAND}"
usage
;;
esac
lib/common.sh
#!/usr/bin/env bash
# common.sh - Shared utilities for CCGM cloud-dispatch VM lifecycle scripts
# Source this file from all other cloud-dispatch scripts.
set -euo pipefail
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
# Server type for agent VMs (CCX63: 48 vCPU, 192 GB RAM)
CCGM_SERVER_TYPE="${CCGM_SERVER_TYPE:-ccx63}"
# Datacenter locations, in round-robin order for VM placement
# exported so sourcing scripts see it without shellcheck SC2034 false-positives
export CCGM_LOCATIONS=("fsn1" "nbg1" "hel1")
# Snapshot label selector used to find the latest golden image
export CCGM_IMAGE_LABEL="${CCGM_IMAGE_LABEL:-purpose=ccgm-agent}"
# SSH key name as stored in Hetzner (set by Terraform)
export CCGM_SSH_KEY_NAME="${CCGM_SSH_KEY_NAME:-ccgm-dispatch-key}"
# Firewall name as created by Terraform
export CCGM_FIREWALL_NAME="${CCGM_FIREWALL_NAME:-ccgm-dispatch-firewall}"
# VM name prefix and pattern
export CCGM_VM_PREFIX="ccgm-agent"
export CCGM_VM_PATTERN="ccgm-agent-*"
# Number of agent users per VM
export CCGM_AGENTS_PER_VM=4
# SSH identity file for dispatch connections
export CCGM_SSH_KEY="${CCGM_SSH_KEY:-${HOME}/.ssh/ccgm-dispatch-session}"
# ControlMaster socket directory
export CCGM_SSH_CTL_DIR="/tmp"
# Minimum free disk space (GB) required for a VM to be healthy
export CCGM_MIN_DISK_GB=20
# Minimum free memory (MB) per agent slot for a VM to be healthy
export CCGM_MIN_MEM_MB_PER_AGENT=4096
# ---------------------------------------------------------------------------
# Color output helpers
# ---------------------------------------------------------------------------
# Detect whether we have a terminal that supports color
if [[ -t 1 ]] && [[ "${TERM:-}" != "dumb" ]]; then
_COLOR_RESET=$'\033[0m'
_COLOR_RED=$'\033[0;31m'
_COLOR_GREEN=$'\033[0;32m'
_COLOR_YELLOW=$'\033[0;33m'
_COLOR_CYAN=$'\033[0;36m'
_COLOR_BOLD=$'\033[1m'
else
_COLOR_RESET=''
_COLOR_RED=''
_COLOR_GREEN=''
_COLOR_YELLOW=''
_COLOR_CYAN=''
_COLOR_BOLD=''
fi
# ---------------------------------------------------------------------------
# Logging utilities
# ---------------------------------------------------------------------------
log_info() {
echo "${_COLOR_CYAN}[INFO]${_COLOR_RESET} $(date '+%Y-%m-%dT%H:%M:%S') $*" >&2
}
log_success() {
echo "${_COLOR_GREEN}[OK]${_COLOR_RESET} $(date '+%Y-%m-%dT%H:%M:%S') $*" >&2
}
log_warn() {
echo "${_COLOR_YELLOW}[WARN]${_COLOR_RESET} $(date '+%Y-%m-%dT%H:%M:%S') $*" >&2
}
log_error() {
echo "${_COLOR_RED}[ERROR]${_COLOR_RESET} $(date '+%Y-%m-%dT%H:%M:%S') $*" >&2
}
# ---------------------------------------------------------------------------
# Prerequisite check
# ---------------------------------------------------------------------------
# require_cmd <command> - exits with error if <command> is not found in PATH
require_cmd() {
local cmd="$1"
if ! command -v "${cmd}" >/dev/null 2>&1; then
log_error "Required command not found: ${cmd}"
log_error "Install it and retry."
exit 1
fi
}
# ---------------------------------------------------------------------------
# VM naming convention
# ---------------------------------------------------------------------------
# vm_name <location> <index>
# Returns a deterministic VM name: ccgm-agent-<location>-<index>
# e.g. vm_name fsn1 0 -> ccgm-agent-fsn1-0
vm_name() {
local location="$1"
local index="$2"
echo "${CCGM_VM_PREFIX}-${location}-${index}"
}
# ---------------------------------------------------------------------------
# SSH configuration
# ---------------------------------------------------------------------------
# ssh_config_snippet - prints an SSH config Host block for ccgm-agent-* VMs.
# Append this to ~/.ssh/config or write to a dedicated include file.
ssh_config_snippet() {
cat <<'EOF'
Host ccgm-agent-*
User root
IdentityFile ~/.ssh/ccgm-dispatch-session
StrictHostKeyChecking accept-new
ControlMaster auto
ControlPath /tmp/ccgm-ssh-%r@%h:%p
ControlPersist 600
ServerAliveInterval 30
ServerAliveCountMax 3
BatchMode yes
ConnectTimeout 10
EOF
}
# ssh_opts - common SSH options array, suitable for use with ssh/scp.
# Usage: ssh "${ssh_opts[@]}" root@<ip> <command>
ssh_opts() {
echo \
-i "${CCGM_SSH_KEY}" \
-o StrictHostKeyChecking=accept-new \
-o ControlMaster=auto \
-o "ControlPath=${CCGM_SSH_CTL_DIR}/ccgm-ssh-%r@%h:%p" \
-o ControlPersist=600 \
-o ServerAliveInterval=30 \
-o ServerAliveCountMax=3 \
-o BatchMode=yes \
-o ConnectTimeout=10
}
# ---------------------------------------------------------------------------
# Hetzner helpers
# ---------------------------------------------------------------------------
# latest_image_id - returns the snapshot ID of the most recent ccgm-agent image.
# Selects by label type=ccgm-agent, sorted by created date descending.
latest_image_id() {
hcloud image list \
--type snapshot \
--selector "${CCGM_IMAGE_LABEL}" \
--output json \
| python3 -c "
import json, sys
images = json.load(sys.stdin)
if not images:
print('', end='')
sys.exit(0)
images.sort(key=lambda x: x.get('created', ''), reverse=True)
print(images[0]['id'])
"
}
# wait_for_vm_running <vm-name> [timeout_secs]
# Polls hcloud until the VM is in 'running' state or timeout is reached.
wait_for_vm_running() {
local name="$1"
local timeout="${2:-120}"
local elapsed=0
local interval=5
log_info "Waiting for ${name} to reach 'running' state (timeout ${timeout}s)..."
while [[ ${elapsed} -lt ${timeout} ]]; do
local status
status=$(hcloud server describe "${name}" --output json 2>/dev/null \
| python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('status','unknown'))" \
|| echo "unknown")
if [[ "${status}" == "running" ]]; then
log_success "${name} is running."
return 0
fi
sleep "${interval}"
elapsed=$(( elapsed + interval ))
done
log_error "${name} did not reach 'running' within ${timeout}s (last status: ${status:-unknown})."
return 1
}
# wait_for_ssh <ip> [timeout_secs]
# Retries SSH connection until it succeeds or timeout is reached.
wait_for_ssh() {
local ip="$1"
local timeout="${2:-120}"
local elapsed=0
local interval=5
log_info "Waiting for SSH on ${ip} (timeout ${timeout}s)..."
while [[ ${elapsed} -lt ${timeout} ]]; do
# ssh_opts returns space-separated options; read into array to avoid word splitting
# shellcheck disable=SC2206
read -ra _ssh_wait_opts <<< "$(ssh_opts)"
if ssh "${_ssh_wait_opts[@]}" "root@${ip}" "true" 2>/dev/null; then
log_success "SSH is reachable on ${ip}."
return 0
fi
sleep "${interval}"
elapsed=$(( elapsed + interval ))
done
log_error "SSH on ${ip} did not become reachable within ${timeout}s."
return 1
}
# vm_ip <vm-name> - prints the public IPv4 address of a named VM.
vm_ip() {
local name="$1"
hcloud server describe "${name}" --output json \
| python3 -c "
import json, sys
d = json.load(sys.stdin)
nets = d.get('public_net', {})
ipv4 = nets.get('ipv4', {})
print(ipv4.get('ip', ''))
"
}
# ---------------------------------------------------------------------------
# Trap / cleanup registration
# ---------------------------------------------------------------------------
# cleanup_fns is an array of function names to call on EXIT.
# Use register_cleanup to add entries.
declare -a _cleanup_fns=()
register_cleanup() {
_cleanup_fns+=("$1")
}
_run_cleanup() {
for fn in "${_cleanup_fns[@]+"${_cleanup_fns[@]}"}"; do
"${fn}" || true
done
}
trap '_run_cleanup' EXIT
lib/vm-create.sh
#!/usr/bin/env bash
# vm-create.sh - Create CCGM agent VMs from the golden image snapshot.
#
# Usage:
# vm-create.sh [count] [--type TYPE] [--image IMAGE_ID]
#
# Arguments:
# count Number of VMs to create (default: 3)
#
# Options:
# --type TYPE Server type (default: ccx63)
# --image IMAGE_ID Snapshot ID to use (default: latest ccgm-agent snapshot)
#
# Environment:
# HCLOUD_TOKEN Hetzner Cloud API token (required)
#
# VMs are named ccgm-agent-<location>-<index> and spread round-robin across
# the configured datacenter locations (fsn1, nbg1, hel1 by default).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Parse arguments
# ---------------------------------------------------------------------------
COUNT=3
SERVER_TYPE="${CCGM_SERVER_TYPE}"
IMAGE_ID=""
while [[ $# -gt 0 ]]; do
case "$1" in
--type)
SERVER_TYPE="$2"; shift 2 ;;
--image)
IMAGE_ID="$2"; shift 2 ;;
--*)
log_error "Unknown option: $1"; exit 1 ;;
*)
COUNT="$1"; shift ;;
esac
done
if ! [[ "${COUNT}" =~ ^[0-9]+$ ]] || [[ "${COUNT}" -lt 1 ]]; then
log_error "count must be a positive integer, got: ${COUNT}"
exit 1
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd hcloud
require_cmd python3
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Export it before running this script."
exit 1
fi
# Resolve image ID if not provided
if [[ -z "${IMAGE_ID}" ]]; then
log_info "Resolving latest ccgm-agent snapshot..."
IMAGE_ID="$(latest_image_id)"
if [[ -z "${IMAGE_ID}" ]]; then
log_error "No snapshot found matching label '${CCGM_IMAGE_LABEL}'."
log_error "Build the golden image first with: packer build packer/agent-image.pkr.hcl"
exit 1
fi
log_info "Using snapshot ID: ${IMAGE_ID}"
fi
# Resolve SSH key and firewall IDs from Hetzner
log_info "Resolving SSH key '${CCGM_SSH_KEY_NAME}'..."
SSH_KEY_ID="$(hcloud ssh-key describe "${CCGM_SSH_KEY_NAME}" --output json \
| python3 -c "import json,sys; print(json.load(sys.stdin)['id'])")"
log_info "Resolving firewall '${CCGM_FIREWALL_NAME}'..."
FIREWALL_ID="$(hcloud firewall describe "${CCGM_FIREWALL_NAME}" --output json \
| python3 -c "import json,sys; print(json.load(sys.stdin)['id'])")"
# ---------------------------------------------------------------------------
# Create VMs
# ---------------------------------------------------------------------------
log_info "Creating ${COUNT} VM(s) with type=${SERVER_TYPE}, image=${IMAGE_ID}..."
declare -a CREATED_NAMES=()
declare -a FAILED_NAMES=()
for (( i=0; i<COUNT; i++ )); do
location="${CCGM_LOCATIONS[$(( i % ${#CCGM_LOCATIONS[@]} ))]}"
name="$(vm_name "${location}" "${i}")"
log_info "Creating ${name} in ${location}..."
if hcloud server create \
--name "${name}" \
--type "${SERVER_TYPE}" \
--image "${IMAGE_ID}" \
--location "${location}" \
--ssh-key "${SSH_KEY_ID}" \
--firewall "${FIREWALL_ID}" \
--output json >/dev/null 2>&1; then
CREATED_NAMES+=("${name}")
log_success "Created ${name}."
else
FAILED_NAMES+=("${name}")
log_error "Failed to create ${name}."
fi
done
if [[ ${#FAILED_NAMES[@]} -gt 0 ]]; then
log_warn "Some VMs failed to create: ${FAILED_NAMES[*]}"
fi
if [[ ${#CREATED_NAMES[@]} -eq 0 ]]; then
log_error "No VMs were created successfully."
exit 1
fi
# ---------------------------------------------------------------------------
# Wait for VMs to be running and SSH-reachable
# ---------------------------------------------------------------------------
declare -a READY_NAMES=()
declare -a UNREADY_NAMES=()
for name in "${CREATED_NAMES[@]}"; do
if wait_for_vm_running "${name}" 120; then
ip="$(vm_ip "${name}")"
if wait_for_ssh "${ip}" 120; then
READY_NAMES+=("${name}")
else
UNREADY_NAMES+=("${name}")
log_warn "${name} (${ip}): VM running but SSH unreachable."
fi
else
UNREADY_NAMES+=("${name}")
log_warn "${name}: did not reach running state in time."
fi
done
# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
echo ""
echo "${_COLOR_BOLD}==== VM Create Summary ====${_COLOR_RESET}"
echo ""
if [[ ${#READY_NAMES[@]} -gt 0 ]]; then
echo "${_COLOR_GREEN}Ready:${_COLOR_RESET}"
for name in "${READY_NAMES[@]}"; do
ip="$(vm_ip "${name}")"
location="$(hcloud server describe "${name}" --output json \
| python3 -c "import json,sys; print(json.load(sys.stdin)['datacenter']['location']['name'])")"
printf " %-30s %-15s %s\n" "${name}" "${ip}" "${location}"
done
fi
if [[ ${#UNREADY_NAMES[@]} -gt 0 ]]; then
echo ""
echo "${_COLOR_YELLOW}Not ready (manual intervention may be needed):${_COLOR_RESET}"
for name in "${UNREADY_NAMES[@]}"; do
printf " %s\n" "${name}"
done
fi
if [[ ${#FAILED_NAMES[@]} -gt 0 ]]; then
echo ""
echo "${_COLOR_RED}Failed to create:${_COLOR_RESET}"
for name in "${FAILED_NAMES[@]}"; do
printf " %s\n" "${name}"
done
fi
echo ""
if [[ ${#UNREADY_NAMES[@]} -gt 0 ]] || [[ ${#FAILED_NAMES[@]} -gt 0 ]]; then
exit 1
fi
lib/vm-destroy.sh
#!/usr/bin/env bash
# vm-destroy.sh - Terminate CCGM agent VMs.
#
# Usage:
# vm-destroy.sh --all [--force]
# vm-destroy.sh [--force] <vm-name> [<vm-name>...]
#
# Options:
# --all Destroy all VMs matching the ccgm-agent-* pattern
# --force Skip confirmation prompt
#
# After destruction, removes SSH known_hosts entries and ControlMaster sockets
# for each destroyed VM.
#
# Environment:
# HCLOUD_TOKEN Hetzner Cloud API token (required)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Parse arguments
# ---------------------------------------------------------------------------
DESTROY_ALL=false
FORCE=false
declare -a TARGET_NAMES=()
while [[ $# -gt 0 ]]; do
case "$1" in
--all)
DESTROY_ALL=true; shift ;;
--force)
FORCE=true; shift ;;
--*)
log_error "Unknown option: $1"; exit 1 ;;
*)
TARGET_NAMES+=("$1"); shift ;;
esac
done
if [[ "${DESTROY_ALL}" == "false" ]] && [[ ${#TARGET_NAMES[@]} -eq 0 ]]; then
echo "Usage: $0 --all [--force]" >&2
echo " $0 [--force] <vm-name> [<vm-name>...]" >&2
exit 1
fi
if [[ "${DESTROY_ALL}" == "true" ]] && [[ ${#TARGET_NAMES[@]} -gt 0 ]]; then
log_error "--all and explicit VM names are mutually exclusive."
exit 1
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd hcloud
require_cmd python3
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Export it before running this script."
exit 1
fi
# ---------------------------------------------------------------------------
# Resolve targets
# ---------------------------------------------------------------------------
if [[ "${DESTROY_ALL}" == "true" ]]; then
log_info "Listing all VMs matching pattern '${CCGM_VM_PATTERN}'..."
mapfile -t TARGET_NAMES < <(
hcloud server list --output json \
| python3 -c "
import json, sys, fnmatch
servers = json.load(sys.stdin)
pattern = '${CCGM_VM_PATTERN}'
for s in servers:
if fnmatch.fnmatch(s['name'], pattern):
print(s['name'])
" \
|| true
)
if [[ ${#TARGET_NAMES[@]} -eq 0 ]]; then
log_info "No VMs matching '${CCGM_VM_PATTERN}' found. Nothing to destroy."
exit 0
fi
fi
log_info "Targets: ${TARGET_NAMES[*]}"
# ---------------------------------------------------------------------------
# Collect IPs before deletion (needed for known_hosts cleanup)
# ---------------------------------------------------------------------------
declare -A VM_IPS=()
for name in "${TARGET_NAMES[@]}"; do
ip="$(vm_ip "${name}" 2>/dev/null || true)"
if [[ -n "${ip}" ]]; then
VM_IPS["${name}"]="${ip}"
else
log_warn "Could not resolve IP for ${name}; known_hosts cleanup may be incomplete."
fi
done
# ---------------------------------------------------------------------------
# Confirmation prompt
# ---------------------------------------------------------------------------
if [[ "${FORCE}" == "false" ]]; then
echo ""
echo "${_COLOR_YELLOW}The following VMs will be permanently destroyed:${_COLOR_RESET}"
for name in "${TARGET_NAMES[@]}"; do
ip="${VM_IPS[${name}]:-unknown}"
printf " %-30s %s\n" "${name}" "${ip}"
done
echo ""
read -r -p "Type 'yes' to confirm: " CONFIRM
if [[ "${CONFIRM}" != "yes" ]]; then
log_info "Aborted."
exit 0
fi
fi
# ---------------------------------------------------------------------------
# Destroy VMs
# ---------------------------------------------------------------------------
declare -a DESTROYED_NAMES=()
declare -a FAILED_NAMES=()
for name in "${TARGET_NAMES[@]}"; do
log_info "Destroying ${name}..."
if hcloud server delete "${name}" 2>/dev/null; then
DESTROYED_NAMES+=("${name}")
log_success "Destroyed ${name}."
else
FAILED_NAMES+=("${name}")
log_error "Failed to destroy ${name}."
fi
done
# ---------------------------------------------------------------------------
# Clean up SSH artifacts
# ---------------------------------------------------------------------------
for name in "${DESTROYED_NAMES[@]}"; do
ip="${VM_IPS[${name}]:-}"
# Remove known_hosts entry for the IP
if [[ -n "${ip}" ]] && [[ -f "${HOME}/.ssh/known_hosts" ]]; then
ssh-keygen -R "${ip}" >/dev/null 2>&1 || true
log_info "Removed ${ip} from ~/.ssh/known_hosts."
fi
# Remove ControlMaster socket files matching this VM
if [[ -n "${ip}" ]]; then
# Socket path pattern: /tmp/ccgm-ssh-root@<ip>:<port>
find "${CCGM_SSH_CTL_DIR}" -maxdepth 1 -name "ccgm-ssh-root@${ip}:*" -exec rm -f {} \; 2>/dev/null || true
fi
# Also clean by name-based socket pattern if present
find "${CCGM_SSH_CTL_DIR}" -maxdepth 1 -name "ccgm-ssh-*${name}*" -exec rm -f {} \; 2>/dev/null || true
done
# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
echo ""
echo "${_COLOR_BOLD}==== VM Destroy Summary ====${_COLOR_RESET}"
echo ""
if [[ ${#DESTROYED_NAMES[@]} -gt 0 ]]; then
echo "${_COLOR_GREEN}Destroyed:${_COLOR_RESET}"
for name in "${DESTROYED_NAMES[@]}"; do
printf " %s\n" "${name}"
done
fi
if [[ ${#FAILED_NAMES[@]} -gt 0 ]]; then
echo ""
echo "${_COLOR_RED}Failed to destroy:${_COLOR_RESET}"
for name in "${FAILED_NAMES[@]}"; do
printf " %s\n" "${name}"
done
exit 1
fi
echo ""
lib/vm-health.sh
#!/usr/bin/env bash
# vm-health.sh - Health check CCGM agent VMs.
#
# Usage:
# vm-health.sh Check all ccgm-agent-* VMs
# vm-health.sh --all Same as above
# vm-health.sh <vm-name> [...] Check specific VM(s) by name
#
# Per-VM checks:
# - SSH reachability (timeout 10s)
# - `claude --version` returns successfully
# - Free disk space > 20 GB
# - Free memory > 4 GB per agent slot (CCGM_AGENTS_PER_VM slots)
# - Agent users exist (agent-0 through agent-3)
# - iptables rules active (non-empty ruleset)
#
# Reports HEALTHY / DEGRADED / UNREACHABLE per VM.
# Exit code: 0 if all VMs are HEALTHY, 1 if any DEGRADED or UNREACHABLE.
#
# Environment:
# HCLOUD_TOKEN Hetzner Cloud API token (required)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Parse arguments
# ---------------------------------------------------------------------------
declare -a TARGET_NAMES=()
CHECK_ALL=false
while [[ $# -gt 0 ]]; do
case "$1" in
--all)
CHECK_ALL=true; shift ;;
--*)
log_error "Unknown option: $1"; exit 1 ;;
*)
TARGET_NAMES+=("$1"); shift ;;
esac
done
# Default: check all if no targets specified
if [[ ${#TARGET_NAMES[@]} -eq 0 ]]; then
CHECK_ALL=true
fi
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd hcloud
require_cmd ssh
require_cmd python3
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Export it before running this script."
exit 1
fi
# ---------------------------------------------------------------------------
# Resolve target list
# ---------------------------------------------------------------------------
if [[ "${CHECK_ALL}" == "true" ]]; then
log_info "Discovering ccgm-agent-* VMs..."
mapfile -t TARGET_NAMES < <(
hcloud server list --output json \
| python3 -c "
import json, sys, fnmatch
servers = json.load(sys.stdin)
for s in sorted(servers, key=lambda x: x['name']):
if fnmatch.fnmatch(s['name'], '${CCGM_VM_PATTERN}'):
print(s['name'])
" \
|| true
)
if [[ ${#TARGET_NAMES[@]} -eq 0 ]]; then
log_info "No ccgm-agent-* VMs found."
exit 0
fi
fi
# ---------------------------------------------------------------------------
# Health check helpers
# ---------------------------------------------------------------------------
# Run a remote command on a VM, capturing output. Returns the exit code.
_remote() {
local ip="$1"
shift
# shellcheck disable=SC2206
read -ra opts <<< "$(ssh_opts)"
ssh "${opts[@]}" -o ConnectTimeout=10 "root@${ip}" "$@" 2>/dev/null
}
# check_ssh <ip> - returns 0 if SSH is reachable within 10s
check_ssh() {
local ip="$1"
# shellcheck disable=SC2206
read -ra opts <<< "$(ssh_opts)"
ssh "${opts[@]}" -o ConnectTimeout=10 "root@${ip}" "true" 2>/dev/null
}
# check_claude <ip> - returns 0 if `claude --version` succeeds
check_claude() {
_remote "$1" "claude --version" >/dev/null 2>&1
}
# check_disk <ip> - returns 0 if free space on / exceeds CCGM_MIN_DISK_GB
check_disk() {
local ip="$1"
local free_kb
free_kb="$(_remote "${ip}" "df -k / | awk 'NR==2{print \$4}'")" || return 1
local free_gb=$(( free_kb / 1024 / 1024 ))
[[ "${free_gb}" -ge "${CCGM_MIN_DISK_GB}" ]]
}
# check_memory <ip> - returns 0 if free memory satisfies per-agent requirement
check_memory() {
local ip="$1"
local free_mb
free_mb="$(_remote "${ip}" "free -m | awk '/^Mem:/{print \$7}'")" || return 1
local required_mb=$(( CCGM_MIN_MEM_MB_PER_AGENT * CCGM_AGENTS_PER_VM ))
[[ "${free_mb}" -ge "${required_mb}" ]]
}
# check_agent_users <ip> - returns 0 if agent-0 through agent-N-1 all exist
check_agent_users() {
local ip="$1"
local script=""
for (( u=0; u<CCGM_AGENTS_PER_VM; u++ )); do
script+="id agent-${u} >/dev/null 2>&1 || exit 1; "
done
_remote "${ip}" "bash -c '${script}exit 0'" >/dev/null 2>&1
}
# check_iptables <ip> - returns 0 if iptables has at least one non-default chain rule
check_iptables() {
local ip="$1"
local rule_count
rule_count="$(_remote "${ip}" "iptables -L -n | grep -c '^[A-Z]' || true")" 2>/dev/null || return 1
[[ "${rule_count}" -gt 3 ]]
}
# ---------------------------------------------------------------------------
# Run health checks on each VM
# ---------------------------------------------------------------------------
declare -a HEALTHY_NAMES=()
declare -a DEGRADED_NAMES=()
declare -a UNREACHABLE_NAMES=()
echo ""
echo "${_COLOR_BOLD}==== CCGM Agent VM Health Check ====${_COLOR_RESET}"
echo ""
for name in "${TARGET_NAMES[@]}"; do
ip="$(vm_ip "${name}" 2>/dev/null || true)"
if [[ -z "${ip}" ]]; then
log_warn "${name}: could not resolve IP (VM may not exist)."
UNREACHABLE_NAMES+=("${name}")
printf " %-32s %s\n" "${name}" "${_COLOR_RED}UNREACHABLE${_COLOR_RESET} (no IP)"
continue
fi
# SSH reachability is a hard requirement
if ! check_ssh "${ip}" 2>/dev/null; then
UNREACHABLE_NAMES+=("${name}")
printf " %-32s %s\n" "${name}" "${_COLOR_RED}UNREACHABLE${_COLOR_RESET} (SSH timeout)"
continue
fi
declare -a failures=()
check_claude "${ip}" || failures+=("claude-not-found")
check_disk "${ip}" || failures+=("low-disk (<${CCGM_MIN_DISK_GB}GB)")
check_memory "${ip}" || failures+=("low-memory (<$(( CCGM_MIN_MEM_MB_PER_AGENT * CCGM_AGENTS_PER_VM ))MB free)")
check_agent_users "${ip}" || failures+=("missing-agent-users")
check_iptables "${ip}" || failures+=("iptables-inactive")
if [[ ${#failures[@]} -eq 0 ]]; then
HEALTHY_NAMES+=("${name}")
printf " %-32s %s\n" "${name}" "${_COLOR_GREEN}HEALTHY${_COLOR_RESET}"
else
DEGRADED_NAMES+=("${name}")
printf " %-32s %s [%s]\n" \
"${name}" \
"${_COLOR_YELLOW}DEGRADED${_COLOR_RESET}" \
"${failures[*]}"
fi
done
# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
echo ""
echo "${_COLOR_BOLD}Summary:${_COLOR_RESET}"
echo " Healthy: ${#HEALTHY_NAMES[@]}"
echo " Degraded: ${#DEGRADED_NAMES[@]}"
echo " Unreachable: ${#UNREACHABLE_NAMES[@]}"
echo " Total: ${#TARGET_NAMES[@]}"
echo ""
if [[ ${#DEGRADED_NAMES[@]} -gt 0 ]] || [[ ${#UNREACHABLE_NAMES[@]} -gt 0 ]]; then
exit 1
fi
lib/vm-ssh.sh
#!/usr/bin/env bash
# vm-ssh.sh - SSH into a CCGM agent VM.
#
# Usage:
# vm-ssh.sh <vm-name> [command [args...]]
#
# If a command is provided, it is executed non-interactively and the script
# exits with its return code. If no command is given, an interactive shell
# session is opened.
#
# ControlMaster sockets are reused when available (see SSH config in common.sh).
#
# Environment:
# HCLOUD_TOKEN Hetzner Cloud API token (required to resolve the VM IP)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Parse arguments
# ---------------------------------------------------------------------------
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <vm-name> [command [args...]]" >&2
exit 1
fi
VM_NAME="$1"
shift
REMOTE_CMD=("$@")
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd hcloud
require_cmd ssh
require_cmd python3
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Export it before running this script."
exit 1
fi
# ---------------------------------------------------------------------------
# Resolve IP
# ---------------------------------------------------------------------------
log_info "Resolving IP for ${VM_NAME}..."
VM_IP="$(vm_ip "${VM_NAME}")"
if [[ -z "${VM_IP}" ]]; then
log_error "Could not resolve IP for VM '${VM_NAME}'. Is it running?"
exit 1
fi
log_info "${VM_NAME} -> ${VM_IP}"
# ---------------------------------------------------------------------------
# Build SSH options array
# ---------------------------------------------------------------------------
# Using read -ra to split the opts string into an array
# shellcheck disable=SC2206
read -ra SSH_OPTS <<< "$(ssh_opts)"
# ---------------------------------------------------------------------------
# Connect
# ---------------------------------------------------------------------------
if [[ ${#REMOTE_CMD[@]} -gt 0 ]]; then
# Non-interactive: run command and exit
exec ssh "${SSH_OPTS[@]}" "root@${VM_IP}" "${REMOTE_CMD[@]}"
else
# Interactive session: allocate a TTY
exec ssh -t "${SSH_OPTS[@]}" "root@${VM_IP}"
fi
lib/vm-status.sh
#!/usr/bin/env bash
# vm-status.sh - List all running CCGM agent VMs with status details.
#
# Usage:
# vm-status.sh
#
# Output:
# Formatted table: NAME | IP | LOCATION | STATUS | UPTIME | TYPE
#
# Environment:
# HCLOUD_TOKEN Hetzner Cloud API token (required)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=common.sh
source "${SCRIPT_DIR}/common.sh"
# ---------------------------------------------------------------------------
# Prerequisite checks
# ---------------------------------------------------------------------------
require_cmd hcloud
require_cmd python3
if [[ -z "${HCLOUD_TOKEN:-}" ]]; then
log_error "HCLOUD_TOKEN is not set. Export it before running this script."
exit 1
fi
# ---------------------------------------------------------------------------
# Fetch and format VM list
# ---------------------------------------------------------------------------
log_info "Fetching VM list..."
python3 - <<'PYEOF'
import json
import subprocess
import sys
from datetime import datetime, timezone
result = subprocess.run(
["hcloud", "server", "list", "--output", "json"],
capture_output=True,
text=True,
)
if result.returncode != 0:
print(f"ERROR: hcloud server list failed: {result.stderr}", file=sys.stderr)
sys.exit(1)
servers = json.loads(result.stdout)
# Filter to ccgm-agent-* VMs only
servers = [s for s in servers if s.get("name", "").startswith("ccgm-agent-")]
if not servers:
print("No ccgm-agent-* VMs found.")
sys.exit(0)
# Sort by name for stable output
servers.sort(key=lambda s: s["name"])
# Column widths
col_name = max(len(s["name"]) for s in servers) + 2
col_ip = 16
col_loc = 8
col_stat = 10
col_up = 16
col_type = 8
header = (
f"{'NAME':<{col_name}} {'IP':<{col_ip}} {'LOC':<{col_loc}}"
f" {'STATUS':<{col_stat}} {'UPTIME':<{col_up}} {'TYPE':<{col_type}}"
)
sep = "-" * len(header)
print(header)
print(sep)
now = datetime.now(timezone.utc)
for s in servers:
name = s.get("name", "")
ip = s.get("public_net", {}).get("ipv4", {}).get("ip", "")
loc = s.get("datacenter", {}).get("location", {}).get("name", "")
status = s.get("status", "")
stype = s.get("server_type", {}).get("name", "")
created_str = s.get("created", "")
if created_str:
try:
created = datetime.fromisoformat(created_str.replace("Z", "+00:00"))
delta = now - created
total = int(delta.total_seconds())
hours, rem = divmod(total, 3600)
mins = rem // 60
uptime = f"{hours}h {mins}m"
except ValueError:
uptime = "unknown"
else:
uptime = "unknown"
print(
f"{name:<{col_name}} {ip:<{col_ip}} {loc:<{col_loc}}"
f" {status:<{col_stat}} {uptime:<{col_up}} {stype:<{col_type}}"
)
print()
print(f"Total: {len(servers)} VM(s)")
PYEOF
lib/secrets-cleanup.sh
#!/usr/bin/env bash
# secrets-cleanup.sh — Revoke session credentials and wipe VM secrets.
#
# Reads session metadata from /tmp/ccgm-session.json (written by secrets-init.sh),
# deletes the Hetzner SSH key, removes it from ssh-agent, and optionally wipes
# /run/secrets/ on all running VMs.
#
# Usage:
# secrets-cleanup.sh [--wipe-vms]
#
# Options:
# --wipe-vms SSH into each running ccgm-agent-* VM and wipe /run/secrets/
#
# Requirements:
# - hcloud CLI authenticated
# - jq installed
set -euo pipefail
SESSION_FILE="/tmp/ccgm-session.json"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
log() { printf '[secrets-cleanup] %s\n' "$*" >&2; }
warn() { printf '[secrets-cleanup] WARN: %s\n' "$*" >&2; }
die() { printf '[secrets-cleanup] ERROR: %s\n' "$*" >&2; exit 1; }
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
WIPE_VMS=false
while [[ $# -gt 0 ]]; do
case "$1" in
--wipe-vms)
WIPE_VMS=true
shift
;;
--help|-h)
grep '^#' "$0" | sed 's/^# \?//'
exit 0
;;
*)
die "Unknown argument: $1"
;;
esac
done
# ---------------------------------------------------------------------------
# Preflight checks
# ---------------------------------------------------------------------------
command -v hcloud >/dev/null 2>&1 || die "hcloud CLI not found in PATH"
command -v jq >/dev/null 2>&1 || die "jq not found in PATH"
command -v ssh-add >/dev/null 2>&1 || die "ssh-add not found in PATH"
# ---------------------------------------------------------------------------
# Read session metadata
# ---------------------------------------------------------------------------
if [[ ! -f "${SESSION_FILE}" ]]; then
warn "Session file not found at ${SESSION_FILE} — nothing to clean up"
exit 0
fi
KEY_ID="$(jq -r '.key_id' "${SESSION_FILE}")"
KEY_NAME="$(jq -r '.key_name' "${SESSION_FILE}")"
FINGERPRINT="$(jq -r '.fingerprint' "${SESSION_FILE}")"
[[ -n "${KEY_ID}" && "${KEY_ID}" != "null" ]] \
|| die "Could not read key_id from ${SESSION_FILE}"
log "Session: key_name=${KEY_NAME} key_id=${KEY_ID} fingerprint=${FINGERPRINT}"
# ---------------------------------------------------------------------------
# Remove key from ssh-agent
# ---------------------------------------------------------------------------
log "Removing session key from ssh-agent (fingerprint: ${FINGERPRINT})"
if ssh-add -l 2>/dev/null | grep -q "${FINGERPRINT}"; then
ssh-add -d - 2>/dev/null <<< "" || {
# ssh-add -d requires the public key or fingerprint; use -D as fallback
# only if no other keys are loaded, to avoid revoking unrelated keys.
LOADED_COUNT="$(ssh-add -l 2>/dev/null | grep -c . || true)"
if [[ "${LOADED_COUNT}" -eq 1 ]]; then
log " Using ssh-add -D (only session key is loaded)"
ssh-add -D
else
warn " Could not remove specific key; ${LOADED_COUNT} keys loaded — manual cleanup may be needed"
fi
}
else
log " Key not found in ssh-agent (may have already been removed)"
fi
# ---------------------------------------------------------------------------
# Delete key from Hetzner
# ---------------------------------------------------------------------------
log "Deleting SSH key from Hetzner Cloud (ID: ${KEY_ID})"
if hcloud ssh-key describe "${KEY_ID}" &>/dev/null; then
hcloud ssh-key delete "${KEY_ID}"
log " Deleted Hetzner SSH key ${KEY_NAME} (${KEY_ID})"
else
log " Key ID ${KEY_ID} not found in Hetzner — may have been deleted already"
fi
# ---------------------------------------------------------------------------
# Optional: wipe /run/secrets/ on all running VMs
# ---------------------------------------------------------------------------
if [[ "${WIPE_VMS}" == "true" ]]; then
log "Discovering running ccgm-agent-* VMs for secret wipe"
RUNNING_VMS="$(hcloud server list --output json 2>/dev/null \
| jq -r '.[] | select(.name | startswith("ccgm-agent")) | select(.status == "running") | "\(.name) \(.public_net.ipv4.ip)"' \
|| true)"
if [[ -z "${RUNNING_VMS}" ]]; then
log "No running VMs found — nothing to wipe"
else
SSH_OPTS=(
-o BatchMode=yes
-o StrictHostKeyChecking=accept-new
-o ConnectTimeout=10
)
while IFS=' ' read -r vm_name vm_ip; do
[[ -z "${vm_name}" ]] && continue
log "Wiping /run/secrets/ on ${vm_name} (${vm_ip})"
if HISTFILE=/dev/null ssh -n "${SSH_OPTS[@]}" "root@${vm_ip}" \
'find /run/secrets -mindepth 2 -type f -exec shred -u {} \; 2>/dev/null; echo "wiped"' 2>/dev/null; then
log " ${vm_name}: wiped"
else
warn " ${vm_name}: wipe failed or VM unreachable"
fi
done <<< "${RUNNING_VMS}"
fi
fi
# ---------------------------------------------------------------------------
# Remove session metadata file
# ---------------------------------------------------------------------------
log "Removing session metadata file ${SESSION_FILE}"
rm -f "${SESSION_FILE}"
log "secrets-cleanup complete"
lib/secrets-init.sh
#!/usr/bin/env bash
# secrets-init.sh — Initialize session SSH credentials for cloud-dispatch.
#
# Generates a per-session ed25519 SSH keypair, loads it into macOS ssh-agent,
# and registers the public key with Hetzner Cloud. Session metadata is written
# to /tmp/ccgm-session.json for use by secrets-cleanup.sh.
#
# Usage: secrets-init.sh
#
# Requirements:
# - hcloud CLI authenticated (HCLOUD_TOKEN env or hcloud context)
# - macOS ssh-agent running (SSH_AUTH_SOCK set)
# - jq installed
set -euo pipefail
SESSION_FILE="/tmp/ccgm-session.json"
TMPKEY="/tmp/ccgm-session-key-$$"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
log() { printf '[secrets-init] %s\n' "$*" >&2; }
die() { printf '[secrets-init] ERROR: %s\n' "$*" >&2; exit 1; }
cleanup_tmpkey() {
rm -f "${TMPKEY}" "${TMPKEY}.pub" 2>/dev/null || true
}
trap cleanup_tmpkey EXIT
# ---------------------------------------------------------------------------
# Preflight checks
# ---------------------------------------------------------------------------
command -v hcloud >/dev/null 2>&1 || die "hcloud CLI not found in PATH"
command -v ssh-add >/dev/null 2>&1 || die "ssh-add not found in PATH"
command -v ssh-keygen >/dev/null 2>&1 || die "ssh-keygen not found in PATH"
command -v jq >/dev/null 2>&1 || die "jq not found in PATH"
[[ -n "${SSH_AUTH_SOCK:-}" ]] || die "SSH_AUTH_SOCK is not set — is ssh-agent running?"
if [[ -f "${SESSION_FILE}" ]]; then
log "WARNING: session file already exists at ${SESSION_FILE}"
log "Run secrets-cleanup.sh first to revoke the previous session."
exit 1
fi
# ---------------------------------------------------------------------------
# Generate session keypair
# ---------------------------------------------------------------------------
SESSION_TS="$(date +%s)"
KEY_NAME="ccgm-session-${SESSION_TS}"
log "Generating ed25519 session keypair (${KEY_NAME})"
# -N "" means no passphrase so the key can be used by automation without prompting.
ssh-keygen -t ed25519 -f "${TMPKEY}" -N "" -C "${KEY_NAME}" -q
log "Adding private key to ssh-agent"
ssh-add "${TMPKEY}"
# Public key is safe to keep until Hetzner upload completes; trap removes both.
PUBKEY_CONTENT="$(cat "${TMPKEY}.pub")"
# ---------------------------------------------------------------------------
# Register public key with Hetzner
# ---------------------------------------------------------------------------
log "Uploading public key to Hetzner Cloud as '${KEY_NAME}'"
HCLOUD_OUTPUT="$(hcloud ssh-key create --name "${KEY_NAME}" --public-key "${PUBKEY_CONTENT}" --output json 2>/dev/null)"
HCLOUD_KEY_ID="$(printf '%s' "${HCLOUD_OUTPUT}" | jq -r '.ssh_key.id // .id')"
HCLOUD_FINGERPRINT="$(printf '%s' "${HCLOUD_OUTPUT}" | jq -r '.ssh_key.fingerprint // .fingerprint')"
[[ -n "${HCLOUD_KEY_ID}" && "${HCLOUD_KEY_ID}" != "null" ]] \
|| die "Failed to parse key ID from hcloud output"
log "Hetzner SSH key registered: ID=${HCLOUD_KEY_ID} fingerprint=${HCLOUD_FINGERPRINT}"
# ---------------------------------------------------------------------------
# Write session metadata
# ---------------------------------------------------------------------------
jq -n \
--arg key_name "${KEY_NAME}" \
--arg key_id "${HCLOUD_KEY_ID}" \
--arg fingerprint "${HCLOUD_FINGERPRINT}" \
--arg timestamp "${SESSION_TS}" \
'{
key_name: $key_name,
key_id: $key_id,
fingerprint: $fingerprint,
timestamp: $timestamp
}' > "${SESSION_FILE}"
chmod 0600 "${SESSION_FILE}"
log "Session metadata written to ${SESSION_FILE}"
log "secrets-init complete"
log ""
log " Key name: ${KEY_NAME}"
log " Key ID: ${HCLOUD_KEY_ID}"
log " Fingerprint: ${HCLOUD_FINGERPRINT}"
log ""
log "Run secrets-cleanup.sh to revoke this session when done."
lib/secrets-inject-all.sh
#!/usr/bin/env bash
# secrets-inject-all.sh — Inject credentials into all agents across all running VMs.
#
# Discovers running ccgm-agent-* Hetzner VMs, then calls secrets-inject.sh for
# each agent (0-3) on each VM. Reports per-agent success/failure.
#
# Usage:
# secrets-inject-all.sh --github-token TOKEN [--claude-auth TOKEN]
#
# Requirements:
# - hcloud CLI authenticated
# - SSH key loaded in ssh-agent (run secrets-init.sh first)
# - secrets-inject.sh in the same directory
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INJECT_SCRIPT="${SCRIPT_DIR}/secrets-inject.sh"
AGENT_COUNT=4
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
log() { printf '[secrets-inject-all] %s\n' "$*" >&2; }
die() { printf '[secrets-inject-all] ERROR: %s\n' "$*" >&2; exit 1; }
usage() {
cat >&2 <<'USAGE'
Usage: secrets-inject-all.sh --github-token TOKEN [--claude-auth TOKEN]
Options:
--github-token TOKEN GitHub personal access token (required)
--claude-auth TOKEN Claude authentication token (optional)
USAGE
exit 1
}
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
[[ $# -ge 1 ]] || usage
GITHUB_TOKEN=""
CLAUDE_AUTH=""
while [[ $# -gt 0 ]]; do
case "$1" in
--github-token)
GITHUB_TOKEN="$2"
shift 2
;;
--claude-auth)
CLAUDE_AUTH="$2"
shift 2
;;
--help|-h)
usage
;;
*)
die "Unknown argument: $1"
;;
esac
done
[[ -n "${GITHUB_TOKEN}" ]] || die "--github-token is required"
# ---------------------------------------------------------------------------
# Preflight checks
# ---------------------------------------------------------------------------
command -v hcloud >/dev/null 2>&1 || die "hcloud CLI not found in PATH"
[[ -x "${INJECT_SCRIPT}" ]] || die "secrets-inject.sh not found or not executable at ${INJECT_SCRIPT}"
[[ -n "${SSH_AUTH_SOCK:-}" ]] || die "SSH_AUTH_SOCK is not set — is ssh-agent running?"
# ---------------------------------------------------------------------------
# Discover running VMs
# ---------------------------------------------------------------------------
log "Discovering running ccgm-agent-* VMs"
# hcloud server list outputs name, status, public IP etc.
# Filter for VMs named ccgm-agent-* that are in 'running' status.
RUNNING_VMS="$(hcloud server list --selector 'ccgm-role=agent' --output columns=name,public_net 2>/dev/null \
| tail -n +2 \
| awk '{print $1, $2}' \
|| true)"
# Fallback: list by name prefix if no label selector works
if [[ -z "${RUNNING_VMS}" ]]; then
log "Label selector returned no results; falling back to name-prefix discovery"
RUNNING_VMS="$(hcloud server list --output json 2>/dev/null \
| jq -r '.[] | select(.name | startswith("ccgm-agent")) | select(.status == "running") | "\(.name) \(.public_net.ipv4.ip)"' \
|| true)"
fi
if [[ -z "${RUNNING_VMS}" ]]; then
log "No running ccgm-agent-* VMs found."
exit 0
fi
VM_COUNT="$(printf '%s\n' "${RUNNING_VMS}" | grep -c . || true)"
log "Found ${VM_COUNT} VM(s)"
# ---------------------------------------------------------------------------
# Inject secrets into each VM / agent combination
# ---------------------------------------------------------------------------
PASS=0
FAIL=0
FAIL_DETAILS=()
while IFS=' ' read -r vm_name vm_ip; do
[[ -z "${vm_name}" ]] && continue
log "Processing VM: ${vm_name} (${vm_ip})"
for i in $(seq 0 $((AGENT_COUNT - 1))); do
INJECT_ARGS=("${vm_ip}" "${i}" "--github-token" "${GITHUB_TOKEN}")
[[ -n "${CLAUDE_AUTH}" ]] && INJECT_ARGS+=("--claude-auth" "${CLAUDE_AUTH}")
if bash "${INJECT_SCRIPT}" "${INJECT_ARGS[@]}" 2>&1; then
PASS=$(( PASS + 1 ))
log " agent-${i}: OK"
else
FAIL=$(( FAIL + 1 ))
FAIL_DETAILS+=("${vm_name}/agent-${i}")
log " agent-${i}: FAILED"
fi
done
done <<< "${RUNNING_VMS}"
# ---------------------------------------------------------------------------
# Summary
# ---------------------------------------------------------------------------
TOTAL=$(( PASS + FAIL ))
log ""
log "Injection complete: ${PASS}/${TOTAL} agents succeeded"
if [[ "${FAIL}" -gt 0 ]]; then
log "Failed agents:"
for detail in "${FAIL_DETAILS[@]}"; do
log " - ${detail}"
done
exit 1
fi
lib/secrets-inject.sh
#!/usr/bin/env bash
# secrets-inject.sh — Inject credentials into a single agent on a VM.
#
# Writes secrets to the agent's tmpfs directory (/run/secrets/agent-N/) via SSH.
# Secrets are written using heredocs over the SSH connection so they never appear
# in process arguments (ps output). HISTFILE is disabled for all SSH sessions.
#
# Usage:
# secrets-inject.sh <vm-ip> <agent-index> [--github-token TOKEN] [--claude-auth TOKEN]
#
# Example:
# secrets-inject.sh 1.2.3.4 0 --github-token ghp_xxx --claude-auth sk-ant-xxx
#
# Requirements:
# - SSH key for root access already loaded in ssh-agent (run secrets-init.sh first)
set -euo pipefail
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
log() { printf '[secrets-inject] %s\n' "$*" >&2; }
die() { printf '[secrets-inject] ERROR: %s\n' "$*" >&2; exit 1; }
usage() {
cat >&2 <<'USAGE'
Usage: secrets-inject.sh <vm-ip> <agent-index> [options]
Options:
--github-token TOKEN GitHub personal access token
--claude-auth TOKEN Claude authentication token (API key or subscription token)
At least one of --github-token or --claude-auth must be provided.
USAGE
exit 1
}
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
[[ $# -ge 2 ]] || usage
VM_IP="$1"
AGENT_INDEX="$2"
shift 2
GITHUB_TOKEN=""
CLAUDE_AUTH=""
while [[ $# -gt 0 ]]; do
case "$1" in
--github-token)
GITHUB_TOKEN="$2"
shift 2
;;
--claude-auth)
CLAUDE_AUTH="$2"
shift 2
;;
*)
die "Unknown argument: $1"
;;
esac
done
# Validate inputs
[[ "${AGENT_INDEX}" =~ ^[0-3]$ ]] || die "agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})"
[[ -n "${VM_IP}" ]] || die "vm-ip is required"
[[ -n "${GITHUB_TOKEN}" || -n "${CLAUDE_AUTH}" ]] \
|| die "At least one of --github-token or --claude-auth must be provided"
AGENT_USER="agent-${AGENT_INDEX}"
SECRETS_DIR="/run/secrets/${AGENT_USER}"
# ---------------------------------------------------------------------------
# SSH helper — HISTFILE disabled for all sessions to prevent secret leakage
# ---------------------------------------------------------------------------
# Common SSH flags: batch mode, key-only auth, no stdin consumption.
SSH_OPTS=(
-n
-o BatchMode=yes
-o StrictHostKeyChecking=accept-new
-o ConnectTimeout=10
)
vm_exec() {
HISTFILE=/dev/null ssh "${SSH_OPTS[@]}" "root@${VM_IP}" "$@"
}
# Variant that reads stdin (for piping secrets); -n flag must be omitted.
SSH_STDIN_OPTS=(
-o BatchMode=yes
-o StrictHostKeyChecking=accept-new
-o ConnectTimeout=10
)
vm_exec_stdin() {
HISTFILE=/dev/null ssh "${SSH_STDIN_OPTS[@]}" "root@${VM_IP}" "$@"
}
# ---------------------------------------------------------------------------
# Verify connectivity
# ---------------------------------------------------------------------------
log "Verifying SSH connectivity to ${VM_IP}"
vm_exec true || die "Cannot SSH to ${VM_IP} — check that secrets-init.sh has been run"
# ---------------------------------------------------------------------------
# Ensure secrets directory exists with correct permissions
# ---------------------------------------------------------------------------
log "Ensuring ${SECRETS_DIR} exists with correct permissions"
# Remote script body stored in a variable so it can be passed via stdin.
# Variables are expanded on the client side before sending — this is intentional
# for non-secret values like AGENT_USER and SECRETS_DIR.
SETUP_SCRIPT="$(cat <<SETUP
set -euo pipefail
if mountpoint -q /run 2>/dev/null; then
FSTYPE=\"\$(findmnt -n -o FSTYPE /run 2>/dev/null || true)\"
if [[ \"\${FSTYPE}\" != \"tmpfs\" ]]; then
echo \"WARNING: /run is not tmpfs (found: \${FSTYPE})\" >&2
fi
fi
mkdir -p '${SECRETS_DIR}'
chmod 0700 '${SECRETS_DIR}'
chown '${AGENT_USER}:${AGENT_USER}' '${SECRETS_DIR}'
SETUP
)"
printf '%s' "${SETUP_SCRIPT}" | vm_exec_stdin bash
# ---------------------------------------------------------------------------
# Inject secrets via stdin (never in command arguments)
# ---------------------------------------------------------------------------
inject_secret() {
local filename="$1"
local content="$2"
local filepath="${SECRETS_DIR}/${filename}"
log " Writing ${filepath}"
# The remote script reads the secret from stdin.
# The secret is piped in and never appears in command arguments or ps output.
# The remote script body is passed as a -c argument so stdin is free for the secret.
local remote_script
remote_script="$(cat <<RSCRIPT
set -euo pipefail
FILEPATH='${filepath}'
AGENT='${AGENT_USER}'
SECRET="\$(cat)"
printf '%s' "\${SECRET}" > "\${FILEPATH}"
chmod 0600 "\${FILEPATH}"
chown "\${AGENT}:\${AGENT}" "\${FILEPATH}"
RSCRIPT
)"
printf '%s' "${content}" \
| HISTFILE=/dev/null ssh "${SSH_STDIN_OPTS[@]}" "root@${VM_IP}" bash -c "${remote_script}"
}
if [[ -n "${GITHUB_TOKEN}" ]]; then
inject_secret "github_token" "${GITHUB_TOKEN}"
fi
if [[ -n "${CLAUDE_AUTH}" ]]; then
inject_secret "claude_auth" "${CLAUDE_AUTH}"
fi
# ---------------------------------------------------------------------------
# Write env file that the agent sources at startup
# ---------------------------------------------------------------------------
log " Writing ${SECRETS_DIR}/env"
ENV_SCRIPT="$(cat <<ENVSCRIPT
set -euo pipefail
ENV_FILE='${SECRETS_DIR}/env'
{
printf '# Auto-generated by secrets-inject.sh -- do not edit\n'
printf '# Source this file to load agent credentials into the current shell.\n'
printf '\n'
ENVSCRIPT
)"
if [[ -n "${GITHUB_TOKEN}" ]]; then
ENV_SCRIPT+="$(cat <<GITHUB
printf 'export GITHUB_TOKEN="\$(cat ${SECRETS_DIR}/github_token)"\n'
GITHUB
)"
fi
if [[ -n "${CLAUDE_AUTH}" ]]; then
ENV_SCRIPT+="$(cat <<CLAUDE
printf 'export ANTHROPIC_API_KEY="\$(cat ${SECRETS_DIR}/claude_auth)"\n'
printf '# Uncomment for subscription-mode auth:\n'
printf '# export CLAUDE_AUTH_TOKEN="\$(cat ${SECRETS_DIR}/claude_auth)"\n'
CLAUDE
)"
fi
ENV_SCRIPT+="$(cat <<ENVEND
} > "\${ENV_FILE}"
chmod 0600 "\${ENV_FILE}"
chown '${AGENT_USER}:${AGENT_USER}' "\${ENV_FILE}"
ENVEND
)"
printf '%s' "${ENV_SCRIPT}" | vm_exec_stdin bash
# ---------------------------------------------------------------------------
# Verify injection
# ---------------------------------------------------------------------------
log "Verifying injection for ${AGENT_USER}"
# Build a list of expected secret files to verify
EXPECTED_FILES=()
[[ -n "${GITHUB_TOKEN}" ]] && EXPECTED_FILES+=("github_token")
[[ -n "${CLAUDE_AUTH}" ]] && EXPECTED_FILES+=("claude_auth")
EXPECTED_FILES+=("env")
VERIFY_SCRIPT="$(cat <<VERIFYSCRIPT
set -euo pipefail
ALL_OK=true
AGENT_USER='${AGENT_USER}'
SECRETS_DIR='${SECRETS_DIR}'
VERIFYSCRIPT
)"
for f in "${EXPECTED_FILES[@]}"; do
VERIFY_SCRIPT+="$(cat <<VFILE
fp="\${SECRETS_DIR}/${f}"
if [[ ! -f "\${fp}" ]]; then
printf 'FAIL: %s does not exist\n' "\${fp}" >&2
ALL_OK=false
else
perms="\$(stat -c '%a' "\${fp}" 2>/dev/null || stat -f '%OLp' "\${fp}" 2>/dev/null)"
owner="\$(stat -c '%U' "\${fp}" 2>/dev/null || stat -f '%Su' "\${fp}" 2>/dev/null)"
if [[ "\${perms}" != "600" ]]; then
printf 'FAIL: %s has permissions %s (expected 600)\n' "\${fp}" "\${perms}" >&2
ALL_OK=false
fi
if [[ "\${owner}" != "\${AGENT_USER}" ]]; then
printf 'FAIL: %s owned by %s (expected %s)\n' "\${fp}" "\${owner}" "\${AGENT_USER}" >&2
ALL_OK=false
fi
fi
VFILE
)"
done
VERIFY_SCRIPT+='
if [[ "${ALL_OK}" == "true" ]]; then
printf "OK: all secrets verified\n" >&2
else
exit 1
fi'
printf '%s' "${VERIFY_SCRIPT}" | vm_exec_stdin bash
log "secrets-inject complete for ${AGENT_USER} on ${VM_IP}"
lib/secrets-rotate.sh
#!/usr/bin/env bash
# secrets-rotate.sh — Rotate a GitHub PAT for a specific agent on a VM.
#
# Re-injects a new GitHub token for a single agent and verifies the token
# works by testing git remote access from the agent's workspace.
#
# Usage:
# secrets-rotate.sh <vm-ip> <agent-index> --github-token NEW_TOKEN
#
# Example:
# secrets-rotate.sh 1.2.3.4 0 --github-token ghp_newtoken
#
# Requirements:
# - SSH key loaded in ssh-agent (run secrets-init.sh first)
# - secrets-inject.sh in the same directory
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INJECT_SCRIPT="${SCRIPT_DIR}/secrets-inject.sh"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
log() { printf '[secrets-rotate] %s\n' "$*" >&2; }
die() { printf '[secrets-rotate] ERROR: %s\n' "$*" >&2; exit 1; }
usage() {
cat >&2 <<'USAGE'
Usage: secrets-rotate.sh <vm-ip> <agent-index> --github-token NEW_TOKEN
Arguments:
vm-ip IP address of the VM
agent-index Agent index (0-3)
Options:
--github-token TOKEN New GitHub personal access token (required)
USAGE
exit 1
}
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
[[ $# -ge 3 ]] || usage
VM_IP="$1"
AGENT_INDEX="$2"
shift 2
GITHUB_TOKEN=""
while [[ $# -gt 0 ]]; do
case "$1" in
--github-token)
GITHUB_TOKEN="$2"
shift 2
;;
--help|-h)
usage
;;
*)
die "Unknown argument: $1"
;;
esac
done
[[ -n "${GITHUB_TOKEN}" ]] || die "--github-token is required"
[[ "${AGENT_INDEX}" =~ ^[0-3]$ ]] || die "agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})"
[[ -n "${VM_IP}" ]] || die "vm-ip is required"
# ---------------------------------------------------------------------------
# Preflight checks
# ---------------------------------------------------------------------------
[[ -x "${INJECT_SCRIPT}" ]] || die "secrets-inject.sh not found or not executable at ${INJECT_SCRIPT}"
[[ -n "${SSH_AUTH_SOCK:-}" ]] || die "SSH_AUTH_SOCK is not set — is ssh-agent running?"
AGENT_USER="agent-${AGENT_INDEX}"
SECRETS_DIR="/run/secrets/${AGENT_USER}"
# ---------------------------------------------------------------------------
# Re-inject the new token
# ---------------------------------------------------------------------------
log "Rotating GitHub token for ${AGENT_USER} on ${VM_IP}"
bash "${INJECT_SCRIPT}" "${VM_IP}" "${AGENT_INDEX}" --github-token "${GITHUB_TOKEN}"
# ---------------------------------------------------------------------------
# Verify the new token works
# ---------------------------------------------------------------------------
log "Verifying new token with git ls-remote"
SSH_OPTS=(
-o BatchMode=yes
-o StrictHostKeyChecking=accept-new
-o ConnectTimeout=10
)
# Run git ls-remote as the agent user to confirm the new GITHUB_TOKEN works.
# The token is read from the secrets file so it never appears in the process list.
VERIFY_RESULT="$(HISTFILE=/dev/null ssh "${SSH_OPTS[@]}" "root@${VM_IP}" \
bash -s "${AGENT_USER}" "${SECRETS_DIR}" <<'VERIFY'
set -euo pipefail
AGENT_USER="$1"
SECRETS_DIR="$2"
# Run as the agent user so the test mirrors actual agent conditions
if [[ ! -f "${SECRETS_DIR}/github_token" ]]; then
echo "FAIL: github_token file not found" >&2
exit 1
fi
TOKEN_FILE="${SECRETS_DIR}/github_token"
# Test API access using the token (does not clone, just checks auth)
HTTP_STATUS="$(curl -s -o /dev/null -w '%{http_code}' \
-H "Authorization: token $(cat ${TOKEN_FILE})" \
https://api.github.com/user 2>/dev/null || echo "000")"
case "${HTTP_STATUS}" in
200)
echo "OK: GitHub API responded 200"
;;
401)
echo "FAIL: token rejected (401)" >&2
exit 1
;;
403)
echo "FAIL: token forbidden (403)" >&2
exit 1
;;
000)
echo "FAIL: no network response (curl error)" >&2
exit 1
;;
*)
echo "WARN: unexpected HTTP status ${HTTP_STATUS}" >&2
;;
esac
VERIFY
)"
log "${VERIFY_RESULT}"
log "secrets-rotate complete for ${AGENT_USER} on ${VM_IP}"
lib/workspace-assign.sh
#!/usr/bin/env bash
# workspace-assign.sh — Assign a GitHub issue to an agent on a VM.
#
# Usage:
# workspace-assign.sh <vm-ip> <agent-index> <issue-number> <issue-title> \
# [--repo OWNER/REPO] [--max-turns N]
#
# Arguments:
# vm-ip Public IP of the Hetzner Cloud VM
# agent-index 0-3, selects agent-N user on the VM
# issue-number GitHub issue number to assign
# issue-title Issue title (used to build the branch slug)
# --repo GitHub repo in owner/repo format (required)
# --max-turns Maximum agent turns before stopping (default: 200)
#
# Side effects:
# - Writes /home/agent-N/assignment.json on the VM
# - Creates a feature branch in /home/agent-N/workspace/<repo>
set -euo pipefail
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -lt 4 ]]; then
echo "Usage: $0 <vm-ip> <agent-index> <issue-number> <issue-title> [--repo OWNER/REPO] [--max-turns N]" >&2
exit 1
fi
VM_IP="$1"
AGENT_INDEX="$2"
ISSUE_NUMBER="$3"
ISSUE_TITLE="$4"
shift 4
REPO=""
MAX_TURNS=200
while [[ $# -gt 0 ]]; do
case "$1" in
--repo)
REPO="$2"
shift 2
;;
--max-turns)
MAX_TURNS="$2"
shift 2
;;
*)
echo "Unknown argument: $1" >&2
exit 1
;;
esac
done
if [[ -z "${REPO}" ]]; then
echo "Error: --repo OWNER/REPO is required" >&2
exit 1
fi
if ! [[ "$AGENT_INDEX" =~ ^[0-3]$ ]]; then
echo "Error: agent-index must be 0, 1, 2, or 3 (got: $AGENT_INDEX)" >&2
exit 1
fi
if ! [[ "$ISSUE_NUMBER" =~ ^[0-9]+$ ]]; then
echo "Error: issue-number must be a positive integer (got: $ISSUE_NUMBER)" >&2
exit 1
fi
if ! [[ "$MAX_TURNS" =~ ^[0-9]+$ ]]; then
echo "Error: --max-turns must be a positive integer (got: $MAX_TURNS)" >&2
exit 1
fi
AGENT_USER="agent-${AGENT_INDEX}"
AGENT_HOME="/home/${AGENT_USER}"
REPO_NAME=$(basename "${REPO}")
CLONE_DIR="${AGENT_HOME}/workspace/${REPO_NAME}"
SSH_KEY="${SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
# Build branch slug: lowercase, replace spaces/non-alphanumeric with hyphens,
# trim leading/trailing hyphens, collapse repeated hyphens.
# printf '%s', not echo: bash's builtin echo flag-parses a leading -n/-e,
# so a title of exactly "-n" would silently produce an empty slug.
SLUG=$(printf '%s' "${ISSUE_TITLE}" \
| tr '[:upper:]' '[:lower:]' \
| sed 's/[^a-z0-9]/-/g' \
| sed 's/-\{2,\}/-/g' \
| sed 's/^-//;s/-$//')
BRANCH_NAME="${ISSUE_NUMBER}-${SLUG}"
# ISO 8601 timestamp (UTC)
ASSIGNED_AT=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
# ---------------------------------------------------------------------------
# Helper: run a command on the VM as root
# ---------------------------------------------------------------------------
ssh_root() {
ssh -i "${SSH_KEY}" \
-o StrictHostKeyChecking=accept-new \
-o BatchMode=yes \
"root@${VM_IP}" "$@"
}
# ---------------------------------------------------------------------------
# Helper: run a command on the VM as agent-N (via root su)
# ---------------------------------------------------------------------------
ssh_agent() {
local cmd="$1"
ssh_root "su - ${AGENT_USER} -c $(printf '%q' "$cmd")"
}
echo "==> Assigning issue #${ISSUE_NUMBER} to ${AGENT_USER} on ${VM_IP}"
# ---------------------------------------------------------------------------
# Step 1: Verify the workspace clone exists before assigning
# ---------------------------------------------------------------------------
if ! ssh_root "test -d '${CLONE_DIR}/.git'"; then
echo "Error: workspace not found at ${CLONE_DIR}. Run workspace-setup.sh first." >&2
exit 1
fi
# ---------------------------------------------------------------------------
# Step 2: Write assignment.json
# ---------------------------------------------------------------------------
echo "--> Writing assignment.json"
ASSIGNMENT_FILE="${AGENT_HOME}/assignment.json"
ASSIGNMENT_JSON=$(cat <<JSON
{
"issue_number": ${ISSUE_NUMBER},
"issue_title": "${ISSUE_TITLE}",
"repo": "${REPO}",
"branch": "${BRANCH_NAME}",
"max_turns": ${MAX_TURNS},
"assigned_at": "${ASSIGNED_AT}"
}
JSON
)
ssh_root "printf '%s\n' '${ASSIGNMENT_JSON}' > '${ASSIGNMENT_FILE}' && chown '${AGENT_USER}:${AGENT_USER}' '${ASSIGNMENT_FILE}'"
# ---------------------------------------------------------------------------
# Step 3: Create the feature branch
# ---------------------------------------------------------------------------
echo "--> Creating branch ${BRANCH_NAME} from origin/main"
ssh_agent "git -C '${CLONE_DIR}' fetch origin"
ssh_agent "git -C '${CLONE_DIR}' checkout -b '${BRANCH_NAME}' origin/main"
# ---------------------------------------------------------------------------
# Step 4: Verify
# ---------------------------------------------------------------------------
echo "--> Verifying branch"
CURRENT_BRANCH=$(ssh_agent "git -C '${CLONE_DIR}' rev-parse --abbrev-ref HEAD")
if [[ "${CURRENT_BRANCH}" != "${BRANCH_NAME}" ]]; then
echo "Error: expected branch ${BRANCH_NAME}, got ${CURRENT_BRANCH}" >&2
exit 1
fi
echo "==> Assignment complete"
echo " Agent: ${AGENT_USER}@${VM_IP}"
echo " Issue: #${ISSUE_NUMBER} - ${ISSUE_TITLE}"
echo " Branch: ${BRANCH_NAME}"
echo " Repo: ${REPO}"
lib/workspace-cleanup.sh
#!/usr/bin/env bash
# workspace-cleanup.sh — Remove workspace artifacts from one or all agent users on a VM.
#
# Usage:
# workspace-cleanup.sh <vm-ip> <agent-index>
# workspace-cleanup.sh <vm-ip> --all
#
# Arguments:
# vm-ip Public IP of the Hetzner Cloud VM
# agent-index 0-3, selects agent-N user to clean up
# --all Clean up all 4 agent users on the VM
#
# What is removed (per agent):
# - /home/agent-N/workspace/ (repo clone)
# - /home/agent-N/assignment.json
# - /home/agent-N/run.log
#
# What is preserved:
# - /home/agent-N/ (home directory)
# - /home/agent-N/.gitconfig (identity config, reused on next assignment)
# - /home/agent-N/.claude/ (CCGM settings, reused on next assignment)
# - /home/agent-N/.ssh/ (SSH config)
# - /home/agent-N/.bashrc
set -euo pipefail
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -lt 2 ]]; then
echo "Usage: $0 <vm-ip> <agent-index>" >&2
echo " $0 <vm-ip> --all" >&2
exit 1
fi
VM_IP="$1"
AGENT_ARG="$2"
SSH_KEY="${SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
AGENTS_PER_VM=4
# Build list of agent indexes to clean
declare -a AGENT_INDEXES
if [[ "${AGENT_ARG}" == "--all" ]]; then
for i in $(seq 0 $(( AGENTS_PER_VM - 1 ))); do
AGENT_INDEXES+=("$i")
done
else
if ! [[ "${AGENT_ARG}" =~ ^[0-3]$ ]]; then
echo "Error: agent-index must be 0, 1, 2, or 3 (got: ${AGENT_ARG})" >&2
exit 1
fi
AGENT_INDEXES=("${AGENT_ARG}")
fi
# ---------------------------------------------------------------------------
# Helper: run a command on the VM as root
# ---------------------------------------------------------------------------
ssh_root() {
ssh -i "${SSH_KEY}" \
-o StrictHostKeyChecking=accept-new \
-o BatchMode=yes \
-o ConnectTimeout=10 \
"root@${VM_IP}" "$@"
}
# ---------------------------------------------------------------------------
# Clean one agent
# ---------------------------------------------------------------------------
cleanup_one() {
local agent_index="$1"
local agent_user="agent-${agent_index}"
local agent_home="/home/${agent_user}"
local workspace_dir="${agent_home}/workspace"
local assignment_file="${agent_home}/assignment.json"
local run_log="${agent_home}/run.log"
echo "==> Cleaning up ${agent_user}@${VM_IP}"
# Stop any running agent process owned by this user before removing files
local pids
pids=$(ssh_root "pgrep -u '${agent_user}' -f 'claude' 2>/dev/null || true")
if [[ -n "${pids}" ]]; then
echo "--> Stopping running agent process(es): ${pids}"
ssh_root "pkill -u '${agent_user}' -f 'claude' 2>/dev/null || true"
# Give process a moment to terminate
sleep 2
fi
# Remove workspace (repo clone)
if ssh_root "test -d '${workspace_dir}'"; then
echo "--> Removing workspace: ${workspace_dir}"
ssh_root "rm -rf '${workspace_dir}'"
else
echo "--> Workspace not found, skipping: ${workspace_dir}"
fi
# Remove assignment.json
if ssh_root "test -f '${assignment_file}'"; then
echo "--> Removing assignment file: ${assignment_file}"
ssh_root "rm -f '${assignment_file}'"
else
echo "--> Assignment file not found, skipping"
fi
# Remove run log
if ssh_root "test -f '${run_log}'"; then
echo "--> Removing run log: ${run_log}"
ssh_root "rm -f '${run_log}'"
else
echo "--> Run log not found, skipping"
fi
echo "--> ${agent_user} cleaned up"
}
# ---------------------------------------------------------------------------
# Execute cleanup
# ---------------------------------------------------------------------------
for agent_index in "${AGENT_INDEXES[@]}"; do
cleanup_one "${agent_index}"
done
echo ""
echo "==> Cleanup complete on ${VM_IP} (agents: ${AGENT_INDEXES[*]})"
lib/workspace-collect.sh
#!/usr/bin/env bash
# workspace-collect.sh — Collect results from agent workspaces.
#
# Usage:
# workspace-collect.sh --all
# workspace-collect.sh <vm-ip> <agent-index>
#
# Options:
# --all Collect from all running ccgm-agent-* VMs (all 4 agents each)
# --json Emit machine-readable JSON instead of formatted text
# --log-lines N Number of log tail lines to include (default: 50)
#
# Requires:
# - hcloud CLI (when --all is used)
# - SSH key in ~/.ssh/id_ed25519 or $SSH_KEY_PATH
set -euo pipefail
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --all [--json] [--log-lines N]" >&2
echo " $0 <vm-ip> <agent-index> [--json] [--log-lines N]" >&2
exit 1
fi
COLLECT_ALL=false
VM_IP=""
AGENT_INDEX=""
JSON_OUTPUT=false
LOG_LINES=50
while [[ $# -gt 0 ]]; do
case "$1" in
--all)
COLLECT_ALL=true
shift
;;
--json)
JSON_OUTPUT=true
shift
;;
--log-lines)
LOG_LINES="$2"
shift 2
;;
-*)
echo "Unknown option: $1" >&2
exit 1
;;
*)
if [[ -z "${VM_IP}" ]]; then
VM_IP="$1"
elif [[ -z "${AGENT_INDEX}" ]]; then
AGENT_INDEX="$1"
else
echo "Unexpected argument: $1" >&2
exit 1
fi
shift
;;
esac
done
if [[ "${COLLECT_ALL}" == "false" ]]; then
if [[ -z "${VM_IP}" || -z "${AGENT_INDEX}" ]]; then
echo "Error: provide --all or both <vm-ip> and <agent-index>" >&2
exit 1
fi
if ! [[ "${AGENT_INDEX}" =~ ^[0-3]$ ]]; then
echo "Error: agent-index must be 0, 1, 2, or 3 (got: ${AGENT_INDEX})" >&2
exit 1
fi
fi
SSH_KEY="${SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
AGENTS_PER_VM=4
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
ssh_root() {
local ip="$1"; shift
ssh -i "${SSH_KEY}" \
-o StrictHostKeyChecking=accept-new \
-o BatchMode=yes \
-o ConnectTimeout=10 \
"root@${ip}" "$@" 2>/dev/null
}
# Collect data for one agent and emit a result block
collect_one() {
local vm_ip="$1"
local agent_index="$2"
local agent_user="agent-${agent_index}"
local agent_home="/home/${agent_user}"
local assignment_file="${agent_home}/assignment.json"
local run_log="${agent_home}/run.log"
# --- assignment ---
local issue_number="" issue_title="" repo="" branch="" assigned_at=""
if ssh_root "${vm_ip}" "test -f '${assignment_file}'" 2>/dev/null; then
local assignment
assignment=$(ssh_root "${vm_ip}" "cat '${assignment_file}'" 2>/dev/null || echo "{}")
issue_number=$(echo "${assignment}" | grep -o '"issue_number":[^,}]*' | sed 's/"issue_number"://' | tr -d ' "')
issue_title=$(echo "${assignment}" | grep -o '"issue_title":"[^"]*"' | sed 's/"issue_title":"//;s/"$//')
repo=$(echo "${assignment}" | grep -o '"repo":"[^"]*"' | sed 's/"repo":"//;s/"$//')
branch=$(echo "${assignment}" | grep -o '"branch":"[^"]*"' | sed 's/"branch":"//;s/"$//')
assigned_at=$(echo "${assignment}" | grep -o '"assigned_at":"[^"]*"' | sed 's/"assigned_at":"//;s/"$//')
fi
# --- git state ---
local current_branch="" last_commit_sha="" last_commit_msg=""
local workspace_dir="${agent_home}/workspace"
if [[ -n "${repo}" ]]; then
local repo_name
repo_name=$(basename "${repo}")
local clone_dir="${workspace_dir}/${repo_name}"
if ssh_root "${vm_ip}" "test -d '${clone_dir}/.git'" 2>/dev/null; then
current_branch=$(ssh_root "${vm_ip}" \
"su - ${agent_user} -c 'git -C ${clone_dir} rev-parse --abbrev-ref HEAD'" 2>/dev/null || echo "unknown")
last_commit_sha=$(ssh_root "${vm_ip}" \
"su - ${agent_user} -c 'git -C ${clone_dir} rev-parse --short HEAD'" 2>/dev/null || echo "unknown")
last_commit_msg=$(ssh_root "${vm_ip}" \
"su - ${agent_user} -c 'git -C ${clone_dir} log -1 --pretty=%s'" 2>/dev/null || echo "")
fi
fi
# --- PR detection ---
local pr_url=""
if [[ -n "${branch}" && -n "${repo}" ]]; then
# Check gh CLI on the VM, or fall back to grepping the run log
pr_url=$(ssh_root "${vm_ip}" \
"su - ${agent_user} -c 'gh pr list --repo ${repo} --head ${branch} --json url --jq .[0].url 2>/dev/null'" \
2>/dev/null || true)
if [[ -z "${pr_url}" && -f "${run_log}" ]]; then
pr_url=$(ssh_root "${vm_ip}" "grep -oE 'https://github.com/[^ ]*/pull/[0-9]+' '${run_log}' | tail -1" 2>/dev/null || true)
fi
fi
# --- agent process status ---
local agent_pid="" agent_status="unknown"
agent_pid=$(ssh_root "${vm_ip}" \
"pgrep -u ${agent_user} -f 'claude' | head -1" 2>/dev/null || true)
if [[ -n "${agent_pid}" ]]; then
agent_status="running"
elif [[ -f "${run_log}" ]]; then
# Check last exit code if logged
local last_exit
last_exit=$(ssh_root "${vm_ip}" "tail -1 '${run_log}'" 2>/dev/null || true)
if echo "${last_exit}" | grep -qi "exit.*0\|completed\|success"; then
agent_status="completed"
elif echo "${last_exit}" | grep -qi "exit\|error\|fail"; then
agent_status="failed"
else
agent_status="idle"
fi
fi
# --- run log tail ---
local log_tail=""
if ssh_root "${vm_ip}" "test -f '${run_log}'" 2>/dev/null; then
log_tail=$(ssh_root "${vm_ip}" "tail -n ${LOG_LINES} '${run_log}'" 2>/dev/null || true)
fi
# --- VM name (best effort) ---
local vm_name=""
if command -v hcloud &>/dev/null; then
vm_name=$(hcloud server list --output columns=name,ipv4 2>/dev/null \
| awk -v ip="${vm_ip}" '$2==ip {print $1}' || true)
fi
[[ -z "${vm_name}" ]] && vm_name="${vm_ip}"
# ---------------------------------------------------------------------------
# Output
# ---------------------------------------------------------------------------
if [[ "${JSON_OUTPUT}" == "true" ]]; then
# Escape strings for JSON (basic escaping)
json_escape() { printf '%s' "$1" | python3 -c "import json,sys; print(json.dumps(sys.stdin.read()))" 2>/dev/null || printf '"%s"' "$1"; }
printf '{\n'
printf ' "vm": %s,\n' "$(json_escape "${vm_name}")"
printf ' "agent": %s,\n' "$(json_escape "${agent_user}")"
printf ' "issue_number": %s,\n' "${issue_number:-null}"
printf ' "issue_title": %s,\n' "$(json_escape "${issue_title}")"
printf ' "repo": %s,\n' "$(json_escape "${repo}")"
printf ' "branch": %s,\n' "$(json_escape "${branch}")"
printf ' "current_branch": %s,\n' "$(json_escape "${current_branch}")"
printf ' "last_commit": %s,\n' "$(json_escape "${last_commit_sha} ${last_commit_msg}")"
printf ' "pr_url": %s,\n' "$(json_escape "${pr_url}")"
printf ' "status": %s,\n' "$(json_escape "${agent_status}")"
printf ' "assigned_at": %s\n' "$(json_escape "${assigned_at}")"
printf '}\n'
else
echo "---"
printf "Agent: %s / %s\n" "${vm_name}" "${agent_user}"
if [[ -n "${issue_number}" ]]; then
printf "Issue: #%s - %s\n" "${issue_number}" "${issue_title}"
printf "Repo: %s\n" "${repo}"
else
printf "Issue: (no assignment)\n"
fi
printf "Branch: %s\n" "${branch:-${current_branch:-(none)}}"
printf "Status: %s\n" "${agent_status}"
if [[ -n "${pr_url}" ]]; then
printf "PR: %s\n" "${pr_url}"
fi
if [[ -n "${last_commit_sha}" && "${last_commit_sha}" != "unknown" ]]; then
printf "Last commit: %s \"%s\"\n" "${last_commit_sha}" "${last_commit_msg}"
fi
if [[ -n "${assigned_at}" ]]; then
printf "Assigned: %s\n" "${assigned_at}"
fi
if [[ -n "${log_tail}" ]]; then
echo ""
echo "Log (last ${LOG_LINES} lines):"
while IFS= read -r log_line; do printf ' %s\n' "${log_line}"; done <<< "${log_tail}"
fi
echo ""
fi
}
# ---------------------------------------------------------------------------
# Single agent
# ---------------------------------------------------------------------------
if [[ "${COLLECT_ALL}" == "false" ]]; then
collect_one "${VM_IP}" "${AGENT_INDEX}"
exit 0
fi
# ---------------------------------------------------------------------------
# All agents across all running VMs
# ---------------------------------------------------------------------------
if ! command -v hcloud &>/dev/null; then
echo "Error: hcloud CLI is required for --all mode" >&2
exit 1
fi
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status | awk '$2=="running" && /ccgm-agent/ {print $1}' | sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
echo "No running ccgm-agent-* VMs found." >&2
exit 0
fi
if [[ "${JSON_OUTPUT}" == "true" ]]; then
echo "["
first=true
fi
for vm_name in "${VM_NAMES[@]}"; do
vm_ip=$(hcloud server describe "${vm_name}" --output format='{{.PublicNet.IPv4.IP}}')
for agent_index in $(seq 0 $(( AGENTS_PER_VM - 1 ))); do
if [[ "${JSON_OUTPUT}" == "true" && "${first}" != "true" ]]; then
echo ","
fi
collect_one "${vm_ip}" "${agent_index}"
first=false
done
done
if [[ "${JSON_OUTPUT}" == "true" ]]; then
echo "]"
fi
lib/workspace-setup-all.sh
#!/usr/bin/env bash
# workspace-setup-all.sh — Provision workspaces across all running agent VMs.
#
# Usage:
# workspace-setup-all.sh <repo-url> --issues "42,43,44,45,46,47,48,49"
#
# Arguments:
# repo-url HTTPS URL of the target repo (e.g. https://github.com/owner/repo)
# --issues Comma-separated list of issue numbers to distribute
# --titles Optional: JSON file mapping issue numbers to titles
# If omitted, titles are fetched from GitHub via gh CLI
# --repo GitHub repo in owner/repo format (derived from repo-url if omitted)
# --max-turns Maximum agent turns per assignment (default: 200)
# --branch Optional base branch for all workspaces (default: main)
# --dry-run Print the assignment plan without executing it
#
# Requires:
# - Hetzner Cloud CLI (hcloud) installed and authenticated
# - workspace-setup.sh and workspace-assign.sh in the same directory as this script
# - SSH key in ~/.ssh/id_ed25519 or $SSH_KEY_PATH
# - gh CLI authenticated (for fetching issue titles when --titles not provided)
#
# NOTE: If common.sh from Epic 3 is available in the same lib/ directory, source it
# for shared SSH helpers. This script defines its own helpers as a fallback
# since Epic 3 may not be merged when this runs.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Source common.sh if available (Epic 3 dependency — may not be merged yet)
if [[ -f "${SCRIPT_DIR}/common.sh" ]]; then
# shellcheck source=/dev/null
source "${SCRIPT_DIR}/common.sh"
fi
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -lt 1 ]]; then
echo "Usage: $0 <repo-url> --issues \"42,43,44\" [--titles FILE] [--repo OWNER/REPO] [--max-turns N] [--branch BRANCH] [--dry-run]" >&2
exit 1
fi
REPO_URL="$1"
shift
ISSUES_CSV=""
TITLES_FILE=""
REPO_OVERRIDE=""
MAX_TURNS=200
BRANCH=""
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case "$1" in
--issues)
ISSUES_CSV="$2"
shift 2
;;
--titles)
TITLES_FILE="$2"
shift 2
;;
--repo)
REPO_OVERRIDE="$2"
shift 2
;;
--max-turns)
MAX_TURNS="$2"
shift 2
;;
--branch)
BRANCH="$2"
shift 2
;;
--dry-run)
DRY_RUN=true
shift
;;
*)
echo "Unknown argument: $1" >&2
exit 1
;;
esac
done
if [[ -z "${ISSUES_CSV}" ]]; then
echo "Error: --issues is required" >&2
exit 1
fi
# Derive repo in owner/repo format from URL
if [[ -n "${REPO_OVERRIDE}" ]]; then
REPO="${REPO_OVERRIDE}"
else
# Strip https://github.com/ prefix and .git suffix. printf '%s', not echo:
# bash's builtin echo flag-parses a leading -n/-e, so a repo-url of
# exactly "-n" would silently strip to empty here (#946).
REPO=$(printf '%s' "${REPO_URL}" | sed 's|https://github.com/||;s|\.git$||')
fi
# SSH_KEY is passed via SSH_KEY_PATH env to child scripts (workspace-setup.sh, workspace-assign.sh)
export SSH_KEY_PATH="${SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
# ---------------------------------------------------------------------------
# Parse issue list
# ---------------------------------------------------------------------------
IFS=',' read -ra ISSUE_LIST <<< "${ISSUES_CSV}"
ISSUE_COUNT="${#ISSUE_LIST[@]}"
# ---------------------------------------------------------------------------
# Fetch issue titles
# ---------------------------------------------------------------------------
declare -A ISSUE_TITLES
if [[ -n "${TITLES_FILE}" ]]; then
# Load from JSON file: {"42": "feat: habit streaks", "43": "fix: login bug"}
if ! command -v jq &>/dev/null; then
echo "Error: jq is required when --titles is specified" >&2
exit 1
fi
while IFS= read -r line; do
num=$(echo "${line}" | jq -r '.key')
title=$(echo "${line}" | jq -r '.value')
ISSUE_TITLES["${num}"]="${title}"
done < <(jq -r 'to_entries[] | {key, value} | @json' "${TITLES_FILE}")
else
echo "==> Fetching issue titles from GitHub"
if ! command -v gh &>/dev/null; then
echo "Error: gh CLI is required to fetch issue titles. Install it or use --titles to provide them." >&2
exit 1
fi
for issue_num in "${ISSUE_LIST[@]}"; do
# printf '%s', not echo: bash's builtin echo flag-parses a leading -n/-e,
# so an issue number of exactly "-n" would silently strip to empty here.
issue_num=$(printf '%s' "${issue_num}" | tr -d '[:space:]')
title=$(gh issue view "${issue_num}" --repo "${REPO}" --json title --jq '.title' 2>/dev/null || echo "issue-${issue_num}")
ISSUE_TITLES["${issue_num}"]="${title}"
done
fi
# ---------------------------------------------------------------------------
# Discover running agent VMs via hcloud CLI
# ---------------------------------------------------------------------------
echo "==> Discovering running agent VMs"
if ! command -v hcloud &>/dev/null; then
echo "Error: hcloud CLI is required. Install it from https://github.com/hetznercloud/cli" >&2
exit 1
fi
# VMs created by the ccgm terraform config are named ccgm-agent-{location}-{n}
# List all running servers with the ccgm-agent prefix, sorted by name.
mapfile -t VM_NAMES < <(hcloud server list --output columns=name,status | awk '$2=="running" && /ccgm-agent/ {print $1}' | sort)
if [[ ${#VM_NAMES[@]} -eq 0 ]]; then
echo "Error: no running ccgm-agent-* VMs found. Create them with Terraform first." >&2
exit 1
fi
echo "Found ${#VM_NAMES[@]} VM(s): ${VM_NAMES[*]}"
# ---------------------------------------------------------------------------
# Build assignment plan: distribute issues round-robin across agent slots
#
# Slot order: VM0-agent0, VM0-agent1, VM0-agent2, VM0-agent3,
# VM1-agent0, VM1-agent1, ...
# ---------------------------------------------------------------------------
declare -a PLAN_VM_IPS
declare -a PLAN_AGENT_INDEXES
declare -a PLAN_ISSUE_NUMBERS
AGENTS_PER_VM=4
slot=0
for issue_num in "${ISSUE_LIST[@]}"; do
# printf '%s', same reasoning as the identical strip above (#946).
issue_num=$(printf '%s' "${issue_num}" | tr -d '[:space:]')
vm_index=$(( slot / AGENTS_PER_VM ))
agent_index=$(( slot % AGENTS_PER_VM ))
if [[ ${vm_index} -ge ${#VM_NAMES[@]} ]]; then
echo "Warning: more issues (${ISSUE_COUNT}) than agent slots ($(( ${#VM_NAMES[@]} * AGENTS_PER_VM ))). Stopping at slot ${slot}." >&2
break
fi
VM_NAME="${VM_NAMES[$vm_index]}"
VM_IP=$(hcloud server describe "${VM_NAME}" --output format='{{.PublicNet.IPv4.IP}}')
PLAN_VM_IPS+=("${VM_IP}")
PLAN_AGENT_INDEXES+=("${agent_index}")
PLAN_ISSUE_NUMBERS+=("${issue_num}")
slot=$(( slot + 1 ))
done
# ---------------------------------------------------------------------------
# Print assignment plan
# ---------------------------------------------------------------------------
echo ""
echo "Assignment plan:"
echo " Repo: ${REPO}"
echo " Issues: ${ISSUE_COUNT}"
echo " VMs: ${#VM_NAMES[@]}"
echo ""
printf " %-6s %-8s %-20s %s\n" "Issue" "Agent" "VM" "Branch"
printf " %-6s %-8s %-20s %s\n" "------" "--------" "--------------------" "------"
for i in "${!PLAN_ISSUE_NUMBERS[@]}"; do
num="${PLAN_ISSUE_NUMBERS[$i]}"
title="${ISSUE_TITLES[$num]:-issue-${num}}"
# printf '%s', not echo: bash's builtin echo flag-parses a leading -n/-e,
# so a title of exactly "-n" would silently produce an empty slug.
slug=$(printf '%s' "${title}" \
| tr '[:upper:]' '[:lower:]' \
| sed 's/[^a-z0-9]/-/g' \
| sed 's/-\{2,\}/-/g' \
| sed 's/^-//;s/-$//')
branch="${num}-${slug}"
printf " %-6s %-8s %-20s %s\n" "#${num}" "agent-${PLAN_AGENT_INDEXES[$i]}" "${PLAN_VM_IPS[$i]}" "${branch}"
done
echo ""
if [[ "${DRY_RUN}" == "true" ]]; then
echo "(--dry-run: no changes made)"
exit 0
fi
# ---------------------------------------------------------------------------
# Execute: setup workspace then assign issue for each slot
# ---------------------------------------------------------------------------
SETUP_SCRIPT="${SCRIPT_DIR}/workspace-setup.sh"
ASSIGN_SCRIPT="${SCRIPT_DIR}/workspace-assign.sh"
for script in "${SETUP_SCRIPT}" "${ASSIGN_SCRIPT}"; do
if [[ ! -x "${script}" ]]; then
echo "Error: ${script} not found or not executable" >&2
exit 1
fi
done
SUCCESS=0
FAILED=0
for i in "${!PLAN_ISSUE_NUMBERS[@]}"; do
vm_ip="${PLAN_VM_IPS[$i]}"
agent_index="${PLAN_AGENT_INDEXES[$i]}"
issue_num="${PLAN_ISSUE_NUMBERS[$i]}"
title="${ISSUE_TITLES[$issue_num]:-issue-${issue_num}}"
echo "==> [${i}] Setting up ${vm_ip} agent-${agent_index} for issue #${issue_num}"
setup_args=("${vm_ip}" "${agent_index}" "${REPO_URL}")
if [[ -n "${BRANCH}" ]]; then
setup_args+=(--branch "${BRANCH}")
fi
if "${SETUP_SCRIPT}" "${setup_args[@]}"; then
if "${ASSIGN_SCRIPT}" "${vm_ip}" "${agent_index}" "${issue_num}" "${title}" \
--repo "${REPO}" --max-turns "${MAX_TURNS}"; then
SUCCESS=$(( SUCCESS + 1 ))
else
echo "Error: workspace-assign.sh failed for issue #${issue_num} on ${vm_ip} agent-${agent_index}" >&2
FAILED=$(( FAILED + 1 ))
fi
else
echo "Error: workspace-setup.sh failed for ${vm_ip} agent-${agent_index}" >&2
FAILED=$(( FAILED + 1 ))
fi
done
echo ""
echo "==> Setup complete: ${SUCCESS} succeeded, ${FAILED} failed"
[[ ${FAILED} -eq 0 ]]
lib/workspace-setup.sh
#!/usr/bin/env bash
# workspace-setup.sh — Set up an agent workspace on a Hetzner Cloud VM.
#
# Usage:
# workspace-setup.sh <vm-ip> <agent-index> <repo-url> [--branch BRANCH]
#
# Arguments:
# vm-ip Public IP of the Hetzner Cloud VM
# agent-index 0-3, selects agent-N user on the VM
# repo-url HTTPS URL of the target repo (e.g. https://github.com/owner/repo)
# --branch Optional branch to check out after clone (default: main)
#
# Requires:
# - SSH access to the VM as root (key in ~/.ssh/id_ed25519 or SSH_KEY_PATH)
# - /run/secrets/agent-N/github_token populated on the VM before this runs
set -euo pipefail
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
if [[ $# -lt 3 ]]; then
echo "Usage: $0 <vm-ip> <agent-index> <repo-url> [--branch BRANCH]" >&2
exit 1
fi
VM_IP="$1"
AGENT_INDEX="$2"
REPO_URL="$3"
shift 3
BRANCH=""
while [[ $# -gt 0 ]]; do
case "$1" in
--branch)
BRANCH="$2"
shift 2
;;
*)
echo "Unknown argument: $1" >&2
exit 1
;;
esac
done
# Validate agent index
if ! [[ "$AGENT_INDEX" =~ ^[0-3]$ ]]; then
echo "Error: agent-index must be 0, 1, 2, or 3 (got: $AGENT_INDEX)" >&2
exit 1
fi
AGENT_USER="agent-${AGENT_INDEX}"
AGENT_HOME="/home/${AGENT_USER}"
WORKSPACE_DIR="${AGENT_HOME}/workspace"
SECRETS_DIR="/run/secrets/${AGENT_USER}"
SSH_KEY="${SSH_KEY_PATH:-${HOME}/.ssh/id_ed25519}"
# Extract repo name from URL (strip .git suffix if present)
REPO_NAME=$(basename "${REPO_URL}" .git)
# ---------------------------------------------------------------------------
# Helper: run a command on the VM as root
# ---------------------------------------------------------------------------
ssh_root() {
ssh -i "${SSH_KEY}" \
-o StrictHostKeyChecking=accept-new \
-o BatchMode=yes \
"root@${VM_IP}" "$@"
}
# ---------------------------------------------------------------------------
# Helper: run a command on the VM as agent-N (via root su)
# ---------------------------------------------------------------------------
ssh_agent() {
local cmd="$1"
ssh_root "su - ${AGENT_USER} -c $(printf '%q' "$cmd")"
}
echo "==> Setting up workspace for ${AGENT_USER} on ${VM_IP}"
# ---------------------------------------------------------------------------
# Step 1: Create workspace directory
# ---------------------------------------------------------------------------
echo "--> Creating workspace directory"
ssh_root "mkdir -p '${WORKSPACE_DIR}' && chown '${AGENT_USER}:${AGENT_USER}' '${WORKSPACE_DIR}'"
# ---------------------------------------------------------------------------
# Step 2: Clone the repo using the agent's GitHub token
# ---------------------------------------------------------------------------
echo "--> Cloning ${REPO_URL}"
CLONE_DIR="${WORKSPACE_DIR}/${REPO_NAME}"
# Build a token-authenticated URL from the plain HTTPS URL.
# The token is read on the VM from /run/secrets/agent-N/github_token — it
# never appears in the SSH command itself or in local shell variables.
CLONE_SCRIPT=$(cat <<'SCRIPT'
set -euo pipefail
AGENT_USER="__AGENT_USER__"
REPO_URL="__REPO_URL__"
CLONE_DIR="__CLONE_DIR__"
SECRETS_DIR="__SECRETS_DIR__"
TOKEN_FILE="${SECRETS_DIR}/github_token"
if [[ ! -f "${TOKEN_FILE}" ]]; then
echo "Error: token file not found at ${TOKEN_FILE}" >&2
exit 1
fi
GIT_TOKEN=$(cat "${TOKEN_FILE}")
# Inject token into HTTPS URL: https://<token>@github.com/owner/repo
AUTH_URL="${REPO_URL/https:\/\//https://${GIT_TOKEN}@}"
if [[ -d "${CLONE_DIR}/.git" ]]; then
echo "Repo already cloned at ${CLONE_DIR}, skipping"
else
git clone "${AUTH_URL}" "${CLONE_DIR}"
fi
SCRIPT
)
CLONE_SCRIPT="${CLONE_SCRIPT//__AGENT_USER__/$AGENT_USER}"
CLONE_SCRIPT="${CLONE_SCRIPT//__REPO_URL__/$REPO_URL}"
CLONE_SCRIPT="${CLONE_SCRIPT//__CLONE_DIR__/$CLONE_DIR}"
CLONE_SCRIPT="${CLONE_SCRIPT//__SECRETS_DIR__/$SECRETS_DIR}"
ssh_root "su - ${AGENT_USER} -c 'bash -s'" <<< "${CLONE_SCRIPT}"
# ---------------------------------------------------------------------------
# Step 3: Configure git identity for the agent
# ---------------------------------------------------------------------------
echo "--> Configuring git identity"
ssh_agent "git -C '${CLONE_DIR}' config user.name 'CCGM Agent ${AGENT_INDEX}'"
ssh_agent "git -C '${CLONE_DIR}' config user.email 'ccgm-agent-${AGENT_INDEX}@dispatch.local'"
# ---------------------------------------------------------------------------
# Step 4: Checkout branch (if specified)
# ---------------------------------------------------------------------------
if [[ -n "${BRANCH}" ]]; then
echo "--> Checking out branch: ${BRANCH}"
ssh_agent "git -C '${CLONE_DIR}' checkout '${BRANCH}'"
fi
# ---------------------------------------------------------------------------
# Step 5: Set up CCGM config (Claude Code settings for headless mode)
# ---------------------------------------------------------------------------
echo "--> Writing Claude Code settings"
CLAUDE_DIR="${AGENT_HOME}/.claude"
ssh_root "mkdir -p '${CLAUDE_DIR}' && chown '${AGENT_USER}:${AGENT_USER}' '${CLAUDE_DIR}'"
# Write settings.json enabling dangerously-skip-permissions for headless runs.
# See: https://docs.anthropic.com/en/docs/claude-code/settings
SETTINGS_JSON='{
"permissions": {
"allow": ["Bash", "Read", "Write", "Edit", "Glob", "Grep"],
"deny": []
},
"dangerouslySkipPermissions": true
}'
ssh_root "printf '%s\n' '${SETTINGS_JSON}' > '${CLAUDE_DIR}/settings.json' && chown '${AGENT_USER}:${AGENT_USER}' '${CLAUDE_DIR}/settings.json'"
# Refresh CCGM installation from the repo on the VM.
# The golden image pre-installs CCGM, but this ensures rules are current
# if the image is stale. Also re-runs headless install to pick up any new modules.
HEADLESS_INSTALLER="/opt/ccgm/repo/modules/cloud-dispatch/lib/ccgm-headless-install.sh"
ssh_root "
if [[ -x '${HEADLESS_INSTALLER}' ]]; then
echo 'Refreshing CCGM cloud-agent preset for ${AGENT_USER}'
bash '${HEADLESS_INSTALLER}' /opt/ccgm/repo cloud-agent '${AGENT_HOME}'
chown -R '${AGENT_USER}:${AGENT_USER}' '${CLAUDE_DIR}'
else
echo 'WARN: CCGM headless installer not found at ${HEADLESS_INSTALLER}'
echo 'Agents will run without CCGM rules. Rebuild golden image to fix.'
fi
"
# ---------------------------------------------------------------------------
# Step 6: Verify
# ---------------------------------------------------------------------------
echo "--> Verifying workspace"
ssh_agent "git -C '${CLONE_DIR}' status"
ssh_agent "git -C '${CLONE_DIR}' config user.email"
echo "==> Workspace setup complete: ${AGENT_USER}@${VM_IP} -> ${CLONE_DIR}"