Admin Guide

This guide is for org OWNERs and ADMINs. It covers every administrative task for the Remote SSH feature: setting policy, managing who can access devices, approving elevated sessions, reading the audit log, and configuring retention and notifications.

---

Setting the per-org access policy

The access policy governs which tiers are available, whether approval is required, session time-boxes, recording retention, and storage quotas.

Who can write: org OWNER or ADMIN only (Depends(org_owner) in app/api/access_policy.py).

Who can read: any org member.

API

GET  /fleets/{fleet_id}/access-policy
PUT  /fleets/{fleet_id}/access-policy

Both routes are aliases for the original /organizations/{org_id}/access-policy routes — fleet_id is the organisation UUID.

Console

Navigate to Settings → Fleet Policy in the console. The FleetPolicy page exposes the most common fields (allowed_tiers, standard_recording_enabled, elevated_approval_mode, recording_retention_days, breakglass_enabled). Remaining policy fields (session/approval timeouts, notification channels, additions-require-approval, storage quota) must currently be set via the API; UI coverage is tracked in a follow-up sprint.

Policy fields

Field	Type	Default	Description
allowed_tiers	string[]	all three	Tiers operators may request. Remove breakglass if you never want emergency access on this fleet.
standard_recording_enabled	bool	false	Record Standard sessions. Enable if you require a full audit trail for non-root access too.
elevated_approval_mode	string	none	none, self_justification, or peer. peer requires a second human before the session opens.
elevated_approval_timeout_minutes	int	15	How long a peer approval request waits before it expires.
elevated_session_timeout_minutes	int	30	Hard cap on Elevated session length (1–1440 minutes).
breakglass_enabled	bool	true	Master switch for Break-Glass. Set to false to completely disable emergency access.
breakglass_notification_channels	string[]	["email","webhook"]	Alert channels when a Break-Glass session opens. Valid values: email, webhook, pagerduty.
breakglass_session_timeout_minutes	int	60	Hard cap on Break-Glass session length.
additions_require_approval	bool	false	When true, adding the first member to the Shell Access Group requires a one-time confirmation step by a second admin.
recording_retention_days	int	90	Days to keep recordings. Range: 1–3650. After this, the daily job deletes them from S3.
recording_storage_quota_bytes	int?	null	Per-fleet storage cap in bytes. null = unlimited. Max value: 100 TiB.

Partial updates are supported — supply only the fields you want to change. Every PUT appends a policy.changed audit event with before/after values for each changed field.

Recommended production settings

For a production fleet:

{
  "allowed_tiers": ["standard", "elevated"],
  "elevated_approval_mode": "peer",
  "elevated_approval_timeout_minutes": 30,
  "elevated_session_timeout_minutes": 60,
  "breakglass_enabled": true,
  "breakglass_notification_channels": ["email", "webhook", "pagerduty"],
  "recording_retention_days": 365,
  "recording_storage_quota_bytes": 107374182400
}

This requires peer approval for all root access, retains recordings for a year, caps storage at 100 GiB, and routes Break-Glass alerts to PagerDuty.

---

Managing the Shell Access Group

The Shell Access Group is a separate construct from roles. Holding a device.shell.* role is necessary but not sufficient — the user must also be an active group member. This two-factor model prevents accidental shell access from broad role grants.

Listing members

GET /organizations/{org_id}/shell-access-group/members

Returns all active (removed_at IS NULL) members.

Adding a member

POST /organizations/{org_id}/shell-access-group/members
Body: { "user_id": "<uuid>" }

Requires OWNER or ADMIN. Adding a member appends a shell.group.member_added audit event in the same transaction. The fleet owner is notified via shell_audit_notify.notify_fleet_owners_on_privilege_event even if the change was made by a platform super-admin.

First-member confirmation: if additions_require_approval is true in the policy, adding the very first group member requires a secondary confirmation before the membership becomes active.

Separation of duties: an admin who adds themselves to the group cannot use that group membership to open a session without also holding the corresponding device.shell.* role. Both conditions must hold simultaneously.

Removing a member

DELETE /organizations/{org_id}/shell-access-group/members/{user_id}

Soft-deletes the membership (removed_at is set). Any active sessions for that user are force-terminated via dispatch_gateway_kills in app/services/shell_access_group.py. A shell.group.member_removed audit event is appended.

---

Granting device.shell.* roles to users

Shell roles are separate from the org tenancy roles (OWNER/ADMIN/MEMBER/VIEWER). Grant them via Settings → Users → Shell Roles or the API:

POST /users/{user_id}/roles?organization_id=<org>
Body: { "role": "device.shell.elevated" }

Valid shell roles: device.shell.standard, device.shell.elevated, device.shell.breakglass, device.shell.approve, audit.recording.view, audit.recording.export.

Separation of duties warning: granting device.shell.elevated or device.shell.breakglass to a user who already holds an org-tenancy OWNER or ADMIN role returns a sod_warning field in the response. This is a warning only; the grant still succeeds. Consider whether the same person should both configure policy (ADMIN) and hold emergency root access (breakglass).

Revoking a role

DELETE /users/{user_id}/roles?organization_id=<org>&role=device.shell.elevated

Soft-revokes the grant (revoked_at is set). Existing sessions for this user are not terminated automatically by the revoke — if you need to kill active sessions, use the session force-terminate endpoint. Future session-open attempts for the revoked tier will return 403.

---

Approving sessions (peer mode)

When elevated_approval_mode is peer, Elevated and Break-Glass session requests land in the Approvals inbox.

Finding pending requests

Navigate to Approvals in the top navigation, or call:

GET /sessions/approvals/pending?organization_id=<org>

Each entry (PendingApprovalResponse) shows:

• approval_id, session_id
• device_id
• tier (elevated or breakglass)
• requested_by (user id + email)
• requested_at, expires_at
• justification (the operator’s stated reason)

Approving a request

POST /sessions/{session_id}/approve?organization_id=<org>
Body: { "reason": "Verified ticket #1234, approved for 30-min root access" }

The reason field is optional (max 1000 chars) but recommended. On success:

1. The Approval row transitions to decision='approved'.
2. The ShellSession row transitions PENDING → ACTIVE.
3. session.approved and session.started audit events are appended atomically.
4. The requesting operator’s browser is unblocked and the terminal opens.

You must hold device.shell.approve. You cannot approve your own requests (HTTP 400 if you try).

Denying a request

POST /sessions/{session_id}/deny?organization_id=<org>
Body: { "reason": "No open ticket, request denied" }

The reason field is required (1–1000 chars). On success:

1. The Approval row transitions to decision='denied'.
2. The ShellSession row remains PENDING but a session.denied audit event is appended.
3. The requesting operator sees a “Session denied” notification.

---

Reading the audit log

API

GET /audit?organization_id=<org>&limit=100

Supports filters: user_id, device_id, tier, event_type, outcome, since, until. Pagination is keyset-based (use the cursor field from the response to fetch the next page).

Console

Navigate to Settings → Audit Log. Use the filter bar to narrow by user, device, event type, or date range. Export to CSV or JSON with the Export button.

Interpreting key events

What you see	What it means
session.requested then no session.started	Session was not approved (peer mode) or was rate-limited.
session.ended with exit_reason: agent_crashed	The device-side agent died. Check device health.
session.ended with exit_reason: timeout	The session reached its configured time-box.
shell.role.granted or shell.group.member_added for an unexpected user	Someone was granted shell access. Review immediately.
quota.alert	Your org is at ≥ 80% of its recording storage quota.
recording.deleted with age_days: N	Normal retention sweep.

Chain verification

To verify the chain has not been tampered with (verify_chain is an async coroutine, so call it from an async context or wrap with asyncio.run):

import asyncio
from app.services.shell_audit import verify_chain

async def _check(db, organization_id):
    return await verify_chain(db, organization_id)

# From sync code:
result = asyncio.run(_check(db, organization_id))
# Or, when already inside an async function:
#   result = await verify_chain(db, organization_id)

verify_chain returns True if every prev_hash links correctly and every event_hash can be recomputed from stored fields. It replays the entire chain for the org in chronological order. Run this as part of your periodic compliance review.

---

Retention and quotas

How the daily job runs

The retention job is an asyncio background task started in app.main.lifespan alongside the heartbeat reaper. It runs every 24 hours. The loop is implemented in app/services/retention.py as retention_daily_loop. It can also be triggered via the CLI shim at app/scripts/retention_job.py.

The job:

1. Finds all orgs with at least one recording.
2. For each org, reads recording_retention_days from ShellAccessPolicy (default 90 days if no policy row exists).
3. Deletes recordings whose ShellSession.created_at < now - retention_days from S3 first, then NULLs the DB row if the S3 delete succeeds.
4. Appends a recording.deleted audit event per deleted recording.

Failures (S3 unreachable, object not found) are logged and added to the job’s error report, but do not abort the sweep for other orgs.

Monitoring usage

The current_usage_bytes field on GET /fleets/{fleet_id}/access-policy shows the total bytes of non-deleted recordings for the org. Check this whenever you adjust recording_storage_quota_bytes.

The 80% quota alert

When recording_storage_quota_bytes is set and usage reaches 80%, the retention job fires a quota.alert audit event. Alerts are de-duplicated within a 24-hour window. The alert appears in the Audit Log with:

• payload.quota_bytes — your configured quota.
• payload.usage_bytes — current usage.
• payload.usage_pct — fraction (0.0–1.0+).

Recovering from an over-quota state

If usage exceeds the quota:

1. Lower recording_retention_days temporarily to trigger faster cleanup on the next retention run.
2. Or increase recording_storage_quota_bytes to raise the cap.
3. Or wait for the daily job to sweep aged recordings.

The server does not block new sessions when a quota is exceeded — the quota only controls alerting and retention behaviour. To enforce hard storage limits, configure bucket-level quotas in your S3/MinIO deployment.

---

Notifications and webhooks

Break-Glass notifications

When a Break-Glass session opens, the server dispatches alerts via every channel listed in breakglass_notification_channels:

• email — sent to the fleet owner’s registered address.
• webhook — HTTP POST to the configured webhook URL (see below).
• pagerduty — PagerDuty Events API v2 (requires PAGERDUTY_ROUTING_KEY in the server environment).

Elevated session notifications (start/end) are sent to the fleet owner’s email regardless of breakglass_notification_channels.

Wiring a webhook receiver

Set the webhook URL via the notification service configuration (environment variable or admin settings page — exact UI location depends on your deployment). The server sends a POST with a JSON body and three security headers:

X-OTAPulse-Signature: sha256=<hex-digest>
X-OTAPulse-Timestamp: <unix-seconds>
X-OTAPulse-Event-Id: <uuid4>
Content-Type: application/json

The signed input is f"{unix_ts}.".encode() + body — the ASCII unix timestamp from X-OTAPulse-Timestamp, a literal . separator, then the raw request body bytes. This binds the timestamp into the HMAC so a captured (body, signature) pair cannot be replayed under a fresh timestamp.

Verifying the signature on your side

import hmac, hashlib, time

def verify_webhook(secret: str, body: bytes, signature_header: str, timestamp_header: str) -> bool:
    # 1. Receiver-side recommendation: reject if timestamp is too skewed.
    #    The ±5-minute window is a receiver-side guideline (see
    #    notifications.py:38-41); the server does not enforce it on outbound.
    ts = int(timestamp_header)
    if abs(time.time() - ts) > 300:
        return False
    # 2. Recompute HMAC-SHA256 over the SIGNED INPUT (timestamp + '.' + body).
    signed = f"{ts}.".encode() + body
    expected = "sha256=" + hmac.new(
        secret.encode(), signed, hashlib.sha256
    ).hexdigest()
    # 3. Constant-time compare.
    return hmac.compare_digest(expected, signature_header)

The ±5-minute window is a receiver-side recommendation that bounds replay attempts. Always use a constant-time comparison to prevent timing attacks. The X-OTAPulse-Event-Id header is a UUID4 idempotency key — receivers SHOULD de-duplicate on it so a retried delivery is processed only once.

---

Pen-test invariants you should maintain

The Remote SSH feature ships with a formal pen-test plan and findings document. Review these periodically and after any significant infrastructure change:

• docs/remote-ssh/pen-test/PEN-TEST-PLAN.md — the full test plan: AND-gate bypass attempts, session token forgery, recording tamper, audit chain manipulation, rate-limit bypass, lateral org access.
• docs/remote-ssh/pen-test/FINDINGS.md — findings from previous runs, their severity, and remediation status.

Key invariants to hold in your own testing:

1. AND-gate bypass. An org OWNER with no shell role grant must receive HTTP 403 from POST /devices/{id}/sessions. Verify after any RBAC change.
2. Cross-org session access. A user in org A must receive HTTP 404 for any session or recording belonging to org B.
3. Self-approval in peer mode. The requester of a peer-mode session must receive HTTP 400 if they call /approve on their own request.
4. Recording access gate. A user without audit.recording.view must receive HTTP 403 from GET /sessions/{id}/recording.
5. Audit chain integrity. verify_chain must return True after any normal operation. A failing chain is a tamper signal.