Admin Guide

This guide is for org OWNERs and ADMINs. It covers every administrative task for the Remote SSH feature: setting policy, managing who can access devices, approving elevated sessions, reading the audit log, and configuring retention and notifications.

---

Setting the per-org access policy

The access policy governs which tiers are available, whether approval is required, session time-boxes, recording retention, and storage quotas.

Who can write: org OWNER or ADMIN only (Depends(org_owner) in app/api/access_policy.py).

Who can read: any org member.

API

GET  /fleets/{fleet_id}/access-policy
PUT  /fleets/{fleet_id}/access-policy

Both routes are aliases for the original /organizations/{org_id}/access-policy routes — fleet_id is the organisation UUID.

Console

Navigate to Fleet Policy (sidebar) in the console. The FleetPolicy page exposes the most common fields (allowed_tiers, standard_recording_enabled, elevated_approval_mode, recording_retention_days, breakglass_enabled). Remaining policy fields (session/approval timeouts, notification channels, additions-require-approval, storage quota) must currently be set via the API; UI coverage is tracked in a follow-up sprint.

Policy fields

Field	Type	Default	Description
allowed_tiers	string[]	["standard"]	Tiers operators may request. Always applied by the console; applied by the API at session open only when SHELL_TIER_POLICY_ENFORCE=true. An organization with no saved policy allows standard only.
standard_recording_enabled	bool	false	Record Standard sessions. Enable if you require a full audit trail for non-root access too.
elevated_approval_mode	string	none	none, self_justification, or peer. peer requires a second human before the session opens.
elevated_approval_timeout_minutes	int	15	How long a peer approval request waits before it expires.
elevated_session_timeout_minutes	int	30	Hard cap on Elevated session length (1–1440 minutes).
breakglass_enabled	bool	false	Master switch for Break-Glass. Enforced at session open only when the server runs with SHELL_TIER_POLICY_ENFORCE=true; with it unset (the default) the server only logs a warning for each open it would have refused, and Break-Glass is gated by the device.shell.breakglass role grant and Shell Access Group membership. When enforced, Break-Glass also requires breakglass in allowed_tiers.
breakglass_notification_channels	string[]	["email"]	Validated (email, webhook, pagerduty). Controls whether PagerDuty fires on Break-Glass open; the default ["email"] (also used when the organization has no saved policy) sends no PagerDuty alert. Email and webhook delivery do not depend on it (see Break-Glass notifications).
breakglass_session_timeout_minutes	int	60	Hard cap on Break-Glass session length.
additions_require_approval	bool	true	When true, adding the first member to the Shell Access Group requires a one-time confirmation step by a second admin.
recording_retention_days	int	90	Days to keep recordings. Range: 1–3650. After this, the daily job deletes them from S3.
recording_storage_quota_bytes	int?	null	Per-fleet storage cap in bytes. null = unlimited. Max value: 100 TiB.
webhook_url	string?	null	HTTPS endpoint that receives HMAC-signed notifications (see Wiring a webhook receiver). Write the full URL; GET returns it masked as https://host/…. null or "" clears it (and its secret).
webhook_secret	string?	null	Write-only. HMAC signing key, 8–100 bytes. Stored encrypted and never returned; GET shows only has_webhook_secret. Requires a webhook_url. null clears only the secret.

Partial updates are supported — supply only the fields you want to change. Every PUT appends a policy.changed audit event with before/after values for each changed field. The one exception is the webhook: the audit event records only set/unset for webhook_url and webhook_secret, never their values.

Recommended production settings

For a production fleet:

{
  "allowed_tiers": ["standard", "elevated", "breakglass"],
  "elevated_approval_mode": "peer",
  "elevated_approval_timeout_minutes": 30,
  "elevated_session_timeout_minutes": 60,
  "breakglass_enabled": true,
  "breakglass_notification_channels": ["email", "webhook", "pagerduty"],
  "recording_retention_days": 365,
  "recording_storage_quota_bytes": 107374182400
}

Break-Glass needs both breakglass in allowed_tiers and breakglass_enabled: true.

This requires peer approval for all root access, retains recordings for a year, caps storage at 100 GiB. It also lists pagerduty in the Break-Glass channels. PagerDuty alerts are sent because pagerduty is listed and the server has PAGERDUTY_INTEGRATION_KEY set.

---

Managing the Shell Access Group

The Shell Access Group is a separate construct from roles. Holding a device.shell.* role is necessary but not sufficient — the user must also be an active group member. This two-factor model prevents accidental shell access from broad role grants.

Listing members

GET /organizations/{org_id}/shell-access-group/members

Returns all active (removed_at IS NULL) members.

Adding a member

POST /organizations/{org_id}/shell-access-group/members
Body: { "user_id": "<uuid>" }

Requires OWNER or ADMIN. Adding a member appends a shell.group.member_added audit event in the same transaction. The fleet owner is notified via shell_audit_notify.notify_fleet_owners_on_privilege_event even if the change was made by a platform super-admin.

First-member confirmation: if additions_require_approval is true in the policy, adding the very first group member requires a secondary confirmation before the membership becomes active.

Separation of duties: an admin who adds themselves to the group cannot use that group membership to open a session without also holding the corresponding device.shell.* role. Both conditions must hold simultaneously.

Removing a member

DELETE /organizations/{org_id}/shell-access-group/members/{user_id}

Soft-deletes the membership (removed_at is set). Any active sessions for that user are force-terminated via dispatch_gateway_kills in app/services/shell_access_group.py. A shell.group.member_removed audit event is appended.

---

Granting device.shell.* roles to users

Shell roles are separate from the org tenancy roles (OWNER/ADMIN/MEMBER/VIEWER). Grant them via the Shell Access Group page (sidebar) or the API:

POST /users/{user_id}/roles?organization_id=<org>
Body: { "role": "device.shell.elevated" }

Valid shell roles: device.shell.standard, device.shell.elevated, device.shell.breakglass, device.shell.approve, audit.recording.view, audit.recording.export.

Separation of duties warning: granting device.shell.elevated or device.shell.breakglass to a user who already holds an org-tenancy OWNER or ADMIN role returns a sod_warning field in the response. This is a warning only; the grant still succeeds. Consider whether the same person should both configure policy (ADMIN) and hold emergency root access (breakglass).

Revoking a role

DELETE /users/{user_id}/roles?organization_id=<org>&role=device.shell.elevated

Soft-revokes the grant (revoked_at is set). Existing sessions for this user are not terminated automatically by the revoke — if you need to kill active sessions, use the session force-terminate endpoint. Future session-open attempts for the revoked tier will return 403.

---

Approving sessions (peer mode)

When elevated_approval_mode is peer, Elevated and Break-Glass session requests land in the Approvals inbox.

Finding pending requests

Navigate to Approvals in the top navigation, or call:

GET /sessions/approvals/pending?organization_id=<org>

Each entry (PendingApprovalResponse) shows:

• approval_id, session_id
• device_id
• tier (elevated or breakglass)
• requested_by (user id + email)
• requested_at, expires_at
• justification (the operator’s stated reason)

Approving a request

POST /sessions/{session_id}/approve?organization_id=<org>
Body: { "reason": "Verified ticket #1234, approved for 30-min root access" }

The reason field is optional (max 1000 chars) but recommended. On success:

1. The Approval row transitions to decision='approved'.
2. The ShellSession row transitions PENDING → ACTIVE.
3. session.approved and session.started audit events are appended atomically.
4. The requesting operator’s browser is unblocked and the terminal opens.

You must hold device.shell.approve. You cannot approve your own requests (HTTP 400 if you try).

Denying a request

POST /sessions/{session_id}/deny?organization_id=<org>
Body: { "reason": "No open ticket, request denied" }

The reason field is required (1–1000 chars). On success:

1. The Approval row transitions to decision='denied'.
2. The ShellSession row remains PENDING but a session.denied audit event is appended.
3. The requesting operator sees a “Session denied” notification.

---

Reading the audit log

API

GET /audit?organization_id=<org>&limit=100

Supports filters: user_id, device_id, tier, event_type, outcome, since, until. Pagination is keyset-based (use the cursor field from the response to fetch the next page).

Console

Navigate to Audit Log (sidebar). Use the filter bar to narrow by user, device, event type, or date range. Export to CSV or JSON with the Export button.

Interpreting key events

What you see	What it means
session.requested then no session.started	Session was not approved (peer mode) or was rate-limited.
session.ended with exit_reason: agent_crashed	The device-side agent died. Check device health.
session.ended with exit_reason: timeout	The session reached its configured time-box.
shell.role.granted or shell.group.member_added for an unexpected user	Someone was granted shell access. Review immediately.
quota.alert	Your org is at ≥ 80% of its recording storage quota.
recording.deleted with age_days: N	Normal retention sweep.

Chain verification

To verify the chain has not been tampered with (verify_chain is an async coroutine, so call it from an async context or wrap with asyncio.run):

import asyncio
from app.services.shell_audit import verify_chain

async def _check(db, organization_id):
    return await verify_chain(db, organization_id)

# From sync code:
result = asyncio.run(_check(db, organization_id))
# Or, when already inside an async function:
#   result = await verify_chain(db, organization_id)

verify_chain returns True if every prev_hash links correctly and every event_hash can be recomputed from stored fields. It replays the entire chain for the org in chronological order. Run this as part of your periodic compliance review.

---

Retention and quotas

How the daily job runs

The retention job is an asyncio background task started in app.main.lifespan alongside the heartbeat reaper. It runs every 24 hours. The loop is implemented in app/services/retention.py as retention_daily_loop. It can also be triggered via the CLI shim at app/scripts/retention_job.py.

The job:

1. Finds all orgs with at least one recording.
2. For each org, reads recording_retention_days from ShellAccessPolicy (default 90 days if no policy row exists).
3. Deletes recordings whose ShellSession.created_at < now - retention_days from S3 first, then NULLs the DB row if the S3 delete succeeds.
4. Appends a recording.deleted audit event per deleted recording.

Failures (S3 unreachable, object not found) are logged and added to the job’s error report, but do not abort the sweep for other orgs.

Monitoring usage

The current_usage_bytes field on GET /fleets/{fleet_id}/access-policy shows the total bytes of non-deleted recordings for the org. Check this whenever you adjust recording_storage_quota_bytes.

The 80% quota alert

When recording_storage_quota_bytes is set and usage reaches 80%, the retention job fires a quota.alert audit event. Alerts are de-duplicated within a 24-hour window. The alert appears in the Audit Log with:

• payload.quota_bytes — your configured quota.
• payload.usage_bytes — current usage.
• payload.usage_pct — fraction (0.0–1.0+).

Recovering from an over-quota state

If usage exceeds the quota:

1. Lower recording_retention_days temporarily to trigger faster cleanup on the next retention run.
2. Or increase recording_storage_quota_bytes to raise the cap.
3. Or wait for the daily job to sweep aged recordings.

The server does not block new sessions when a quota is exceeded — the quota only controls alerting and retention behaviour. To enforce hard storage limits, configure bucket-level quotas in your S3/MinIO deployment.

---

Notifications and webhooks

Break-Glass notifications

When a Break-Glass session opens, the server tries each of these channels. email and webhook are always attempted when they are set up (an approver address exists; a webhook URL is configured). pagerduty is sent only when pagerduty is listed in breakglass_notification_channels. The default list is ["email"] (an organization with no saved policy gets the same default), so add pagerduty to turn PagerDuty on. The server does not drop PagerDuty on purpose when the policy cannot be interpreted. A database outage still aborts the whole dispatch at the email step, and is logged.

• email — sent to the fleet owner’s registered address.
• webhook — HTTP POST to the configured webhook URL (see below).
• pagerduty — PagerDuty Events API v2. Needs pagerduty in breakglass_notification_channels and PAGERDUTY_INTEGRATION_KEY in the server environment. If either is missing the server skips PagerDuty. With PAGERDUTY_INTEGRATION_KEY unset it writes one INFO log line. With the key set but pagerduty not listed, it writes a WARNING naming the organization.

Elevated session notifications (start/end) are sent to the fleet owner’s email regardless of breakglass_notification_channels.

Wiring a webhook receiver

Set the webhook on the fleet’s access policy. As an OWNER or ADMIN:

curl -X PUT "$API/api/organizations/$ORG_ID/access-policy" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"webhook_url": "https://hooks.example.com/otapulse", "webhook_secret": "<random 32+ char string>"}'

The URL is checked when you save it and again on every delivery:

• It must use https, have no embedded credentials, and use port 443 or 8443.
• Every address it resolves to must be publicly routable. Loopback, private (RFC 1918), link-local, and other internal addresses are rejected with HTTP 422 (webhook_url rejected: …).
• Redirects are not followed.

The secret is encrypted at rest and is write-only. To rotate it, PUT a new webhook_secret. To remove the webhook, PUT {"webhook_url": null}.

Once a URL is set, it receives every Remote SSH notification for the fleet, not only Break-Glass openings. The server sends a POST with a JSON body and three security headers (the signature header is omitted if no secret is set):

X-OTAPulse-Signature: sha256=<hex-digest>
X-OTAPulse-Timestamp: <unix-seconds>
X-OTAPulse-Event-Id: <uuid4>
Content-Type: application/json

The signed input is f"{unix_ts}.".encode() + body — the ASCII unix timestamp from X-OTAPulse-Timestamp, a literal . separator, then the raw request body bytes. This binds the timestamp into the HMAC so a captured (body, signature) pair cannot be replayed under a fresh timestamp.

Verifying the signature on your side

import hmac, hashlib, time

def verify_webhook(secret: str, body: bytes, signature_header: str, timestamp_header: str) -> bool:
    # 1. Receiver-side recommendation: reject if timestamp is too skewed.
    #    The ±5-minute window is a receiver-side guideline (see
    #    notifications.py:38-41); the server does not enforce it on outbound.
    ts = int(timestamp_header)
    if abs(time.time() - ts) > 300:
        return False
    # 2. Recompute HMAC-SHA256 over the SIGNED INPUT (timestamp + '.' + body).
    signed = f"{ts}.".encode() + body
    expected = "sha256=" + hmac.new(
        secret.encode(), signed, hashlib.sha256
    ).hexdigest()
    # 3. Constant-time compare.
    return hmac.compare_digest(expected, signature_header)

The ±5-minute window is a receiver-side recommendation that bounds replay attempts. Always use a constant-time comparison to prevent timing attacks. The X-OTAPulse-Event-Id header is a UUID4 idempotency key — receivers SHOULD de-duplicate on it so a retried delivery is processed only once.

---

Pen-test invariants you should maintain

Review the invariants below periodically and after any significant infrastructure change. The pen-test plan and findings documents are not published with this documentation site; ask your OTA-Pulse contact if you need them.

Key invariants to hold in your own testing:

1. AND-gate bypass. An org OWNER with no shell role grant must receive HTTP 403 from POST /devices/{id}/sessions. Verify after any RBAC change.
2. Cross-org session access. A user in org A must receive HTTP 404 for any session or recording belonging to org B.
3. Self-approval in peer mode. The requester of a peer-mode session must receive HTTP 400 if they call /approve on their own request.
4. Recording access gate. A user without audit.recording.view must receive HTTP 403 from GET /sessions/{id}/recording.
5. Audit chain integrity. verify_chain must return True after any normal operation. A failing chain is a tamper signal.