Security Model

This page is the authoritative reference for the Remote SSH security model. It describes every invariant that the server enforces and every control that an administrator can configure.

Role separation invariant — the AND-gate

OTA admin / org-owner roles do NOT grant shell access. This is the feature’s core invariant and is enforced server-side on every session-open request.

A user may open a shell session if and only if both of the following conditions hold simultaneously:

1. The user holds an active device.shell.<tier> role grant for the organisation (ShellRoleGrant row with revoked_at IS NULL).
2. The user is an active Shell Access Group member for the organisation (ShellAccessGroupMember row with removed_at IS NULL).

Neither condition alone is sufficient. A user who is an org OWNER or ADMIN but has no shell role grant is rejected with HTTP 403. A user who has a shell role grant but is not in the Shell Access Group is also rejected with HTTP 403.

Code path: app/api/sessions.py → open_session calls enforce_shell_access_for_tier(db, current_user, organization, body.tier) (imported from app/core/deps.py) at step 1, before any DB side effects. The underlying checks are user_has_shell_role and user_in_shell_access_group from app/services/shell_authz.py.

The AND-gate applies to all three tiers. There is no bypass path — no other code in sessions.py creates session records.

Shell roles

Role	What it grants
device.shell.standard	Open a Standard-tier session (non-root diagnostic user).
device.shell.elevated	Open an Elevated-tier session (root).
device.shell.breakglass	Open a Break-Glass-tier session (root, emergency).
device.shell.approve	Approve or deny pending peer-approval requests.
audit.recording.view	View (replay) a session recording via GET /sessions/{id}/recording.
audit.recording.export	Export a recording for compliance/SIEM use via GET /sessions/{id}/recording/export.

Role grants are managed via app/api/rbac.py:

• POST /users/{user_id}/roles?organization_id=<org> — grant a role.
• DELETE /users/{user_id}/roles?organization_id=<org>&role=<role> — revoke a role.

Granting device.shell.elevated or device.shell.breakglass to a user who already holds an org-tenancy OWNER or ADMIN role emits a separation-of-duties (sod_warning) field in the grant response. This is a warning only; it does not block the grant. The fleet owner is notified regardless (shell_audit_notify.py).

---

Tiers

Three tiers exist. The tier controls the OS user the gateway connects as, whether recording is mandatory, the approval mode, and the session time-box.

Standard

• OS user: configurable non-root user (e.g. support) with a curated sudoers allowlist baked into the firmware via OTA. The operator cannot modify sudoers at session time.
• Recording: optional. Controlled by standard_recording_enabled in the org’s ShellAccessPolicy. Default: off.
• Approval: none. Sessions become ACTIVE immediately.
• Justification: optional.
• Role required: device.shell.standard.

Elevated

• OS user: root.
• Recording: mandatory. Cannot be disabled by the org policy or the operator. The recording flag is set to True on the ShellSession row at creation and is never re-evaluated (app/api/sessions.py, step 4a).
• Approval: configurable per org: none, self_justification, or peer.
  • none — session becomes ACTIVE immediately.
  • self_justification — session becomes ACTIVE immediately; an Approval row with decision='approved' and approver_id=current_user.id is written for the audit trail.
  • peer — session stays PENDING until a second user holding device.shell.approve calls POST /sessions/{id}/approve. Self-approval is always rejected (HTTP 400).
• Justification: required. Empty/whitespace-only input returns HTTP 422.
• Session time-box: elevated_session_timeout_minutes in policy (default 30 min; max 1440 min / 24 h).
• Approval timeout: elevated_approval_timeout_minutes in policy (default 15 min; max 1440 min).
• Role required: device.shell.elevated.

Break-Glass

• OS user: root.
• Recording: mandatory. Same enforcement as Elevated.
• Approval: follows the org’s elevated_approval_mode setting (the same field applies to both Elevated and Break-Glass).
• Justification: required.
• Session time-box: breakglass_session_timeout_minutes in policy (default 60 min; max 1440 min).
• Incident record: a ShellIncident row is created immediately after the session row commits and before the gateway call, so the audit trail is intact even if the gateway is unreachable (app/api/sessions.py, step 7, create_breakglass_incident).
• Notifications: immediate fan-out to email + webhook (whenever set up) and PagerDuty (only when pagerduty is listed in breakglass_notification_channels and the server has PAGERDUTY_INTEGRATION_KEY). The default list is ["email"], so PagerDuty is off until an admin lists it. The server logs a WARNING when the key is set but pagerduty is not listed.
• Role required: device.shell.breakglass (grant sparingly).

Policy schema

The org’s ShellAccessPolicy row is read and written via:

• GET /organizations/{org_id}/access-policy (or /fleets/{fleet_id}/access-policy)
• PUT /organizations/{org_id}/access-policy (OWNER / ADMIN only)

Key fields (app/api/access_policy.py, AccessPolicyResponse):

Field	Type	Default	Description
allowed_tiers	string[]	["standard"]	Tiers operators may request. Always applied by the console; applied by the API at session open only when SHELL_TIER_POLICY_ENFORCE=true. An organization with no saved policy allows standard only.
standard_recording_enabled	bool	false	Record Standard sessions.
elevated_approval_mode	string	"none"	none, self_justification, or peer.
elevated_approval_timeout_minutes	int	15	How long a peer-approval request waits before expiry.
elevated_session_timeout_minutes	int	30	Hard session cap for Elevated.
breakglass_enabled	bool	false	Master switch for Break-Glass. Enforced at session open only when the server runs with SHELL_TIER_POLICY_ENFORCE=true; with it unset (the default) the server only logs a warning for each open it would have refused, and Break-Glass is gated by the device.shell.breakglass role grant and Shell Access Group membership. When enforced, Break-Glass also requires breakglass in allowed_tiers.
breakglass_notification_channels	string[]	["email"]	Controls the PagerDuty extra on Break-Glass open (default and no-policy-row: none). Email and webhook are sent whenever set up.
breakglass_session_timeout_minutes	int	60	Hard session cap for Break-Glass.
additions_require_approval	bool	true	Whether adding a first Shell Access Group member requires a confirmation step.
recording_retention_days	int	90	How long to keep recordings before the daily retention job deletes them.
recording_storage_quota_bytes	int?	null (unlimited)	Per-org recording storage cap in bytes.
webhook_url	string?	null	Notification webhook. Accepted on PUT only after an SSRF check (HTTPS, public addresses only); returned by GET masked as https://host/….
webhook_secret	string?	null	Write-only HMAC key, encrypted at rest. GET returns only has_webhook_secret. Never written to the audit log.

---

Recording and immutability

Where capture happens

Recording is performed at the gateway, not on the device. The gateway’s AsciinemaRecorder (gateway/app/recorder.py) intercepts every PTY byte flowing through the SSH bridge in both directions (device output and operator input). A compromised device or a rogue operator cannot disable or tamper with the recording because the capture path runs entirely outside the device.

RecorderError (disk full, permission denied, etc.) is treated as fatal: the bridge terminates the session rather than silently continuing unrecorded.

Format and storage

• Format: asciinema v2 (.cast). Line 1 is a JSON header; subsequent lines are [seconds, "o"|"i"|"r", ...] events.
• Storage: S3-compatible object store (MinIO in self-hosted deployments; any AWS S3-compatible endpoint). Configured via RECORDINGS_S3_* env vars.
• Per-org S3 prefix: recordings are stored under <org-id>/<session-id>/ so each organisation’s recordings are logically isolated within the bucket.
• Hash on completion: when the gateway uploads the .cast file it computes a SHA-256 digest and stores it in ShellSession.recording_sha256. This hash is also included in the recording.attached audit event in the hash chain.
• Presigned access: GET /sessions/{id}/recording returns a short-lived presigned URL (TTL configured by recording_presign_ttl_seconds). The download URL is never stored; it is minted on each request.
• Export gate: GET /sessions/{id}/recording/export requires the higher audit.recording.export role and emits a recording.exported event (distinct from recording.viewed) for compliance pipelines.

S3 object-lock

Live S3 object-lock (write-once WORM) for Break-Glass recordings is a deferred feature. See docs/remote-ssh/DEFERRED-VERIFICATION.md.

Retention

The daily retention job (app/services/retention.py, enforce_retention) deletes recordings whose ShellSession.created_at is older than the org’s recording_retention_days horizon. It:

1. Deletes the object from S3 first.
2. Only if the S3 delete succeeds: NULLs recording_uri, recording_sha256, and recording_size_bytes on the DB row and appends a recording.deleted audit event in the same transaction.

If the S3 delete fails, the DB row is left untouched (atomicity preserved). The job runs as a background asyncio loop (interval 24 h) wired into app.main.lifespan.

Storage quotas

When recording_storage_quota_bytes is set, the check_quota_alerts function (app/services/retention.py) fires a quota.alert audit event when the org’s total recording usage reaches 80% of the quota. Alerts are de-duplicated within a 24-hour window. The org’s current_usage_bytes is returned on every GET /access-policy call so the admin can monitor usage without querying the audit log.

---

Audit chain

Structure

Audit events are stored in ShellAuditEvent rows. Each event carries:

• prev_hash — the event_hash of the immediately preceding event for this org, or "0" * 64 (the genesis constant) for the first event.
• event_hash — SHA-256 of a deterministic canonical JSON encoding of all event fields (sorted keys, UTC ISO-8601 timestamps, UUIDs as lowercase hyphenated strings).
• event_type — one of the constants below.
• actor_id, target_user_id, payload, reason, created_at.
• actor_ref — an immutable text copy of the actor (the actor’s id as a string). It is part of the hashed fields only for events that carry it (non-null); events written before it existed recompute to their original hash. API consumers that verify the chain independently must include actor_ref in the canonical encoding whenever it is non-null.

The chain is per-organisation and uses a PostgreSQL advisory transaction lock (pg_advisory_xact_lock(hashtext(org_id))) to serialise concurrent appenders. On SQLite (unit tests) the lock is skipped. See app/services/shell_audit.py, append_event.

Events

Event type	When it fires
session.requested	User calls POST /devices/{id}/sessions (any tier).
session.started	Session transitions to ACTIVE (immediately for none/self_justification modes; after peer approval for peer mode).
session.approved	A peer approver calls POST /sessions/{id}/approve.
session.denied	A peer approver calls POST /sessions/{id}/deny.
session.ended	Session ends. payload.exit_reason is one of normal, timeout, agent_crashed, agent_exited, gateway_failover.
session.rate_limited	Session-open request is rejected by rate limiter (per-user, per-device, or per-fleet bucket).
session.resumed	WebSocket reconnect after a network blip.
recording.attached	Gateway uploads the .cast file and registers the URI + SHA-256 on the session row.
recording.viewed	GET /sessions/{id}/recording is called.
recording.exported	GET /sessions/{id}/recording/export is called.
recording.deleted	Daily retention job deletes a recording from S3 and NULLs the DB row.
quota.alert	Org recording usage reaches 80% of the configured quota.
policy.changed	PUT /organizations/{org_id}/access-policy succeeds. Payload includes changes with before/after for every changed field.
shell.role.granted	POST /users/{user_id}/roles succeeds.
shell.role.revoked	DELETE /users/{user_id}/roles succeeds.
shell.group.member_added	User is added to the Shell Access Group.
shell.group.member_removed	User is removed from the Shell Access Group.

Chain verification

app/services/shell_audit.verify_chain(db, organization_id) replays the full chain for an org and returns True if every prev_hash links correctly and every event_hash can be recomputed from the stored fields. Use this in compliance reviews or after a suspected tamper event.

---

Approval workflows

Three approval modes are available, configured per-org via elevated_approval_mode:

Mode	Behaviour
none	Session becomes ACTIVE immediately. No Approval row is written.
self_justification	Session becomes ACTIVE immediately. An Approval row with decision='approved' and approver_id=requester.id is written for the audit trail. The session.started event lands in the same transaction as the approval.decided event.
peer	Session status is PENDING. An Approval row with decision='pending' is created. A notification is sent to users holding device.shell.approve. The session becomes ACTIVE only when a different user calls POST /sessions/{id}/approve. Self-approval is rejected (HTTP 400).

Approval timeout is governed by elevated_approval_timeout_minutes. When a peer-approval request expires without a decision, the session is cleaned up during the next reaper cycle.

HMAC-signed webhook notifications

When a webhook_url is set on the access policy (see the table above and the admin guide), outgoing POST requests are signed with HMAC-SHA256 (see app/services/notifications.py, lines 18–46 and 381–423). Three headers are sent on every webhook:

• X-OTAPulse-Signature: sha256=<hex> — HMAC-SHA256 over the signed input.
• X-OTAPulse-Timestamp: <unix_int> — request timestamp (signed; replay guard).
• X-OTAPulse-Event-Id: <uuid4> — idempotency key.

The signed input is f"{unix_ts}.".encode() + body — the ASCII unix timestamp from X-OTAPulse-Timestamp, the literal . separator, then the raw JSON request body bytes. The HMAC is keyed with the per-fleet webhook secret and the lowercase hex digest is placed in X-OTAPulse-Signature prefixed with sha256=. Verification:

1. Recompute HMAC-SHA256(secret, f"{header_ts}.".encode() + body).
2. Compare to the header value (constant-time compare).
3. Receivers are recommended to reject requests whose X-OTAPulse-Timestamp is more than a few minutes skewed from local clock to bound replay windows (typical receiver-side window: ±5 minutes; see notifications.py:38-41). This window is a receiver-side recommendation — the server signs the timestamp but does not enforce a window on outgoing requests.

Binding the timestamp into the HMAC means an attacker who captures a valid (body, signature) pair cannot replay it under a fresh timestamp: changing the header without re-signing invalidates the HMAC.

---

Token lifetimes and MFA

WS session token

When a session is opened, the backend calls the gateway’s internal mint endpoint to issue a WebSocket upgrade token (ws_token in SessionResponse). Properties:

• 32 bytes of URL-safe random entropy (256 bits).
• Single-use: consume() in gateway/app/session_tokens.py marks the token consumed on first WebSocket upgrade; any subsequent attempt returns None.
• Short TTL: 60 seconds (configurable via WS_TOKEN_TTL_SECONDS on the gateway).
• Never logged at any level.

Resume tokens

After a WebSocket reconnect (network blip), the gateway issues a resume token to the browser. Resume tokens are also single-use and short-lived. They are rate-limited per-session via gateway/app/resume_rate_limit.py.

MFA freshness gate

For Elevated and Break-Glass session opens, the server checks that the requesting user’s last MFA assertion is recent. The freshness window is shell_mfa_freshness_minutes in app.config.settings. When SHELL_MFA_ENFORCE=false (the default for dev environments), the check logs a warning but does not block the request. In production, set SHELL_MFA_ENFORCE=true to enforce the gate (app/api/sessions.py, step 0b).

Tier policy gate

At session open (app/api/sessions.py, step 1b, after the role and group gates) the server checks the organization’s Shell Access Policy: the requested tier must be in allowed_tiers, and Break-Glass also needs breakglass_enabled: true. An organization with no saved policy allows standard only. By default (SHELL_TIER_POLICY_ENFORCE=false) the check is log-only: the server writes a WOULD BLOCK warning for each open it would have refused and lets it proceed. Set SHELL_TIER_POLICY_ENFORCE=true to enforce it. Before enabling it, give every organization that needs Elevated or Break-Glass a saved policy row, otherwise those sessions are refused.

Rate limiting

Session creation is rate-limited across three independent buckets: per-user, per-device, and per-fleet (org). Exceeding any bucket returns HTTP 429 with a Retry-After header and emits a session.rate_limited audit event. See app/core/rate_limit.py.