Security Model

This page is the authoritative reference for the Remote SSH security model. It describes every invariant that the server enforces and every control that an administrator can configure.

Role separation invariant — the AND-gate

OTA admin / org-owner roles do NOT grant shell access. This is the feature’s core invariant and is enforced server-side on every session-open request.

A user may open a shell session if and only if both of the following conditions hold simultaneously:

The user holds an active device.shell.<tier> role grant for the organisation (ShellRoleGrant row with revoked_at IS NULL).
The user is an active Shell Access Group member for the organisation (ShellAccessGroupMember row with removed_at IS NULL).

Neither condition alone is sufficient. A user who is an org OWNER or ADMIN but has no shell role grant is rejected with HTTP 403. A user who has a shell role grant but is not in the Shell Access Group is also rejected with HTTP 403.

Code path: app/api/sessions.py → open_session calls enforce_shell_access_for_tier(db, current_user, organization, body.tier) (imported from app/core/deps.py) at step 1, before any DB side effects. The underlying checks are user_has_shell_role and user_in_shell_access_group from app/services/shell_authz.py.

The AND-gate applies to all three tiers. There is no bypass path — no other code in sessions.py creates session records.

Shell roles

Role	What it grants
`device.shell.standard`	Open a Standard-tier session (non-root diagnostic user).
`device.shell.elevated`	Open an Elevated-tier session (root).
`device.shell.breakglass`	Open a Break-Glass-tier session (root, emergency).
`device.shell.approve`	Approve or deny pending peer-approval requests.
`audit.recording.view`	View (replay) a session recording via `GET /sessions/{id}/recording`.
`audit.recording.export`	Export a recording for compliance/SIEM use via `GET /sessions/{id}/recording/export`.

Role grants are managed via app/api/rbac.py:

POST /users/{user_id}/roles?organization_id=<org> — grant a role.
DELETE /users/{user_id}/roles?organization_id=<org>&role=<role> — revoke a role.

Granting device.shell.elevated or device.shell.breakglass to a user who already holds an org-tenancy OWNER or ADMIN role emits a separation-of-duties (sod_warning) field in the grant response. This is a warning only; it does not block the grant. The fleet owner is notified regardless (shell_audit_notify.py).

Tiers

Three tiers exist. The tier controls the OS user the gateway connects as, whether recording is mandatory, the approval mode, and the session time-box.

Standard

OS user: configurable non-root user (e.g. support) with a curated sudoers allowlist baked into the firmware via OTA. The operator cannot modify sudoers at session time.
Recording: optional. Controlled by standard_recording_enabled in the org’s ShellAccessPolicy. Default: off.
Approval: none. Sessions become ACTIVE immediately.
Justification: optional.
Role required: device.shell.standard.

Elevated

OS user: root.
Recording: mandatory. Cannot be disabled by the org policy or the operator. The recording flag is set to True on the ShellSession row at creation and is never re-evaluated (app/api/sessions.py, step 4a).
Approval: configurable per org: none, self_justification, or peer.
- none — session becomes ACTIVE immediately.
- self_justification — session becomes ACTIVE immediately; an Approval row with decision='approved' and approver_id=current_user.id is written for the audit trail.
- peer — session stays PENDING until a second user holding device.shell.approve calls POST /sessions/{id}/approve. Self-approval is always rejected (HTTP 400).
Justification: required. Empty/whitespace-only input returns HTTP 422.
Session time-box: elevated_session_timeout_minutes in policy (default 30 min; max 1440 min / 24 h).
Approval timeout: elevated_approval_timeout_minutes in policy (default 15 min; max 1440 min).
Role required: device.shell.elevated.

Break-Glass

OS user: root.
Recording: mandatory. Same enforcement as Elevated.
Approval: follows the org’s elevated_approval_mode setting (the same field applies to both Elevated and Break-Glass).
Justification: required.
Session time-box: breakglass_session_timeout_minutes in policy (default 60 min; max 1440 min).
Incident record: a ShellIncident row is created immediately after the session row commits and before the gateway call, so the audit trail is intact even if the gateway is unreachable (app/api/sessions.py, step 7, create_breakglass_incident).
Notifications: immediate fan-out to email + webhook + PagerDuty/SMS channels configured in breakglass_notification_channels.
Role required: device.shell.breakglass (grant sparingly).

Policy schema

The org’s ShellAccessPolicy row is read and written via:

GET /organizations/{org_id}/access-policy (or /fleets/{fleet_id}/access-policy)
PUT /organizations/{org_id}/access-policy (OWNER / ADMIN only)

Key fields (app/api/access_policy.py, AccessPolicyResponse):

Field	Type	Default	Description
`allowed_tiers`	`string[]`	`["standard","elevated","breakglass"]`	Which tiers the org allows.
`standard_recording_enabled`	`bool`	`false`	Record Standard sessions.
`elevated_approval_mode`	`string`	`"none"`	`none`, `self_justification`, or `peer`.
`elevated_approval_timeout_minutes`	`int`	`15`	How long a peer-approval request waits before expiry.
`elevated_session_timeout_minutes`	`int`	`30`	Hard session cap for Elevated.
`breakglass_enabled`	`bool`	`true`	Whether Break-Glass can be requested.
`breakglass_notification_channels`	`string[]`	`["email","webhook"]`	Alert channels on Break-Glass open.
`breakglass_session_timeout_minutes`	`int`	`60`	Hard session cap for Break-Glass.
`additions_require_approval`	`bool`	`false`	Whether adding a first Shell Access Group member requires a confirmation step.
`recording_retention_days`	`int`	`90`	How long to keep recordings before the daily retention job deletes them.
`recording_storage_quota_bytes`	`int?`	`null` (unlimited)	Per-org recording storage cap in bytes.

Recording and immutability

Where capture happens

Recording is performed at the gateway, not on the device. The gateway’s AsciinemaRecorder (gateway/app/recorder.py) intercepts every PTY byte flowing through the SSH bridge in both directions (device output and operator input). A compromised device or a rogue operator cannot disable or tamper with the recording because the capture path runs entirely outside the device.

RecorderError (disk full, permission denied, etc.) is treated as fatal: the bridge terminates the session rather than silently continuing unrecorded.

Format and storage

Format: asciinema v2 (.cast). Line 1 is a JSON header; subsequent lines are [seconds, "o"|"i"|"r", ...] events.
Storage: S3-compatible object store (MinIO in self-hosted deployments; any AWS S3-compatible endpoint). Configured via RECORDINGS_S3_* env vars.
Per-org S3 prefix: recordings are stored under <org-id>/<session-id>/ so each organisation’s recordings are logically isolated within the bucket.
Hash on completion: when the gateway uploads the .cast file it computes a SHA-256 digest and stores it in ShellSession.recording_sha256. This hash is also included in the recording.attached audit event in the hash chain.
Presigned access: GET /sessions/{id}/recording returns a short-lived presigned URL (TTL configured by recording_presign_ttl_seconds). The download URL is never stored; it is minted on each request.
Export gate: GET /sessions/{id}/recording/export requires the higher audit.recording.export role and emits a recording.exported event (distinct from recording.viewed) for compliance pipelines.

S3 object-lock

Live S3 object-lock (write-once WORM) for Break-Glass recordings is a deferred feature. See docs/remote-ssh/DEFERRED-VERIFICATION.md.

Retention

The daily retention job (app/services/retention.py, enforce_retention) deletes recordings whose ShellSession.created_at is older than the org’s recording_retention_days horizon. It:

Deletes the object from S3 first.
Only if the S3 delete succeeds: NULLs recording_uri, recording_sha256, and recording_size_bytes on the DB row and appends a recording.deleted audit event in the same transaction.

If the S3 delete fails, the DB row is left untouched (atomicity preserved). The job runs as a background asyncio loop (interval 24 h) wired into app.main.lifespan.

Storage quotas

When recording_storage_quota_bytes is set, the check_quota_alerts function (app/services/retention.py) fires a quota.alert audit event when the org’s total recording usage reaches 80% of the quota. Alerts are de-duplicated within a 24-hour window. The org’s current_usage_bytes is returned on every GET /access-policy call so the admin can monitor usage without querying the audit log.

Audit chain

Structure

Audit events are stored in ShellAuditEvent rows. Each event carries:

prev_hash — the event_hash of the immediately preceding event for this org, or "0" * 64 (the genesis constant) for the first event.
event_hash — SHA-256 of a deterministic canonical JSON encoding of all event fields (sorted keys, UTC ISO-8601 timestamps, UUIDs as lowercase hyphenated strings).
event_type — one of the constants below.
actor_id, target_user_id, payload, reason, created_at.

The chain is per-organisation and uses a PostgreSQL advisory transaction lock (pg_advisory_xact_lock(hashtext(org_id))) to serialise concurrent appenders. On SQLite (unit tests) the lock is skipped. See app/services/shell_audit.py, append_event.

Events

Event type	When it fires
`session.requested`	User calls `POST /devices/{id}/sessions` (any tier).
`session.started`	Session transitions to `ACTIVE` (immediately for `none`/`self_justification` modes; after peer approval for `peer` mode).
`session.approved`	A peer approver calls `POST /sessions/{id}/approve`.
`session.denied`	A peer approver calls `POST /sessions/{id}/deny`.
`session.ended`	Session ends. `payload.exit_reason` is one of `normal`, `timeout`, `agent_crashed`, `agent_exited`, `gateway_failover`.
`session.rate_limited`	Session-open request is rejected by rate limiter (per-user, per-device, or per-fleet bucket).
`session.resumed`	WebSocket reconnect after a network blip.
`recording.attached`	Gateway uploads the `.cast` file and registers the URI + SHA-256 on the session row.
`recording.viewed`	`GET /sessions/{id}/recording` is called.
`recording.exported`	`GET /sessions/{id}/recording/export` is called.
`recording.deleted`	Daily retention job deletes a recording from S3 and NULLs the DB row.
`quota.alert`	Org recording usage reaches 80% of the configured quota.
`policy.changed`	`PUT /organizations/{org_id}/access-policy` succeeds. Payload includes `changes` with `before`/`after` for every changed field.
`shell.role.granted`	`POST /users/{user_id}/roles` succeeds.
`shell.role.revoked`	`DELETE /users/{user_id}/roles` succeeds.
`shell.group.member_added`	User is added to the Shell Access Group.
`shell.group.member_removed`	User is removed from the Shell Access Group.

Chain verification

app/services/shell_audit.verify_chain(db, organization_id) replays the full chain for an org and returns True if every prev_hash links correctly and every event_hash can be recomputed from the stored fields. Use this in compliance reviews or after a suspected tamper event.

Approval workflows

Three approval modes are available, configured per-org via elevated_approval_mode:

Mode	Behaviour
`none`	Session becomes `ACTIVE` immediately. No `Approval` row is written.
`self_justification`	Session becomes `ACTIVE` immediately. An `Approval` row with `decision='approved'` and `approver_id=requester.id` is written for the audit trail. The `session.started` event lands in the same transaction as the `approval.decided` event.
`peer`	Session status is `PENDING`. An `Approval` row with `decision='pending'` is created. A notification is sent to users holding `device.shell.approve`. The session becomes `ACTIVE` only when a different user calls `POST /sessions/{id}/approve`. Self-approval is rejected (HTTP 400).

Approval timeout is governed by elevated_approval_timeout_minutes. When a peer-approval request expires without a decision, the session is cleaned up during the next reaper cycle.

HMAC-signed webhook notifications

When a webhook URL is configured in breakglass_notification_channels (or via the notification service configuration), outgoing POST requests are signed with HMAC-SHA256 (see app/services/notifications.py, lines 18–46 and 381–423). Three headers are sent on every webhook:

X-OTAPulse-Signature: sha256=<hex> — HMAC-SHA256 over the signed input.
X-OTAPulse-Timestamp: <unix_int> — request timestamp (signed; replay guard).
X-OTAPulse-Event-Id: <uuid4> — idempotency key.

The signed input is f"{unix_ts}.".encode() + body — the ASCII unix timestamp from X-OTAPulse-Timestamp, the literal . separator, then the raw JSON request body bytes. The HMAC is keyed with the per-fleet webhook secret and the lowercase hex digest is placed in X-OTAPulse-Signature prefixed with sha256=. Verification:

Recompute HMAC-SHA256(secret, f"{header_ts}.".encode() + body).
Compare to the header value (constant-time compare).
Receivers are recommended to reject requests whose X-OTAPulse-Timestamp is more than a few minutes skewed from local clock to bound replay windows (typical receiver-side window: ±5 minutes; see notifications.py:38-41). This window is a receiver-side recommendation — the server signs the timestamp but does not enforce a window on outgoing requests.

Binding the timestamp into the HMAC means an attacker who captures a valid (body, signature) pair cannot replay it under a fresh timestamp: changing the header without re-signing invalidates the HMAC.

Token lifetimes and MFA

WS session token

When a session is opened, the backend calls the gateway’s internal mint endpoint to issue a WebSocket upgrade token (ws_token in SessionResponse). Properties:

32 bytes of URL-safe random entropy (256 bits).
Single-use: consume() in gateway/app/session_tokens.py marks the token consumed on first WebSocket upgrade; any subsequent attempt returns None.
Short TTL: 60 seconds (configurable via WS_TOKEN_TTL_SECONDS on the gateway).
Never logged at any level.

Resume tokens

After a WebSocket reconnect (network blip), the gateway issues a resume token to the browser. Resume tokens are also single-use and short-lived. They are rate-limited per-session via gateway/app/resume_rate_limit.py.

MFA freshness gate

For Elevated and Break-Glass session opens, the server checks that the requesting user’s last MFA assertion is recent. The freshness window is shell_mfa_freshness_minutes in app.config.settings. When SHELL_MFA_ENFORCE=false (the default for dev environments), the check logs a warning but does not block the request. In production, set SHELL_MFA_ENFORCE=true to enforce the gate (app/api/sessions.py, step 0b).

Rate limiting

Session creation is rate-limited across three independent buckets: per-user, per-device, and per-fleet (org). Exceeding any bucket returns HTTP 429 with a Retry-After header and emits a session.rate_limited audit event. See app/core/rate_limit.py.