Operator How-To

This guide is for operators — users who will open terminal sessions to devices. It assumes your account has already been granted the necessary role and group membership by an admin. If you are not sure whether you are set up, see Pre-requisites below.

Pre-requisites

Before you can open a session you need all three of the following:

Role grant. Your account must hold at least one device.shell.* role for the organisation:
- device.shell.standard — Standard-tier sessions.
- device.shell.elevated — Elevated-tier sessions (root).
- device.shell.breakglass — Break-Glass sessions (emergency root).
Ask an org OWNER or ADMIN to grant you the role via Settings → Users → Shell Roles in the console.
Shell Access Group membership. You must also be an active member of the fleet’s Shell Access Group. A role grant alone is not enough — both gates must pass simultaneously. Ask an admin to add you via Settings → Shell Access Group.
MFA enrolled (when enforcement is on). MFA enforcement is currently advisory-only in default deployments (SHELL_MFA_ENFORCE=false); when production-enforced, the server checks that your most recent MFA assertion is within the configured freshness window (typically 30–60 minutes) before it lets you open an Elevated or Break-Glass session. The enrollment UI (“Enrol TOTP or a hardware key under Account → Security”) is a planned workflow — see docs/remote-ssh/DEFERRED-VERIFICATION.md (DV-S20-005-MFA-enrollment). Until the workflow ships and your deployment sets SHELL_MFA_ENFORCE=true, the gate logs a warning but does not block the session.

Opening a session

1. Find the device

From the main console, open the Devices list and locate the device you need to access. The device must show as Online (green status indicator). If it is offline, wait for a fresh heartbeat or ask the team responsible for the device.

2. Open the Terminal action

Click the device row to open the device detail view, then click Open Terminal. If the button is greyed out, the device tunnel is offline — the agent on the device has not connected to the gateway.

3. Choose a tier

A dialog appears with the tiers you are permitted to use (tiers you do not have a role for are greyed out, and tiers disabled by the org policy are hidden entirely):

Tier	When to use
Standard	Routine diagnostics — log inspection, service status, network checks. Non-root.
Elevated	Root access when Standard is insufficient — configuration changes, debugging kernel issues. Recording is mandatory.
Break-Glass	Emergency root access — use when normal channels are down and you need immediate access. Alerts fire immediately to the fleet owner.

Select the appropriate tier. Default to the lowest tier that meets your need.

4. Provide a justification (Elevated / Break-Glass)

For Elevated and Break-Glass sessions the justification field is required. Write a clear, factual reason (e.g. “Investigating systemd unit crash on device fleet-42 — ticket #1234”). The justification is stored in the audit log and is visible to your org’s admins.

Justification is optional for Standard sessions.

5. Wait for approval (peer mode only)

If the org’s policy sets elevated_approval_mode to peer, your session stays in Pending state after you submit. The console shows a waiting indicator. Users with device.shell.approve receive a notification. Once one of them approves, the terminal opens automatically. If the approval window expires (default 15 minutes) the request is cancelled and you will need to re-submit.

You can ping approvers directly — the pending request is visible in their Approvals inbox.

In-session UI

Once the session is active, the console shows a full-screen terminal view.

Tier badge — top-right corner shows the current tier (STANDARD, ELEVATED, or BREAK-GLASS) so you always know what privilege level is in effect.
REC indicator — a red REC badge is visible whenever recording is active. For Elevated and Break-Glass this is always on and cannot be hidden. For Standard sessions it shows only when the org policy enables recording.
Session countdown — a timer in the top bar shows the time remaining before the server-enforced session timeout. Standard sessions may not have a hard cap depending on org policy; Elevated and Break-Glass sessions always have one (visible as a countdown).
Terminal area — a standard PTY backed by xterm.js. Supports:
- Copy/paste via the browser clipboard API.
- Terminal resize (propagated to the device PTY via a resize event).
- Clear scrollback (Ctrl+L or the clear button).
End Session button — at the top right. Closes the session cleanly.

Handling network blips

Temporary disconnect (auto-resume)

If your network connection drops briefly (Wi-Fi handoff, VPN reconnect), the terminal shows a Reconnecting… overlay and automatically retries the WebSocket connection. On a successful reconnect, the gateway issues a single-use resume token and the session continues. No data is lost; the recording is continuous.

The resume overlay clears automatically when the connection is restored. You do not need to do anything.

Agent crash

If the device-side agent process dies unexpectedly (OOM kill, watchdog reboot, power cycle), the terminal shows a Session ended — agent crashed banner. The session is marked with exit_reason: agent_crashed in the audit log. You must open a new session once the device comes back online.

Gateway failover

In rare cases the gateway process restarts mid-session. The terminal shows a Gateway restarted — click Retry to reconnect modal. Clicking Retry mints a new WebSocket token and re-establishes the bridge. The recording up to the failover point is preserved; the resumed portion appends to the same .cast file. The audit log records a session.resumed event.

Closing a session

Click End Session in the top bar, or type exit in the terminal. Either action sends a clean close to the gateway. The server records exit_reason: normal and the session status transitions to ended.

If you close the browser tab without clicking End Session, the gateway detects the WebSocket close and ends the session within a few seconds.

Do not leave sessions open when you are done. Abandoned sessions consume the per-user session slot and may trigger security alerts for your org.

Viewing your recording

After a recorded session ends:

Open the device detail view and click Session History.
Find the session by date/time. Recorded sessions show a Play icon.
Click Play to open the in-dashboard asciinema player.

You need the audit.recording.view role to access recordings. If you don’t see the Play icon, ask an admin to grant you that role.

Recordings are retained for the number of days configured by the org admin (default 90 days). After that, the daily retention job deletes them from storage and the Play icon disappears.

Troubleshooting

”Device offline”

The device heartbeat is stale or the frp agent is not connected. Options:

Wait a few minutes for the device to reconnect and check again.
Ask the team responsible for the device to check the soc-ota-agent and frp agent services on the device.
If the device has a Tuya power socket registered in the console, an admin can power-cycle it.

”Cannot open a shell session: the device tunnel is offline”

The device’s tunnel_online flag is false — the frp connection has dropped. Same resolution as “Device offline” above.

”Device has not been seen for N seconds”

Even when the frp tunnel is marked connected, the device’s heartbeat is stale. The server refuses to open a session to a device that may be unresponsive. Wait for a fresh heartbeat (the Devices list will show the last-seen timestamp).

”Connection lost”

WebSocket closed unexpectedly. Click Retry in the reconnect modal. If retries keep failing, the gateway may be restarting — wait 30 seconds and try again.

”Approval pending — waiting for a peer”

Your Elevated or Break-Glass session is in peer approval mode. Ping a colleague who holds device.shell.approve. They can approve from the Approvals inbox in the console (top navigation).

“403 Forbidden — device.shell.standard role required”

You do not hold the required shell role, or you are not in the Shell Access Group, or both. Contact your org admin. Both conditions must be satisfied simultaneously.