Engine & Worker Health — how to read it

🩺 Engine & Worker Health

What the status hero at the top of the admin page means, and what every background job does.

The whole system runs on a single worker process that fires scheduled warm jobs on their own intervals (the trading engine, the boards, the digests, etc.). After every run, the worker writes a heartbeat to Redis. This panel reads those heartbeats — so a job that silently crashes shows RED here instead of just looking like a quiet day. (That's exactly how the engine's ChallengeTrade crash was caught.)

The overall verdict

Badge	Means
🟢 SYSTEM HEALTHY	Worker alive and every job ran successfully and recently.
🟡 SYSTEM DEGRADED	Worker alive, but at least one job is stale (no recent run) or failing. Open the detail to see which.
🔴 SYSTEM DOWN	The worker process is dead (no heartbeat in 3 min) or the trading engine itself is failing. Investigate now.

Badge

Means

🟢 SYSTEM HEALTHY

Worker alive and every job ran successfully and recently.

🟡 SYSTEM DEGRADED

Worker alive, but at least one job is stale (no recent run) or failing. Open the detail to see which.

🔴 SYSTEM DOWN

The worker process is dead (no heartbeat in 3 min) or the trading engine itself is failing. Investigate now.

Infrastructure row (Web · Worker · DB · Redis)

The four chips under the badge are the core processes & stores — both halves of the app, not just the worker:

Chip	What it is	Down means
Web	The FastAPI app serving the site & API (this page). Shows its uptime.	If it were down you couldn't load this panel — so it's effectively always green while you're looking.
Worker	The background process running all the warm jobs (heartbeat every run). Shows uptime + last heartbeat.	No heartbeat in 3 min → nothing scheduled runs (engine, boards, digests). SYSTEM DOWN.
Database	Postgres — a live `SELECT 1` with latency.	DB unreachable → the app can't read/write anything. SYSTEM DOWN.
Redis	The cache — a live `PING` with latency. Backs the boards, heartbeats, dedup, locks.	Redis down → slower (uncached) + heartbeats can't write. DEGRADED.

Chip

What it is

Down means

Web

The FastAPI app serving the site & API (this page). Shows its uptime.

If it were down you couldn't load this panel — so it's effectively always green while you're looking.

Worker

The background process running all the warm jobs (heartbeat every run). Shows uptime + last heartbeat.

No heartbeat in 3 min → nothing scheduled runs (engine, boards, digests). SYSTEM DOWN.

Database

Postgres — a live SELECT 1 with latency.

DB unreachable → the app can't read/write anything. SYSTEM DOWN.

Redis

The cache — a live PING with latency. Backs the boards, heartbeats, dedup, locks.

Redis down → slower (uncached) + heartbeats can't write. DEGRADED.

Latency in ms is shown on the chip (e.g. Database 8ms) — a spike there is an early warning even when everything is still "up".

Per‑job status

Dot	Status	Means
🟢	up	Last run succeeded and was within ~3× the job's interval. Normal.
🟡	stale	Last run succeeded but was a while ago (overdue). The worker may be slow or the job stuck.
🔴	down	The last run failed — an exception. The error + consecutive‑fail count show in the note column.
⚪	unknown	No heartbeat yet — the job hasn't completed a run since the worker started (long intervals), or it's registered but the worker isn't running it.

Dot

Status

Means

🟢

Last run succeeded and was within ~3× the job's interval. Normal.

🟡

stale

Last run succeeded but was a while ago (overdue). The worker may be slow or the job stuck.

🔴

down

The last run failed — an exception. The error + consecutive‑fail count show in the note column.

⚪

unknown

No heartbeat yet — the job hasn't completed a run since the worker started (long intervals), or it's registered but the worker isn't running it.

The uptime strip / engine bar

The green/red bar is the last ~120 cycles of that job — green = ran OK, red = failed, newest on the right. The engine uptime % in the hero is the green fraction of challenge:engine's recent history. A solid green wall = healthy; any red = a cycle failed (hover the note for the error).

What each job does

Job	Every	Purpose
challenge:engine	~60s	THE critical one. The autonomous trading engine — scans the board, applies the rulebook, places & manages PAPER trades each cycle.
challenge:equity_snapshot	30m	Writes each lane's daily equity point for the equity curve.
challenge:participation	4h	Daily fleet alert — of the lanes that should trade, how many did.
journal:review	4h	End‑of‑day nudge if a DTE bucket is net‑negative expectancy.
spread:digest	4h	End‑of‑day spread‑journal digest to admins + Discord.
econ:print_alert	5m	Auto‑posts a flashy card to Discord when a red‑folder econ event prints (CPI/NFP/FOMC/JOLTs…).
alerts:evaluate	8m	Web‑push alerts for newly‑surfaced A+/A picks.
board:top_options	9m	Refreshes the Top Options board (graded flow trades).
board:social_hot	4m	Refreshes the Social Hot Tickers board (WSB/StockTwits buzz).
board:catalysts	20m	Refreshes the Catalyst / FinTwit board (runners, earnings, sympathy).
market:trending	4m	Refreshes the macro / trending market tape.
community:leaderboard	9m	Refreshes the member P&L leaderboard.
community:top_traders	4.5m	Refreshes Top Traders (shared trades + upvotes).
track:settle	1h	Settles the track record + watchlist paper tracker daily.
orb:sim	~60s	ORB paper‑sim tick (inert while ORB is paused).
orbit:engine	5m	Orbit user‑bot sim tick (gated on `orbit_enabled`).

Job

Every

Purpose

challenge:engine

~60s

THE critical one. The autonomous trading engine — scans the board, applies the rulebook, places & manages PAPER trades each cycle.

challenge:equity_snapshot

30m

Writes each lane's daily equity point for the equity curve.

challenge:participation

Daily fleet alert — of the lanes that should trade, how many did.

journal:review

End‑of‑day nudge if a DTE bucket is net‑negative expectancy.

spread:digest

End‑of‑day spread‑journal digest to admins + Discord.

econ:print_alert

Auto‑posts a flashy card to Discord when a red‑folder econ event prints (CPI/NFP/FOMC/JOLTs…).

alerts:evaluate

Web‑push alerts for newly‑surfaced A+/A picks.

board:top_options

Refreshes the Top Options board (graded flow trades).

board:social_hot

Refreshes the Social Hot Tickers board (WSB/StockTwits buzz).

board:catalysts

20m

Refreshes the Catalyst / FinTwit board (runners, earnings, sympathy).

market:trending

Refreshes the macro / trending market tape.

community:leaderboard

Refreshes the member P&L leaderboard.

community:top_traders

4.5m

Refreshes Top Traders (shared trades + upvotes).

track:settle

Settles the track record + watchlist paper tracker daily.

orb:sim

~60s

ORB paper‑sim tick (inert while ORB is paused).

orbit:engine

Orbit user‑bot sim tick (gated on orbit_enabled).

"Registered but NOT run by the worker"

Some jobs can be registered by the web process but never actually executed by the worker (the worker has to import that service module to run its job). If a job appears here, it is not being refreshed proactively — usually a missing import in app/worker.py. The panel surfaces these in amber so the gap is visible.

Check the API directly

The panel is fed by GET /api/admin/worker-health (admin‑only). It returns overall, worker_alive, up_count/total, not_run_by_worker, and a jobs[] array with each job's status, interval, last_ok_ago, up_pct, fails, err, and the strip (recent run outcomes). The authoritative job list comes from worker:jobs in Redis (what the worker actually runs), not the web registry.