Promethist Watchdog

Crash and freeze detection

Understand this before tuning, as the two are not equally reliable.

Crash detection is reliable and always on. Watchdog supervises the whole process tree; if it dies, Watchdog reads the exit code and restarts according to your policy. Windows crash codes (e.g. 0xC0000005 access violation) are recognised as crashes rather than clean exits.

Freeze detection is on by default for GUI apps. Watchdog uses the same "Not Responding" signal as Task Manager (IsHungAppWindow plus a WM_NULL probe). The default tolerance is 8 consecutive hung polls (about 16 seconds). Raise it for an app with a legitimately slow operation — a big file open, a long query — or lower it to react faster.

For headless and .bat apps, freeze detection is opt-in and heuristic, as a windowless process gives Windows no hang signal. The available checks:

  • Log-file staleness — point it at the app’s own rotating log.

  • Output cadence — based on the stdout/stderr Watchdog already captures. Set max_quiet_s to at least the app’s longest legitimate quiet interval.

  • CPU is corroborate-only and never restarts an app on its own.

  • Memory is a leak-to-restart action, not a freeze check.

Three guards prevent these from firing incorrectly: quiet windows (suppress checks when an app is expected to be idle, e.g. overnight), launch cooldown (skip checks while it warms up), and the circuit breaker (max_restarts / restart_window_s), which stops restart storms and emails you instead of looping forever.

Known blind spot: without the app’s cooperation, a GUI app repainting while a worker thread is deadlocked, or a headless app deadlocked with no observable side effects, cannot be distinguished from "busy" or "idle". Crash detection still covers the common case.