feature: threshold rules with hysteresis and sustained breach #56

Merged

Gilles merged 1 commit from feat/threshold-hysteresis into master

2026-10-31 08:33:35 +00:00

Gilles commented

2026-10-31 08:21:19 +00:00

Owner

Closes #54

garion's swap_pct (rule >= 95) sits at 90-100 % every night: host-health flipped ok/warn 9 times between 2026-10-29 and 10-31, several of them single-report (1 h) warn episodes.

Two optional params on metric_threshold rules, both unset by default -- existing rules behave exactly as before and store nothing:

recover: hysteresis. Once warn/fail is confirmed, back to ok only when the value no longer passes recover (warn >= 95, recover 85 → still warn at 90).
for_count (1-100): the level is only raised after N consecutive reports at least that high; dropping stays immediate. While unconfirmed, the metric carries pending: {status, count, of}.

How: thresholds.sustained_level() reads the rule's state from the stream's previous report (summary["thresholds"][rule id]: level + streaks capped at N, so a steady breach keeps an identical summary and the source de-dup still applies). _on_metric_threshold reuses the stored level instead of recomputing the raw one, so events and status agree. _previous_in_stream() is now shared with the de-dup check.

Replaying garion's real values (2026-10-30 22:00 → 10-31 07:20): with for_count: 2, the isolated 23:09 warn disappears and 03:13-06:18 becomes a single warn episode confirmed at 04:15 (back to ok at 07:20). recover: 85 is not a fit for swap: it sits at 93-94 % between peaks, so it would hold warn the whole night -- recover is for metrics with a real gap between their normal and breach values.

Tests: apps/events/test_sustained_thresholds.py + param validation; full suite, pylint 10/10, autopep8 clean locally. Frontend form: argus/frontend#46.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUcAVt7dKqdZsFwQdwZkxW

Closes #54 garion's `swap_pct` (rule `>= 95`) sits at 90-100 % every night: host-health flipped ok/warn 9 times between 2026-10-29 and 10-31, several of them single-report (1 h) warn episodes. Two optional params on `metric_threshold` rules, both unset by default -- existing rules behave exactly as before and store nothing: - `recover`: hysteresis. Once warn/fail is confirmed, back to ok only when the value no longer passes `recover` (warn >= 95, recover 85 → still warn at 90). - `for_count` (1-100): the level is only raised after N consecutive reports at least that high; dropping stays immediate. While unconfirmed, the metric carries `pending: {status, count, of}`. How: `thresholds.sustained_level()` reads the rule's state from the stream's previous report (`summary["thresholds"][rule id]`: level + streaks capped at N, so a steady breach keeps an identical summary and the source de-dup still applies). `_on_metric_threshold` reuses the stored level instead of recomputing the raw one, so events and status agree. `_previous_in_stream()` is now shared with the de-dup check. Replaying garion's real values (2026-10-30 22:00 → 10-31 07:20): with `for_count: 2`, the isolated 23:09 warn disappears and 03:13-06:18 becomes a single warn episode confirmed at 04:15 (back to ok at 07:20). `recover: 85` is *not* a fit for swap: it sits at 93-94 % between peaks, so it would hold warn the whole night -- recover is for metrics with a real gap between their normal and breach values. Tests: `apps/events/test_sustained_thresholds.py` + param validation; full suite, pylint 10/10, autopep8 clean locally. Frontend form: argus/frontend#46. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AUcAVt7dKqdZsFwQdwZkxW

Gilles added the

feature

label

2026-10-31 08:21:19 +00:00

Gilles added 1 commit

2026-10-31 08:21:19 +00:00

feature: threshold rules with hysteresis and sustained breach

ci / test (pull_request) Successful in 3m59s

Details

ci / e2e (pull_request) Has been skipped

Details

ci / docker (pull_request) Has been skipped

Details

ci / deploy (pull_request) Has been skipped

Details

01e6d6b22d

Two optional metric_threshold params, unset by default (unchanged
behaviour):
- recover: once a level above ok is confirmed, it only drops back to ok
  when the value no longer passes this second limit;
- for_count: a level is only reached after N consecutive reports at
  least that high (a pending breach is marked on the metric).

Each sustained rule's state lives in the report's
summary["thresholds"][rule id], read back from the stream's previous
report; the event engine reuses that level. Streaks are capped at N so
a steady breach still deduplicates source reports.

Closes #54

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUcAVt7dKqdZsFwQdwZkxW

Gilles referenced this pull request from argus/frontend

2026-10-31 08:22:58 +00:00

feature: threshold rule form: hysteresis and sustained breach fields #48

Gilles scheduled this pull request to auto merge when all checks succeed

2026-10-31 08:26:14 +00:00

Gilles commented

2026-10-31 08:33:25 +00:00

Author

Owner