Agentic Threat Hunter: offline demo

Deterministic detection first, evidence-constrained investigation second. This page was built on this machine in 1.2 s at 2026-09-28T21:11:20+00:00. It used no API key, no network and no model call. Each section links the files its numbers come from.

1. Detect

23 deterministic rules ran over 1118 synthetic events (462 process, 389 network, 267 logon, 0 control), generated with seed 1337 into this folder.

1118events

16findings

14 / 2true / false positives

12/12labelled attack stages detected

Severity	Rule	Finding	Host	Account	ATT&CK (candidate)
CRITICAL	`ATH-004`	Possible LSASS credential access	PC01	jdoe	T1003.001 (Credential Access); T1218.011 (Stealth)
CRITICAL	`ATH-005`	Failed logon burst followed by successful authentication	FS02	svc_backup	T1110.001 (Credential Access); T1078 (Persistence)
CRITICAL	`ATH-012`	Security product disabled or reconfigured	PC03	achen	T1685 (Defense Impairment)
CRITICAL	`ATH-011`	Windows recovery mechanisms destroyed	PC03	achen	T1490 (Impact)
HIGH	`ATH-001`	Office application spawned a script interpreter	PC01	jdoe	T1059.001 (Execution); T1204.002 (Execution)
HIGH	`ATH-002`	Encoded PowerShell command execution	PC01	jdoe	T1027.010 (Stealth); T1059.001 (Execution); T1105 (Command and Control)
HIGH	`ATH-003`	Script interpreter connected to an external host	PC01	jdoe	T1071.001 (Command and Control); T1105 (Command and Control)
HIGH	`ATH-014`	Credential material access from a command line	PC01	jdoe	T1003.001 (Credential Access)
HIGH	`ATH-006`	Account authenticated from a host where it has no session	FS02	svc_backup	T1021.002 (Lateral Movement); T1078 (Persistence)
HIGH	`ATH-007`	Remote service execution (PsExec-style)	FS02	svc_backup	T1569.002 (Execution); T1021.002 (Lateral Movement)
HIGH	`ATH-008`	Bulk data staged into an archive	FS02	svc_backup	T1560.001 (Collection); T1074.001 (Collection)
MEDIUM	`ATH-009`	Macro-enabled document opened via email client	PC01	jdoe	T1204.002 (Execution)
MEDIUM	`ATH-010`	Sequence of discovery commands from one parent process	PC01	jdoe	T1033 (Discovery); T1069.002 (Discovery); T1482 (Discovery)
MEDIUM	`ATH-003`	Script interpreter connected to an external host	FS02	svc_backup	T1071.001 (Command and Control)
MEDIUM	`ATH-003`	Script interpreter connected to an external host	PC07	adm_sarah	T1071.001 (Command and Control); T1105 (Command and Control)
LOW	`ATH-002`	Encoded PowerShell command execution	PC07	adm_sarah	T1027.010 (Stealth); T1059.001 (Execution)

Per-rule precision and recall against ground_truth.json

Rule	TP	FP	Precision	Recall	Note
`ATH-001`	1	0	1.00	1.00
`ATH-002`	1	1	0.50	1.00	1 FP from the labelled benign look-alike
`ATH-003`	2	1	0.67	1.00	1 FP from the labelled benign look-alike
`ATH-004`	1	0	1.00	1.00
`ATH-005`	1	0	1.00	1.00
`ATH-006`	1	0	1.00	1.00
`ATH-007`	1	0	1.00	1.00
`ATH-008`	1	0	1.00	1.00
`ATH-009`	1	0	1.00	1.00
`ATH-010`	1	0	1.00	1.00
`ATH-011`	1	0	1.00	1.00
`ATH-012`	1	0	1.00	1.00
`ATH-014`	1	0	1.00	1.00

10 rules target telemetry this scenario does not contain (for example cloud and Kubernetes rules) and stay silent: ATH-013, ATH-015, AWS-001, AWS-002, AWS-003, AWS-004, AWS-005, AWS-006, K8S-001, K8S-002.

These are the constructed demo scenarios, scored against the labels the generator itself wrote. They show that the pipeline works and is measured. They are not a production detection rate.

Evidence: ground_truth.json · findings.json · evaluation.json

2. Correlate

Findings are joined into cases only on shared evidence: process lineage, hosts, accounts and authentication context. 14 findings formed 2 case(s); 2 stayed isolated. Tactics are listed in ATT&CK v19.2 (Enterprise, content release August 2026) kill-chain order. Each mapping must cite telemetry.

CASE-001 · CRITICAL · grouping confidence high · 12 findings · hosts FS02, PC01 · accounts jdoe, svc_backup

ExecutionPersistenceStealthCredential AccessDiscoveryLateral MovementCollectionCommand and Control

CASE-001: 12 findings correlated across 2 host(s) (FS02, PC01) and 2 account(s) (jdoe, svc_backup) over 1566s (09:12:04-09:38:10). Behaviour is consistent with 8 ATT&CK tactic(s): Execution -> Persistence -> Stealth -> Credential Access -> Discovery -> Lateral Movement -> Collection -> Command and Control. 30 of 30 links rest on structural evidence (shared events, process lineage, or host-to-host movement) rather than timing alone. Grouping confidence: high. Backed by 31 telemetry events. This is a correlation of observed behaviour and requires analyst confirmation; it does not establish that an intrusion occurred.

CASE-002 · CRITICAL · grouping confidence medium · 2 findings · hosts PC03 · accounts achen

ImpactDefense Impairment

CASE-002: 2 findings correlated across 1 host(s) (PC03) and 1 account(s) (achen) over 220s (11:12:00-11:15:40). Behaviour is consistent with 2 ATT&CK tactic(s): Impact -> Defense Impairment. 1 of 1 links rest on structural evidence (shared events, process lineage, or host-to-host movement) rather than timing alone. Grouping confidence: medium. Backed by 6 telemetry events. This is a correlation of observed behaviour and requires analyst confirmation; it does not establish that an intrusion occurred.

Evidence: cases.json

3. Investigate with verified evidence

The workflow runs telemetry → seed → investigation → report as sealed, resumable stages. Read-only tools retrieve the evidence. Each report separates FACT from INFERENCE and HYPOTHESIS. Verification rejects a cited event id that the investigation never retrieved and any unsupported typed premise. It does not prove that free prose is correct. Engine: deterministic, profile operational-v6, over the three synthetic dev scenarios.

Scenario	What happens	Expected	Verdict	Correct	Report
sample-01	Remote service shell, then a credential export. `reg.exe save HKLM\SAM C:\ProgramData\sam.hiv /y`	malicious	Malicious	yes	html · md
sample-02	Remote service shell, then routine maintenance. `dism.exe /Online /Cleanup-Image /ScanHealth`	benign	Malicious	no	html · md
sample-03	Remote service shell; the deciding child process is deliberately absent.	abstain	Malicious	no	html · md

The deterministic engine gets 1 of 3 right. The rules fire the same way whether the child process is a credential export or a health scan, so triage calls 2 non-malicious case(s) malicious. Telling these apart is the job of the model arm, recorded in section 4.

All reports: investigations/index.html

4. The model's recorded runs

No model runs in this demo. These are recorded outputs of qwen3.5:9b (operational-v6) from Colab run holdout-9b-v6-ecd865890442, replayed as static pages. The rows sample-01 to sample-03 are the same synthetic scenarios as section 3. The rows h05, h07 and h14 are real Windows logs from the DEDALE dataset (CC BY 4.0). The gallery keeps the failure in view.

Case	Arm	Data	Expected	Verdict	What it shows	Report
sample-01	model (D1)	synthetic	malicious	Malicious	correct malicious: Probed the process tree, found reg.exe saving the SAM hive, called it malicious.	html · md
sample-01	deterministic	synthetic	malicious	Malicious	correct malicious: Deterministic triage calls the same incident malicious from the detection findings.	html · md
sample-02	model (D1)	synthetic	benign	Benign	correct benign: Found dism.exe /ScanHealth under the remote shell and cleared it as maintenance.	html · md
sample-02	deterministic	synthetic	benign	Malicious	FALSE ACCUSATION: Deterministic triage cannot tell maintenance from credential export: it accuses.	html · md
sample-03	model (D1)	synthetic	abstain	Abstain	correct abstain: The deciding child process is deliberately missing; the model abstained and named the gap.	html · md
sample-03	deterministic	synthetic	abstain	Malicious	OVER-CALL: Deterministic triage calls it malicious although the telemetry cannot decide.	html · md
h14	model (D1)	DEDALE (CC BY 4.0)	benign	Benign	correct benign: Real Windows logs: a user opening a local PDF in Firefox, correctly cleared.	html · md
h14	deterministic	DEDALE (CC BY 4.0)	benign	Abstain	safe abstain: The deterministic arm abstains, as it did on all 18 holdout cases.	html · md
h07	model (D1)	DEDALE (CC BY 4.0)	malicious	Abstain	safe abstain: APT implant (svcmon.exe): two process-tree probes, opaque child, abstained. Safe, but no answer.	html · md
h07	deterministic	DEDALE (CC BY 4.0)	malicious	Abstain	safe abstain: The deterministic arm abstains.	html · md
h05	model (D1)	DEDALE (CC BY 4.0)	malicious	Benign	UNSAFE CLEAR: The failure: an APT step cleared as benign, partly because whoami.exe is signed by Microsoft.	html · md
h05	deterministic	DEDALE (CC BY 4.0)	malicious	Abstain	safe abstain: The deterministic arm abstains instead of clearing.	html · md

Gallery · Attribution (DEDALE, CC BY 4.0)

5. Measured honestly

Investigative value on Windows: NOT demonstrated.

Pre-registered evaluation on 12 fresh Windows cases from DEDALE (6 malicious, 6 benign), run holdout-9b-v6-ecd865890442, qwen3.5:9b on operational-v6:

0/6malicious cases called malicious

3unsafe clears: malicious called benign

5/6benign cases cleared

0.42balanced accuracy (rule: > 0.5)

18/18deterministic arm abstained

The sealed decision rule failed on two criteria: 3 unsafe clears (at most 1 allowed) and no correct malicious case.
1 benign case was accused. The deterministic arm cleared nothing and accused nothing: it abstained on all 18 cases (0 correct), including 6 secondary Kubernetes benign checks.
Kubernetes malicious discrimination was not evaluated: no suitable fresh, labelled Kubernetes attack dataset was available.
A safety fix, operational-v8, is implemented. These 12 cases are now seen, so it will be measured on a fresh holdout (holdout-v2), not on them.

Evidence: pre-registration · results

DEV-SEEN · not a result

Dev re-run on seen cases: v6 → v8 (not a result)

These same 12 cases were seen and used to design v8, so this is not a result; only a fresh holdout (holdout-v2) can measure v8.

qwen3.5:9b, model arm, run devseen-ladder-a92585184ba4. Each rung adds one change to the one above it. The v6 row is the sealed run above, not re-run.

Rung	Adds	Correct	Malicious recall	Benign accuracy	Unsafe clears	False accusations	Abstentions	Complete
v6	sealed baseline	5/12	0/6	5/6	3/6	1/6	3/12	12/12
v8a	+ evidence surface	3/12	2/6	1/6	0/6	0/6	9/12	10/12
v8b	+ sibling & prevalence probes, coverage-aware menu	7/12	4/6	3/6	0/6	0/6	5/12	11/12
v8c	+ 3 probes, 32k budget	7/12	5/6	2/6	0/6	1/6	4/12	10/12
v8d	+ v8 prompt	8/12	6/6	2/6	0/6	0/6	4/12	10/12
v8	+ guard v2, no-probe rule and its prompt sentence	9/12	6/6	3/6	0/6	0/6	3/12	11/12

Cost: full v8 uses about 2.7× the tokens and 2.2× the median latency of v6 per case.

What the ladder suggests on seen cases (hypotheses to test on holdout-v2):

Unsafe clears went from 3 to 0 once the model could see the parent's image path (v8a), before the prompt or guard changed.
The sibling probe (v8b) was the biggest single step: correct 3 → 7, malicious recall 2 → 4.
v8 trades benign accuracy for safety: 3/6 benign right vs 5/6 on v6; its abstentions moved from attacks to benign cases.

Failure mode: in round 1 the model sometimes cites observation references that do not exist (8 of 251 model rounds across the ladder); the call fails closed and counts as an abstention. A fix (operational-v8.1) is implemented but not yet measured; it must be frozen before holdout-v2.

Guard v2 and the no-probe rule never fired in the v8 rung; they remain untested insurance.

Details: docs/v8-dev-seen-ladder.md

ATH is a research prototype, not a production SOC product. README