SIEM health check: how to find silent log sources and dead rules.

Why SIEMs fail quietly

A SIEM is a chain: a device writes an event, a collector forwards it, a parser turns it into fields, a rule searches those fields on a schedule, a match becomes an alert, and the alert lands in a queue that a person or an agent works. Any link can break, and most breaks raise no error. The device keeps running, the SIEM keeps ingesting whatever else arrives, and the dashboards stay green because nothing they measure has changed.

That is why a SIEM health check is different from SIEM monitoring. Monitoring asks whether the platform is up. A health check asks whether the detections you rely on could still fire today. The answer needs a separate test for each link in the chain, because each one fails in its own way.

Finding silent log sources

A silent log source is one that has stopped sending, or sends far less than it should, without any error. It is the most damaging failure because every rule that depends on it stops working at the same moment. Picus Security describes two variants when it analyzes why rules fail: unavailable log sources, which have stopped sending, and broken log sources, which are disabled or misconfigured (S01, S02).

To find them, compare what is arriving with what should be arriving:

1. Build the expected inventory. List every device and application that should log to the SIEM: domain controllers, firewalls, VPN gateways, endpoint agents, cloud audit logs, identity providers, email security. The asset inventory and the detection rules both feed this list.
2. Record last-seen time per source and per host. A source-level check misses a single domain controller that went dark while its peers keep sending. Check at host level for the sources your critical rules depend on.
3. Baseline the volume. For each source, learn the normal daily and weekly pattern. A drop well below baseline is a warning even when some events still arrive.
4. Alert on transitions, not states. Raise one alert when a source goes silent and one when it recovers. A daily list of everything that is silent turns into noise within a week.

Industry analysts now describe this "silence detection" as a standard use case for security data pipelines: the pipeline watches every source and flags drops in activity or stale schemas (S04). Whether the check lives in the pipeline, the SIEM or a separate tool matters less than that it exists and that someone owns its alerts.

Data that arrives but is broken

A source can be loud and still useless. The events arrive, but a vendor update changed the format, the parser now drops a field, timestamps are in the wrong zone, or events from many hosts are merged so the host name is lost. Rules that match on the missing field stop firing, and volume-based checks see nothing wrong.

The DeTT&CT framework gives a good vocabulary for this. It scores each data source on device completeness (are all the hosts sending), data field completeness (are the fields you need populated), timeliness (does the data arrive soon enough to act on), consistency (does it follow the same format) and retention (is it kept long enough to investigate) (S03). For a health check, the most useful of these is field completeness, measured for the fields your rules actually use:

• For every enabled rule, list the fields it matches or aggregates on.
• For each field, measure the share of events from the relevant source where it is empty or unparsed, over the last day and the last month.
• Flag any field whose empty share has jumped. That is usually a parser or format change, and it tells you exactly which rules went blind.
• Check clock skew between the source and the SIEM. Events that arrive with the wrong time can fall outside a rule's search window.

Splunk's Security Essentials Data Inventory is one example of a native tool that maps products to data and shows which content your current data supports (S05). It is a useful start, but it reports what the data could support, not whether today's events still carry the fields.

Finding dead rules

A dead rule looks active in the rule list but cannot fire. Finding dead rules means checking each of the states in Exhibit 1 separately, because a rule can pass one and fail the next.

Enabled but not running

Scheduled searches can be skipped when the search head is busy, time out, or fail on a syntax error introduced by an upgrade. The rule stays enabled. Read the scheduler or rule-execution logs and list every enabled rule that has not completed a run in its expected interval.

Running but blind

The rule runs, but against data that no longer contains what it looks for. Join the rule-to-field list from the previous section with the field completeness results. Any rule that depends on an empty field is blind, however healthy its schedule looks.

Running but silent for too long

Some rules should rarely fire, so silence alone proves nothing. But a rule for a common behavior that has produced no alert in months deserves a test. Execute the behavior safely and see whether it fires. This is the validation step described in our ATT&CK coverage guide, and it is the only way to tell a quiet rule from a dead one.

Suppressed into silence

Suppression and allow-lists are necessary, but they accumulate. Review every suppression for an owner, a reason and an expiry date. A suppression that matches almost everything turns the rule off without anyone disabling it.

Alerts that reach nobody

The last link is delivery. An alert that fires into a retired channel, an unmonitored mailbox or a queue nobody is assigned to protects no one. List every alert destination, confirm that each one is worked, and send a harmless test alert end to end at least after every change to routing. Count the alerts that were raised but never opened; that number tells you more about the SOC's real coverage than any heatmap.

The health check checklist

Use this as the agenda for a first health check, then turn each item into a recurring automated check.

1. Expected log source inventory agreed and versioned.
2. Last-seen time per source, and per host for critical sources.
3. Volume baselines with transition alerts for silence and recovery.
4. Field completeness for every field an enabled rule uses.
5. Clock skew per source within the tolerance your rules need.
6. Rule inventory with enabled state, schedule and last successful run.
7. Rules that depend on silent sources or empty fields flagged as blind.
8. Suppressions reviewed for owner, reason and expiry.
9. High-value rules validated by safely executing the behavior.
10. Alert destinations confirmed as worked, with an end-to-end test alert.
11. A gap list ranked by the detections each fix brings back, with owners.

SOCGenX runs this check for you. The existing SIEM health check starts with a free, read-only snapshot through your SIEM's own API: it inventories the sources, scores their health and shows which rules exist and are enabled. The full assessment adds validation, and the Proof Loop then keeps checking every agreed source, detection and alert path, so a silent failure is found by a check rather than by an incident. It works with the SIEM you already run; see the integrations page.