Skip to content

[SEC-30403] Add API schema reference to Cloud SIEM detection rule docs #36142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
chelsea-xu-dd wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

[SEC-30403] Add API schema reference to Cloud SIEM detection rule docs #36142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b5aaeb
Add API schema reference to Cloud SIEM detection rule docs
chelsea-xu-dd Apr 20, 2026
6be45eb
Add Third Party nav entry and reframe schema reference intros
chelsea-xu-dd Apr 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions config/_default/menus/main.en.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7251,6 +7251,11 @@ menu:
parent: cloud_siem_custom_detection_rules
identifier: cloud_siem_sequence_rule
weight: 2029
- name: Third Party
url: security/cloud_siem/detect_and_monitor/custom_detection_rules/third_party
parent: cloud_siem_custom_detection_rules
identifier: cloud_siem_third_party_rule
weight: 2030
- name: Version History
url: security/cloud_siem/detect_and_monitor/version_history
parent: cloud_siem_detect_and_monitor
Expand Down
151 changes: 149 additions & 2 deletions ...ent/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/anomaly.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The anomaly detection rule:

The anomaly method adapts to your normal patterns and reduces noise from routine fluctuations.

**Note**: The anomaly method detects spikes only. It does not alert on drops in log volume.
**Note**: The anomaly method uses a statistical baseline (not machine learning). It detects spikes only — it does not alert on drops in log volume.

### Seasonality and learning period

Expand All @@ -34,5 +34,152 @@ A short learning period is applied for new rules or newly observed values for a
- Scope the query narrowly. Filter by service, environment, team, or endpoint to reduce noise.
- Start with managed default rules for broad coverage, then add custom anomaly rules for high-volume log sources.

## API schema reference

This section describes the `anomaly_detection` rule schema. It is also used by the [Security Monitoring API][5] to create and update rules.

### Top-level payload fields

#### Required

| Field | Type | Description |
|---|---|---|
| `name` | string | The name of the rule. |
| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. |
| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][3]). |
| `isEnabled` | bool | Whether the rule is enabled. |
| `options` | object | Contains `detectionMethod` and `anomalyDetectionOptions`. See [Options](#options). |
| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). |
| `cases` | array | Cases for generating signals. See [Cases](#cases). |

#### Optional

| Field | Type | Description |
|---|---|---|
| `tags` | array\<string\> | Tags for generated signals. |
| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. |
| `groupSignalsBy` | array\<string\> | Additional grouping to perform on top of the existing groups in the query section. Must be a subset of the existing groups. |
| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. |
| `schedulingOptions` | object | Options for scheduled rules. When this field is present, the rule runs based on the schedule (RRULE, RFC 5545). |
| `calculatedFields` | array | Calculated fields. Only allowed for scheduled rules. |

### Queries

An `anomaly_detection` rule has exactly one query.

| Field | Type | Required | Description |
|---|---|---|---|
| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. |
| `query` | string | yes | Query to run on logs. See [Log search syntax][4]. |
| `aggregation` | string enum | yes | The aggregation type. Typically `cardinality` or `count` for anomaly detection. Allowed values: `count`, `cardinality`, `sum`, `max`, `new_value`, `geo_data`, `event_count`, `none`. |
| `distinctFields` | array\<string\> | conditional | Field for which the cardinality is measured. Required when `aggregation` is `cardinality`. |
| `groupByFields` | array\<string\> | yes | Fields to group by. Used to define the per-entity scope for anomaly detection. |
| `metrics` | array\<string\> | no | Not used for `cardinality` or `count` aggregations. |
| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. |
| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored. When `true`, events with missing group-by fields are processed with `N/A` replacing the missing values. |

### Cases

Anomaly rules accept a narrow subset of the case condition grammar. Only the following forms are valid:

| Form | Example |
|---|---|
| `query op NUMBER` | `anomalous_events > 1` |
| `NUMBER op query` | `1 < anomalous_events` |

Not accepted inside an anomaly rule case condition:
- `&&`, `||` (logical combinators)
- `THEN` (sequential operator)
- Arithmetic against literals (for example, `*`)
- Multiple queries combined in one condition

An optional trailing second `NUMBER` encodes a decimal percentile. For example, `a > 99 5` means "above the 99.5th percentile" — the percentile at which the detector itself alerts.

| Field | Type | Required | Description |
|---|---|---|---|
| `name` | string | no | Name of the case. Often empty. |
| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. |
| `condition` | string | no | Condition expression. When present, must follow the narrow form above. |
| `notifications` | array\<string\> | no | Notification targets (for example, `@slack-*`, `@team-*`, `@email-...`, `@webhook-...`). |

### Options

Common `options` fields shared with other detection methods:

| Field | Type | Typical | Description |
|---|---|---|---|
| `detectionMethod` | string enum | required | The detection method. Must be `anomaly_detection` for this method. |
| `evaluationWindow` | int (seconds) | 1800 | A time window that matches when at least one of the cases evaluates true. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. |
| `keepAlive` | int (seconds) | ≥ `evaluationWindow` | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. |
| `maxSignalDuration` | int (seconds) | 3600–86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. |
| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. Decreased by one level when the environment tag starts with `staging`, `test`, or `dev`. |

**Cross-field constraint (server-enforced):** `evaluationWindow ≤ keepAlive ≤ maxSignalDuration`.

#### `anomalyDetectionOptions`

Method-specific sub-object under `options`.

| Field | Type | Typical | Description |
|---|---|---|---|
| `learningDuration` | int (hours) | 24 | Learning duration in hours. Anomaly detection waits for at least this amount of historical data before it starts evaluating. Allowed values: `1`, `6`, `12`, `24`, `48`, `168`, `336`. |
| `detectionTolerance` | int | 5 | Sets how permissive anomaly detection is. Higher values require higher deviations before triggering a signal. Allowed values: `1`, `2`, `3`, `4`, `5`. |
| `bucketDuration` | int (seconds) | 300 | Duration of the time buckets used to aggregate events matched by the rule. Must be greater than or equal to 300. Allowed values: `300`, `600`, `900`, `1800`, `3600`, `10800`. |
| `instantaneousBaseline` | bool | `false` | When `true`, Datadog uses previous values within the defined learning window to construct the baseline, enabling faster baseline establishment. |
| `learningPeriodBaseline` | int | — | Optional override baseline applied while the rule is in the learning period. Must be ≥ 0. |

### Example payload

Example of an `anomaly_detection` rule that alerts on an anomalous number of OCI `LaunchInstance` events across availability domains, per user. Cardinality of `@oci.availability_domain` is measured per `@usr.name`; a signal fires when that cardinality is unusually high for that user.

```json
{
"name": "Anomalous number of OCI instances created across availability domains",
"type": "log_detection",
"isEnabled": true,
"message": "User {{@usr.name}} launched OCI instances across an unusual number of availability domains.",
"tags": [
"source:oracle-cloud-infrastructure",
"security:attack",
"tactic:TA0040-impact"
],
"options": {
"detectionMethod": "anomaly_detection",
"evaluationWindow": 1800,
"keepAlive": 7200,
"maxSignalDuration": 86400,
"anomalyDetectionOptions": {
"learningDuration": 24,
"detectionTolerance": 5
}
},
"queries": [
{
"name": "anomalous_oci_instance_creation",
"query": "source:oci.audit @evt.name:LaunchInstance",
"aggregation": "cardinality",
"distinctFields": ["@oci.availability_domain"],
"groupByFields": ["@usr.name"]
}
],
"cases": [
{
"status": "medium",
"name": "",
"notifications": []
}
]
}
```

### Further reading

- [Create a detection rule (API reference)][5]
- [Log search syntax][4]
- [Security notification variables][3]

[1]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=anomaly
[2]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule/?tab=anomaly#rule-multi-triggering-rt-anomaly
[2]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule/?tab=anomaly#rule-multi-triggering-rt-anomaly
[3]: /security/notifications/variables/
[4]: /logs/search_syntax/
[5]: /api/latest/security-monitoring/#create-a-detection-rule
118 changes: 117 additions & 1 deletion ...ecurity/cloud_siem/detect_and_monitor/custom_detection_rules/content_anomaly.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,119 @@ To calculate the Jaccard similarity between the two logs:

$$\text"J(log1,log2)" = 2 / 8 = 0.25$$

## API schema reference

This section describes the `content_anomaly` rule schema. It is also used by the [Security Monitoring API][8] to create and update rules.

### Top-level payload fields

#### Required

| Field | Type | Description |
|---|---|---|
| `name` | string | The name of the rule. |
| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. |
| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][6]). |
| `isEnabled` | bool | Whether the rule is enabled. |
| `options` | object | Contains `detectionMethod` and content anomaly options. See [Options](#options). |
| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). |
| `cases` | array | Cases for generating signals. See [Cases](#cases). |

#### Optional

| Field | Type | Description |
|---|---|---|
| `tags` | array\<string\> | Tags for generated signals. |
| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. |
| `groupSignalsBy` | array\<string\> | Additional grouping to perform on top of the existing groups in the query section. Must be a subset of the existing groups. |
| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. |

### Queries

Content anomaly supports multiple queries per rule.

| Field | Type | Required | Description |
|---|---|---|---|
| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. |
| `query` | string | yes | Query to run on logs. See [Log search syntax][7]. |
| `aggregation` | string enum | yes | The aggregation type. Allowed values: `count`, `cardinality`, `sum`, `max`, `new_value`, `geo_data`, `event_count`, `none`. |
| `groupByFields` | array\<string\> | no | Fields to group by. |
| `distinctFields` | array\<string\> | conditional | Field for which the cardinality is measured. Required when `aggregation` is `cardinality`. |
| `metrics` | array\<string\> | conditional | Target fields to aggregate over. Required for `sum`, `max`, `geo_data`, `new_value` aggregations. |
| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. |
| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored. When `true`, events with missing group-by fields are processed with `N/A`. |

### Cases

| Field | Type | Required | Description |
|---|---|---|---|
| `name` | string | no | Name of the case. |
| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. |
| `condition` | string | no | A case contains logical operations (`>`, `>=`, `<`, `<=`, `&&`, `||`) to determine if a signal should be generated based on the anomalous event counts from the defined queries. |
| `notifications` | array\<string\> | no | Notification targets (for example, `@slack-*`, `@team-*`). |

### Options

Common `options` fields:

| Field | Type | Typical | Description |
|---|---|---|---|
| `detectionMethod` | string enum | required | The detection method. Must be `content_anomaly` for this method. |
| `evaluationWindow` | int (seconds) | 0–86400 | Defines the time frame for counting anomalous logs. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. |
| `keepAlive` | int (seconds) | ≥ `evaluationWindow` | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. |
| `maxSignalDuration` | int (seconds) | 3600–86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. |
| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. |

**Cross-field constraint (server-enforced):** `evaluationWindow ≤ keepAlive ≤ maxSignalDuration`.

#### Content anomaly options

Method-specific options controlling the anomaly detection logic. See [Anomaly detection parameters](#anomaly-detection-parameters) above for UI-level configuration.

| Field | Type | Typical | Description |
|---|---|---|---|
| `learningDuration` | int (days) | 7 | Time window during which values are learned. No signals are generated during this phase. Range: `1`–`10` days. |
| `forgetAfter` | int (days) | 7 | How long learned values are retained before being discarded. Range: `1`–`10` days. |
| Similarity percentage threshold | int (percent) | 70 | Minimum similarity required to consider a log as normal. Range: `35`–`100`%. |
| Similar items threshold | int | 1 | Number of matching historical logs required for an incoming value to be considered normal. Range: `1`–`20`. |

### Example payload

```json
{
"name": "Anomalous content in CloudTrail user creation events",
"type": "log_detection",
"isEnabled": true,
"message": "Anomalous event content detected for {{@usr.name}}.",
"tags": [
"source:cloudtrail",
"security:attack"
],
"options": {
"detectionMethod": "content_anomaly",
"evaluationWindow": 3600,
"keepAlive": 7200,
"maxSignalDuration": 86400
},
"queries": [
{
"name": "anomalous_user_creation",
"query": "source:cloudtrail @evt.name:CreateUser",
"aggregation": "count",
"groupByFields": ["@usr.name"]
}
],
"cases": [
{
"name": "",
"status": "medium",
"condition": "anomalous_user_creation >= 1",
"notifications": []
}
]
}
```

## Comparing content anomaly method with other detection methods

| Feature | Anomaly Detection | New Value Detection | Content Anomaly Detection |
Expand All @@ -130,4 +243,7 @@ $$\text"J(log1,log2)" = 2 / 8 = 0.25$$
[2]: https://www.unicode.org/reports/tr29/tr29-22.html
[3]: https://en.wikipedia.org/wiki/Jaccard_index
[4]: https://en.wikipedia.org/wiki/MinHash
[5]: https://en.wikipedia.org/wiki/Locality-sensitive_hashing
[5]: https://en.wikipedia.org/wiki/Locality-sensitive_hashing
[6]: /security/notifications/variables/
[7]: /logs/search_syntax/
[8]: /api/latest/security-monitoring/#create-a-detection-rule
Loading
Loading