diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 359f5ab4313..cd1ee8ab2a2 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -7251,6 +7251,11 @@ menu: parent: cloud_siem_custom_detection_rules identifier: cloud_siem_sequence_rule weight: 2029 + - name: Third Party + url: security/cloud_siem/detect_and_monitor/custom_detection_rules/third_party + parent: cloud_siem_custom_detection_rules + identifier: cloud_siem_third_party_rule + weight: 2030 - name: Version History url: security/cloud_siem/detect_and_monitor/version_history parent: cloud_siem_detect_and_monitor diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/anomaly.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/anomaly.md index aea2e828975..76a956af929 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/anomaly.md +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/anomaly.md @@ -21,7 +21,7 @@ The anomaly detection rule: The anomaly method adapts to your normal patterns and reduces noise from routine fluctuations. -**Note**: The anomaly method detects spikes only. It does not alert on drops in log volume. +**Note**: The anomaly method uses a statistical baseline (not machine learning). It detects spikes only — it does not alert on drops in log volume. ### Seasonality and learning period @@ -34,5 +34,152 @@ A short learning period is applied for new rules or newly observed values for a - Scope the query narrowly. Filter by service, environment, team, or endpoint to reduce noise. - Start with managed default rules for broad coverage, then add custom anomaly rules for high-volume log sources. +## API schema reference + +This section describes the `anomaly_detection` rule schema. It is also used by the [Security Monitoring API][5] to create and update rules. + +### Top-level payload fields + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][3]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` and `anomalyDetectionOptions`. See [Options](#options). | +| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). | +| `cases` | array | Cases for generating signals. See [Cases](#cases). | + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | +| `groupSignalsBy` | array\ | Additional grouping to perform on top of the existing groups in the query section. Must be a subset of the existing groups. | +| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. | +| `schedulingOptions` | object | Options for scheduled rules. When this field is present, the rule runs based on the schedule (RRULE, RFC 5545). | +| `calculatedFields` | array | Calculated fields. Only allowed for scheduled rules. | + +### Queries + +An `anomaly_detection` rule has exactly one query. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. | +| `query` | string | yes | Query to run on logs. See [Log search syntax][4]. | +| `aggregation` | string enum | yes | The aggregation type. Typically `cardinality` or `count` for anomaly detection. Allowed values: `count`, `cardinality`, `sum`, `max`, `new_value`, `geo_data`, `event_count`, `none`. | +| `distinctFields` | array\ | conditional | Field for which the cardinality is measured. Required when `aggregation` is `cardinality`. | +| `groupByFields` | array\ | yes | Fields to group by. Used to define the per-entity scope for anomaly detection. | +| `metrics` | array\ | no | Not used for `cardinality` or `count` aggregations. | +| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. | +| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored. When `true`, events with missing group-by fields are processed with `N/A` replacing the missing values. | + +### Cases + +Anomaly rules accept a narrow subset of the case condition grammar. Only the following forms are valid: + +| Form | Example | +|---|---| +| `query op NUMBER` | `anomalous_events > 1` | +| `NUMBER op query` | `1 < anomalous_events` | + +Not accepted inside an anomaly rule case condition: +- `&&`, `||` (logical combinators) +- `THEN` (sequential operator) +- Arithmetic against literals (for example, `*`) +- Multiple queries combined in one condition + +An optional trailing second `NUMBER` encodes a decimal percentile. For example, `a > 99 5` means "above the 99.5th percentile" — the percentile at which the detector itself alerts. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. Often empty. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `condition` | string | no | Condition expression. When present, must follow the narrow form above. | +| `notifications` | array\ | no | Notification targets (for example, `@slack-*`, `@team-*`, `@email-...`, `@webhook-...`). | + +### Options + +Common `options` fields shared with other detection methods: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Must be `anomaly_detection` for this method. | +| `evaluationWindow` | int (seconds) | 1800 | A time window that matches when at least one of the cases evaluates true. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. | +| `keepAlive` | int (seconds) | ≥ `evaluationWindow` | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. | +| `maxSignalDuration` | int (seconds) | 3600–86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. Decreased by one level when the environment tag starts with `staging`, `test`, or `dev`. | + +**Cross-field constraint (server-enforced):** `evaluationWindow ≤ keepAlive ≤ maxSignalDuration`. + +#### `anomalyDetectionOptions` + +Method-specific sub-object under `options`. + +| Field | Type | Typical | Description | +|---|---|---|---| +| `learningDuration` | int (hours) | 24 | Learning duration in hours. Anomaly detection waits for at least this amount of historical data before it starts evaluating. Allowed values: `1`, `6`, `12`, `24`, `48`, `168`, `336`. | +| `detectionTolerance` | int | 5 | Sets how permissive anomaly detection is. Higher values require higher deviations before triggering a signal. Allowed values: `1`, `2`, `3`, `4`, `5`. | +| `bucketDuration` | int (seconds) | 300 | Duration of the time buckets used to aggregate events matched by the rule. Must be greater than or equal to 300. Allowed values: `300`, `600`, `900`, `1800`, `3600`, `10800`. | +| `instantaneousBaseline` | bool | `false` | When `true`, Datadog uses previous values within the defined learning window to construct the baseline, enabling faster baseline establishment. | +| `learningPeriodBaseline` | int | — | Optional override baseline applied while the rule is in the learning period. Must be ≥ 0. | + +### Example payload + +Example of an `anomaly_detection` rule that alerts on an anomalous number of OCI `LaunchInstance` events across availability domains, per user. Cardinality of `@oci.availability_domain` is measured per `@usr.name`; a signal fires when that cardinality is unusually high for that user. + +```json +{ + "name": "Anomalous number of OCI instances created across availability domains", + "type": "log_detection", + "isEnabled": true, + "message": "User {{@usr.name}} launched OCI instances across an unusual number of availability domains.", + "tags": [ + "source:oracle-cloud-infrastructure", + "security:attack", + "tactic:TA0040-impact" + ], + "options": { + "detectionMethod": "anomaly_detection", + "evaluationWindow": 1800, + "keepAlive": 7200, + "maxSignalDuration": 86400, + "anomalyDetectionOptions": { + "learningDuration": 24, + "detectionTolerance": 5 + } + }, + "queries": [ + { + "name": "anomalous_oci_instance_creation", + "query": "source:oci.audit @evt.name:LaunchInstance", + "aggregation": "cardinality", + "distinctFields": ["@oci.availability_domain"], + "groupByFields": ["@usr.name"] + } + ], + "cases": [ + { + "status": "medium", + "name": "", + "notifications": [] + } + ] +} +``` + +### Further reading + +- [Create a detection rule (API reference)][5] +- [Log search syntax][4] +- [Security notification variables][3] + [1]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=anomaly -[2]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule/?tab=anomaly#rule-multi-triggering-rt-anomaly \ No newline at end of file +[2]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule/?tab=anomaly#rule-multi-triggering-rt-anomaly +[3]: /security/notifications/variables/ +[4]: /logs/search_syntax/ +[5]: /api/latest/security-monitoring/#create-a-detection-rule diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/content_anomaly.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/content_anomaly.md index 88a78bcfa00..7ea62df262f 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/content_anomaly.md +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/content_anomaly.md @@ -112,6 +112,119 @@ To calculate the Jaccard similarity between the two logs: $$\text"J(log1,log2)" = 2 / 8 = 0.25$$ +## API schema reference + +This section describes the `content_anomaly` rule schema. It is also used by the [Security Monitoring API][8] to create and update rules. + +### Top-level payload fields + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][6]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` and content anomaly options. See [Options](#options). | +| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). | +| `cases` | array | Cases for generating signals. See [Cases](#cases). | + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | +| `groupSignalsBy` | array\ | Additional grouping to perform on top of the existing groups in the query section. Must be a subset of the existing groups. | +| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. | + +### Queries + +Content anomaly supports multiple queries per rule. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. | +| `query` | string | yes | Query to run on logs. See [Log search syntax][7]. | +| `aggregation` | string enum | yes | The aggregation type. Allowed values: `count`, `cardinality`, `sum`, `max`, `new_value`, `geo_data`, `event_count`, `none`. | +| `groupByFields` | array\ | no | Fields to group by. | +| `distinctFields` | array\ | conditional | Field for which the cardinality is measured. Required when `aggregation` is `cardinality`. | +| `metrics` | array\ | conditional | Target fields to aggregate over. Required for `sum`, `max`, `geo_data`, `new_value` aggregations. | +| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. | +| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored. When `true`, events with missing group-by fields are processed with `N/A`. | + +### Cases + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `condition` | string | no | A case contains logical operations (`>`, `>=`, `<`, `<=`, `&&`, `||`) to determine if a signal should be generated based on the anomalous event counts from the defined queries. | +| `notifications` | array\ | no | Notification targets (for example, `@slack-*`, `@team-*`). | + +### Options + +Common `options` fields: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Must be `content_anomaly` for this method. | +| `evaluationWindow` | int (seconds) | 0–86400 | Defines the time frame for counting anomalous logs. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. | +| `keepAlive` | int (seconds) | ≥ `evaluationWindow` | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. | +| `maxSignalDuration` | int (seconds) | 3600–86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. | + +**Cross-field constraint (server-enforced):** `evaluationWindow ≤ keepAlive ≤ maxSignalDuration`. + +#### Content anomaly options + +Method-specific options controlling the anomaly detection logic. See [Anomaly detection parameters](#anomaly-detection-parameters) above for UI-level configuration. + +| Field | Type | Typical | Description | +|---|---|---|---| +| `learningDuration` | int (days) | 7 | Time window during which values are learned. No signals are generated during this phase. Range: `1`–`10` days. | +| `forgetAfter` | int (days) | 7 | How long learned values are retained before being discarded. Range: `1`–`10` days. | +| Similarity percentage threshold | int (percent) | 70 | Minimum similarity required to consider a log as normal. Range: `35`–`100`%. | +| Similar items threshold | int | 1 | Number of matching historical logs required for an incoming value to be considered normal. Range: `1`–`20`. | + +### Example payload + +```json +{ + "name": "Anomalous content in CloudTrail user creation events", + "type": "log_detection", + "isEnabled": true, + "message": "Anomalous event content detected for {{@usr.name}}.", + "tags": [ + "source:cloudtrail", + "security:attack" + ], + "options": { + "detectionMethod": "content_anomaly", + "evaluationWindow": 3600, + "keepAlive": 7200, + "maxSignalDuration": 86400 + }, + "queries": [ + { + "name": "anomalous_user_creation", + "query": "source:cloudtrail @evt.name:CreateUser", + "aggregation": "count", + "groupByFields": ["@usr.name"] + } + ], + "cases": [ + { + "name": "", + "status": "medium", + "condition": "anomalous_user_creation >= 1", + "notifications": [] + } + ] +} +``` + ## Comparing content anomaly method with other detection methods | Feature | Anomaly Detection | New Value Detection | Content Anomaly Detection | @@ -130,4 +243,7 @@ $$\text"J(log1,log2)" = 2 / 8 = 0.25$$ [2]: https://www.unicode.org/reports/tr29/tr29-22.html [3]: https://en.wikipedia.org/wiki/Jaccard_index [4]: https://en.wikipedia.org/wiki/MinHash -[5]: https://en.wikipedia.org/wiki/Locality-sensitive_hashing \ No newline at end of file +[5]: https://en.wikipedia.org/wiki/Locality-sensitive_hashing +[6]: /security/notifications/variables/ +[7]: /logs/search_syntax/ +[8]: /api/latest/security-monitoring/#create-a-detection-rule \ No newline at end of file diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule.md index 5df48210542..6a89d7e686a 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule.md +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule.md @@ -343,4 +343,201 @@ In the **Preview detection** section, check the steps, transitions, and time win {{% cloud_siem/create_suppression %}} -[1]: https://app.datadoghq.com/security/siem/rules/new \ No newline at end of file +## API schema reference + +This section describes the real-time detection rule schema, covering the `threshold` detection method (the default) and the case condition grammar that applies across all real-time detection methods. It is also used by the [Security Monitoring API][10] to create and update rules. + +For method-specific schemas, see the dedicated pages: [Anomaly][11], [New Value][12], [Impossible Travel][13], [Content Anomaly][14], [Third Party][15], [Sequence][16]. + +### Top-level payload fields + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][17]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` plus method-specific sub-objects. See [Options](#options). | +| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). | +| `cases` | array | Cases for generating signals. See [Cases](#cases). | + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | +| `groupSignalsBy` | array\ | Additional grouping to perform on top of the existing groups in the query section. Must be a subset of the existing groups. | +| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. | +| `schedulingOptions` | object | Options for scheduled rules. When this field is present, the rule runs based on the schedule (RRULE, RFC 5545). Minimum frequency is 1 day. | +| `calculatedFields` | array | Calculated fields. Only allowed for scheduled rules — in other words, when `schedulingOptions` is also defined. | +| `thirdPartyCases` | array | Required when `detectionMethod` is `third_party`; replaces `cases`. See [Third Party][15]. | + +### Queries + +A threshold rule can contain one or more queries. See method-specific pages for constraints on other detection methods. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. | +| `query` | string | yes | Query to run on logs. See [Log search syntax][18]. | +| `aggregation` | string enum | yes | The aggregation type. Allowed values: `count`, `cardinality`, `sum`, `max`, `new_value`, `geo_data`, `event_count`, `none`. | +| `groupByFields` | array\ | usually | Fields to group by. One signal is emitted per distinct group-by tuple. | +| `distinctFields` | array\ | conditional | Field for which the cardinality is measured. Required when `aggregation` is `cardinality`. | +| `metrics` | array\ | conditional | Target fields to aggregate over. Required for `sum`, `max`, `geo_data`, and `new_value` aggregations. | +| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. | +| `indexes` | array\ | no | List of indexes to query when the `dataSource` is `logs`. Only used for scheduled rules. | +| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored by the rule. When `true`, events with missing group-by fields are processed with `N/A` replacing the missing values. | + +#### Aggregation rules matrix + +Server-enforced combinations of `aggregation`, `distinctFields`, `metrics`, and `groupByFields`: + +| Aggregation | `distinctFields` | `metrics` | `groupByFields` | +|---|---|---|---| +| `count` | no | no | yes | +| `cardinality` | yes | no | yes | +| `sum` | no | yes (numeric field) | yes | +| `max` | no | yes (numeric field) | yes | +| `new_value` | no | yes (watched attribute) | yes | +| `geo_data` | no | yes (`@network.client.geoip`) | yes | +| `event_count` | no | no | no | +| `none` | no | no | no (used by `third_party`) | + +### Cases + +Cases turn query results into signals. They are evaluated **top-down, first match wins**. Order cases by decreasing severity (`critical` → `high` → `medium` → `low` → `info`) so the most important verdict is picked first. If no case matches, no signal is generated. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. Available as `{{case_name}}` in the `message` template. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `condition` | string | conditional | Expression over query aliases (for example, `failed_logins > 5`). Required when multiple cases compete; may be omitted when a rule has a single trivial case. | +| `notifications` | array\ | no | Notification targets (for example, `@slack-*`, `@team-*`, `@email-...`, `@webhook-...`). Mustache templates like `@slack-{{service.name}}` are supported. | +| `actions` | array | no | AppSec only — omit on `log_detection` rules. | + +#### Case condition grammar + +The `condition` field accepts a grammar of logical, comparison, sequential, and multiplicative operators over query aliases and non-negative integer literals. + +**Supported operators:** + +| Category | Operators | +|---|---| +| Comparison | `>`, `>=`, `<`, `<=` | +| Logical | `&&` (AND), `||` (OR) | +| Sequential | `THEN` (uppercase keyword) | +| Multiplicative | `*` against a non-negative integer literal (for example, `a > 2 * b`, `3 * a > b`) | +| Grouping | parentheses | + +**Not supported** — rejected at validation time: + +- `==`, `!=` (equality / inequality) +- `!` (logical NOT) +- `+`, `-`, `/`, `%` (arithmetic other than multiplicative) +- Negative numbers +- Decimal numbers + +**Operand rules:** + +- Query aliases must match `[a-z][a-z0-9_]*` (lowercase start, lowercase / digits / underscore after). +- `NUMBER` tokens are non-negative integers only. +- A comparison accepts these shapes: + - `query op NUMBER` (for example, `failed_logins > 5`) + - `NUMBER op query` (for example, `5 < failed_logins`) — **both forms are valid** + - `query op query` (for example, `errors > warnings`) + - Either side may be multiplied by an integer literal: `query * NUMBER` or `NUMBER * query` + +**Precedence:** `&&` binds tighter than `||`. Use parentheses to disambiguate: + +``` +(a > 0 && b > 0) || c > 10 +``` + +**`THEN` (sequential):** + +A case condition is either a single expression or ` THEN `. Exactly one `THEN` per condition is allowed. Do not combine `THEN` with top-level `&&` / `||` — group into the two sides of the `THEN`. + +`THEN` enforces **temporal ordering**: `create_user > 0 THEN attach_admin > 0` fires only when a `create_user` match precedes an `attach_admin` match in time. Rewriting the same condition as `create_user > 0 && attach_admin > 0` silently drops the ordering requirement. + +`THEN` here is the threshold-style sequential operator for `log_detection` cases. It is **not** the same feature as the [Sequence][16] detection method, which encodes ordering explicitly via `options.sequenceDetectionOptions.steps` and `stepTransitions`. + +**Anomaly detection subset:** For `anomaly_detection` rules, the parser accepts only the narrow forms `query op NUMBER` or `NUMBER op query`. See [Anomaly][11]. + +### Options + +Common `options` fields shared across detection methods: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Allowed values: `threshold`, `new_value`, `anomaly_detection`, `impossible_travel`, `third_party`, `sequence_detection`. | +| `evaluationWindow` | int (seconds) | 300 | A time window that matches when at least one of the cases evaluates true. Sliding, real-time. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. Not used for `third_party`. | +| `keepAlive` | int (seconds) | ≥ `evaluationWindow` | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. Not used for `third_party`. | +| `maxSignalDuration` | int (seconds) | 3600–86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. Decreased by one level when the environment tag starts with `staging`, `test`, or `dev`. `INFO` remains `INFO`. | + +**Cross-field constraint (server-enforced):** `evaluationWindow ≤ keepAlive ≤ maxSignalDuration`. + +For method-specific options (`newValueOptions`, `anomalyDetectionOptions`, `impossibleTravelOptions`, `thirdPartyRuleOptions`, `sequenceDetectionOptions`), see the corresponding method page. + +### Example threshold payload + +Example of a `threshold` rule that fires when there are more than 5 failed authentications within a 5-minute window, grouped by user. + +```json +{ + "name": "Multiple failed authentications from a single user", + "type": "log_detection", + "isEnabled": true, + "message": "User {{@usr.name}} failed authentication {{events_matched}} times.", + "tags": [ + "source:cloudtrail", + "security:attack", + "technique:T1110-brute-force", + "tactic:TA0006-credential-access" + ], + "options": { + "detectionMethod": "threshold", + "evaluationWindow": 300, + "keepAlive": 3600, + "maxSignalDuration": 86400, + "decreaseCriticalityBasedOnEnv": true + }, + "queries": [ + { + "name": "failed_logins", + "query": "source:cloudtrail @evt.name:ConsoleLogin @responseElements.ConsoleLogin:Failure", + "aggregation": "count", + "groupByFields": ["@usr.name"], + "distinctFields": [] + } + ], + "cases": [ + { + "name": "brute_force_attempt", + "status": "high", + "condition": "failed_logins > 5", + "notifications": [] + } + ] +} +``` + +### Further reading + +- [Create a detection rule (API reference)][10] +- [Log search syntax][18] +- [Security notification variables][17] + +[1]: https://app.datadoghq.com/security/siem/rules/new +[10]: /api/latest/security-monitoring/#create-a-detection-rule +[11]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/anomaly/ +[12]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/new_value/ +[13]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/impossible_travel/ +[14]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/content_anomaly/ +[15]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/third_party/ +[16]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/sequence/ +[17]: /security/notifications/variables/ +[18]: /logs/search_syntax/ \ No newline at end of file diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/impossible_travel.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/impossible_travel.md index 4180f329da2..53e51aa1a9f 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/impossible_travel.md +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/impossible_travel.md @@ -35,4 +35,140 @@ When **Baseline User Locations** is enabled: - If it's not an impossible travel situation, Datadog moves on to the next log or event. - It it's an impossible travel situation, Datadog checks if there is an IP transition pattern. From example, if a user travels from location A to location B and that travel pattern has occurred in the past, a signal is not triggered. +## API schema reference + +This section describes the `impossible_travel` rule schema. It is also used by the [Security Monitoring API][5] to create and update rules. + +### Trigger thresholds + +A signal triggers only when **both** thresholds are exceeded between two successive successful authentication events for the same user: + +| Condition | Threshold | +|---|---| +| Travel speed | > 1000 km/h | +| Travel distance | > 500 km | + +### Top-level payload fields + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][3]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` and `impossibleTravelOptions`. See [Options](#options). | +| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). | +| `cases` | array | Cases for generating signals. See [Cases](#cases). | + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | +| `groupSignalsBy` | array\ | Additional grouping to perform on top of existing groups. Must be a subset of the existing groups. | +| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. | + +### Queries + +An `impossible_travel` rule has exactly one query with `aggregation: "geo_data"`. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. | +| `query` | string | yes | Query to run on logs. See [Log search syntax][4]. Scope to successful authentication events from the desired source. | +| `aggregation` | string enum | yes | The aggregation type. Must be `geo_data` for this detection method. | +| `metrics` | array\ | yes | Must be `["@network.client.geoip"]` — the geo-enriched IP attribute the detector reads. | +| `groupByFields` | array\ | yes | Per-user key, typically `["@usr.name"]` or `["@usr.id"]`. One group = one user. | +| `distinctFields` | array\ | no | Leave empty (`[]`) for `impossible_travel`. | +| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. | +| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored. When `true`, events with missing group-by fields are processed with `N/A`. | + +### Cases + +An `impossible_travel` rule typically has a single case with no `condition` — the rule fires inherently when impossible travel is detected. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. Often empty for `impossible_travel`. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `condition` | string | no | Typically omitted for single-case `impossible_travel` rules. | +| `notifications` | array\ | no | Notification targets. | + +### Options + +Common `options` fields: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Must be `impossible_travel` for this method. | +| `keepAlive` | int (seconds) | 21600 | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. | +| `maxSignalDuration` | int (seconds) | 86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. Allowed values: same as `keepAlive`. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. | + +**Note:** `evaluationWindow` is **not used** for `impossible_travel` rules. Omit it from the `options` block — the detector drives its own evaluation cadence. + +#### `impossibleTravelOptions` + +Method-specific sub-object under `options`. + +| Field | Type | Description | +|---|---|---| +| `baselineUserLocations` | bool | If `true`, signals are suppressed for the first 24 hours while Datadog learns the user's regular access locations. Can reduce noise and help infer VPN usage or credentialed API access. Known-location retention is 30 days. | +| `detectIpTransition` | bool | When `true`, also considers a transition between IPs (not only between geo regions) when deciding whether two events are separate "trips". Useful where multiple distinct IPs in the same region should be treated as one location. | + +### Example payload + +Example of an `impossible_travel` rule that detects OCI SSO logins without MFA where the travel between two successive logins is implausible. Events are grouped by `@usr.name`, and corporate VPN sources are excluded. + +```json +{ + "name": "OCI ConsoleLogin without MFA — impossible travel", + "type": "log_detection", + "isEnabled": true, + "message": "Impossible travel detected for {{@usr.name}} from {{@network.client.geoip.country.name}}.", + "tags": [ + "source:oracle-cloud-infrastructure", + "security:attack", + "tactic:TA0001-initial-access" + ], + "options": { + "detectionMethod": "impossible_travel", + "keepAlive": 21600, + "maxSignalDuration": 86400, + "impossibleTravelOptions": { + "baselineUserLocations": true + } + }, + "queries": [ + { + "name": "impossible_travel_no_mfa", + "query": "source:oci.audit @evt.name:AccessApp @data.additionalDetails.eventId:sso.app.access.success -@threat_intel.results.category:corp_vpn", + "aggregation": "geo_data", + "metrics": ["@network.client.geoip"], + "groupByFields": ["@usr.name"], + "distinctFields": [] + } + ], + "cases": [ + { + "status": "medium", + "name": "", + "notifications": [] + } + ] +} +``` + +### Further reading + +- [Create a detection rule (API reference)][5] +- [Log search syntax][4] +- [Security notification variables][3] + [1]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule +[3]: /security/notifications/variables/ +[4]: /logs/search_syntax/ +[5]: /api/latest/security-monitoring/#create-a-detection-rule diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/new_value.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/new_value.md index 9e899633fc1..f0a9bdcbccb 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/new_value.md +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/new_value.md @@ -47,5 +47,154 @@ The learning duration has the following options: The [Forget value][2] option determines how long the rule keeps a value known. After this period has passed, the value is forgotten and the rule alerts on the value again. The maximum number of days for **Forget value** is 30 days. +## API schema reference + +This section describes the `new_value` rule schema. It is also used by the [Security Monitoring API][5] to create and update rules. + +### Top-level payload fields + +A `new_value` rule accepts the following top-level fields on create or update. + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][3]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` and `newValueOptions`. See [Options](#options). | +| `queries` | array | Queries for selecting logs which are part of the rule. See [Queries](#queries). | +| `cases` | array | Cases for generating signals. See [Cases](#cases). | + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | +| `groupSignalsBy` | array\ | Additional grouping to perform on top of the existing groups in the query section. Must be a subset of the existing groups. | +| `referenceTables` | array | Reference tables for the rule. Maximum of 1,000,000 rows. | +| `schedulingOptions` | object | Options for scheduled rules. When this field is present, the rule runs based on the schedule (RRULE, RFC 5545). Minimum frequency is 1 day. | +| `calculatedFields` | array | Calculated fields. Only allowed for scheduled rules - in other words, when `schedulingOptions` is also defined. | + +### Queries + +A `new_value` rule has exactly one query with `aggregation: "new_value"`. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Name of the query. Used as an alias referenced in `cases[].condition`. | +| `query` | string | yes | Query to run on logs. See [Log search syntax][4]. | +| `aggregation` | string enum | yes | The aggregation type. Must be `new_value` for this detection method. | +| `metrics` | array\ | yes | Group of target fields to aggregate over. For `new_value` aggregation, accepts up to five values — each is an attribute whose values are being watched. | +| `groupByFields` | array\ | yes | Fields to group by. Entity scope under which values are learned (for example, `["@account.id"]`). | +| `distinctFields` | array\ | no | Field for which the cardinality is measured. Leave empty (`[]`) for `new_value`. | +| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. | +| `indexes` | array\ | no | List of indexes to query when the `dataSource` is `logs`. Only used for scheduled rules. | +| `hasOptionalGroupByFields` | bool | no | When `false`, events without a group-by value are ignored by the rule. When `true`, events with missing group-by fields are processed with `N/A` replacing the missing values. | + +### Cases + +A `new_value` rule typically has a single case with no `condition` — the rule fires inherently when a new value is observed. Severity is chosen via `status`. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. Often empty for `new_value`. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `condition` | string | no | Logical expression referencing query aliases. Typically omitted for single-case `new_value` rules. | +| `notifications` | array\ | no | Notification targets (for example, `@slack-*`, `@team-*`, `@email-...`, `@webhook-...`). | + +### Options + +Common `options` fields shared with other detection methods: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Must be `new_value` for this method. | +| `evaluationWindow` | int (seconds) | 300 | A time window that matches when at least one of the cases evaluates true. Sliding, real-time. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. | +| `keepAlive` | int (seconds) | ≥ `evaluationWindow` | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. Allowed values: same as `evaluationWindow`. | +| `maxSignalDuration` | int (seconds) | 3600–86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration (measured from first-seen timestamp). Allowed values: same as `evaluationWindow`. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If true, signals in non-production environments have a lower severity than what is defined by the rule case. Severity is decreased by one level when the environment tag starts with `staging`, `test`, or `dev`. `INFO` remains `INFO`. | + +**Cross-field constraint (server-enforced):** `evaluationWindow ≤ keepAlive ≤ maxSignalDuration`. + +#### `newValueOptions` + +Method-specific sub-object under `options`. + +| Field | Type | Typical | Description | +|---|---|---|---| +| `learningMethod` | string enum | `"duration"` | The learning method used to determine when signals should be generated for values that weren't learned. Allowed values: `duration`, `threshold`. | +| `learningDuration` | int (days) | 7 | The duration in days during which values are learned, and after which signals are generated for values that weren't learned. If set to `0`, a signal is generated for all new values after the first value is learned. Allowed values: `0`, `1`, `7`. | +| `learningThreshold` | int | 0 | A number of occurrences after which signals will be generated for values that weren't learned. Used when `learningMethod: "threshold"`. Allowed values: `0`, `1`. | +| `forgetAfter` | int (days) | 28 | The duration in days after which a learned value is forgotten. Allowed values: `1`, `2`, `7`, `14`, `21`, `28`. | + +### Limits + +| Limit | Value | +|---|---| +| Attributes per rule (`queries[0].metrics`) | ≤ 5 | +| `newValueOptions.learningDuration` | ≤ 30 days | +| `newValueOptions.forgetAfter` | ≤ 30 days | +| Queries per rule | exactly 1 | + +### Example payload + +Example of a `new_value` rule that alerts when a credential is added to an Azure AD application that has rarely been used. The watched attribute is `@properties.targetResources.displayName`, grouped by `@usr.id`, so each user has their own "seen applications" list. + +```json +{ + "name": "Credential added to rarely used Azure AD application", + "type": "log_detection", + "isEnabled": true, + "message": "Credential added to Azure AD application {{@properties.targetResources.displayName}}.", + "tags": [ + "source:azure", + "security:attack", + "technique:T1098-account-manipulation", + "tactic:TA0003-persistence" + ], + "options": { + "detectionMethod": "new_value", + "maxSignalDuration": 86400, + "keepAlive": 3600, + "decreaseCriticalityBasedOnEnv": true, + "newValueOptions": { + "learningMethod": "duration", + "learningDuration": 7, + "forgetAfter": 28, + "learningThreshold": 0 + } + }, + "queries": [ + { + "name": "new_credential_added", + "query": "source:azure.activedirectory @evt.name:(\"Add service principal credentials\" OR \"Update application – Certificates and secrets management\") @evt.outcome:success @evt.category:AuditLogs", + "aggregation": "new_value", + "metrics": ["@properties.targetResources.displayName"], + "groupByFields": ["@usr.id"], + "distinctFields": [] + } + ], + "cases": [ + { + "status": "medium", + "name": "", + "notifications": [] + } + ] +} +``` + +### Further reading + +- [Create a detection rule (API reference)][5] +- [Log search syntax][4] +- [Security notification variables][3] + [1]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=newvalue -[2]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=newvalue#forget-value-rt-new-value \ No newline at end of file +[2]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=newvalue#forget-value-rt-new-value +[3]: /security/notifications/variables/ +[4]: /logs/search_syntax/ +[5]: /api/latest/security-monitoring/#create-a-detection-rule diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/sequence.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/sequence.md index 59c53801b10..0150394035f 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/sequence.md +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/sequence.md @@ -61,8 +61,183 @@ When you [create a sequence detection rule][1], you can configure these options: - Sequence detection supports up to 10 steps per rule and a total evaluation window of 24 hours. - Steps must be in a linear sequence. +## API schema reference + +This section describes the `sequence_detection` rule schema. It is also used by the [Security Monitoring API][4] to create and update rules. + +### Top-level payload fields + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. Allowed values: `log_detection`, `workload_security`, `application_security`, `api_security`, `workload_activity`. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][2]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` and `sequenceDetectionOptions`. See [Options](#options). | +| `queries` | array | One query per step in the sequence. See [Queries](#queries). | +| `cases` | array | Cases for generating signals. See [Cases](#cases). | + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | +| `groupSignalsBy` | array\ | Additional grouping. Must be a subset of the existing groups. | + +### Queries + +A `sequence_detection` rule has one query per step. Each query is linked to a step via its `name`. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Name of the query. Referenced by the corresponding step's `condition`. | +| `query` | string | yes | Query to run on logs. See [Log search syntax][3]. | +| `aggregation` | string enum | yes | Must be `count` for sequence detection. | +| `groupByFields` | array\ | yes | Fields to group by. **Must be identical across every step's query** so stages join on the same entity (for example, all queries group by `@network.client.ip`). | +| `distinctFields` | array\ | no | Leave empty (`[]`) for `sequence_detection`. | +| `dataSource` | string enum | no | Source of events. Defaults to `logs`. Allowed values: `logs`, `audit`, `app_sec_spans`, `spans`, `security_runtime`, `network`, `events`, `security_signals`. | + +**Note:** `sequence_detection` is a count-based method. `cardinality`, `new_value`, and other aggregations are not valid for step queries. + +### Cases + +A `sequence_detection` rule carries a single case whose `condition` references only the **terminal step's alias**. Do **not** use the `THEN` operator in the case condition — sequential ordering lives in `sequenceDetectionOptions`, not in the case grammar. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. Often empty. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `condition` | string | yes | Expression over the terminal step's alias (for example, `final_step > 0`). Must not contain `THEN`. | +| `notifications` | array\ | no | Notification targets. | + +### Options + +Common `options` fields: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Must be `sequence_detection` for this method. | +| `keepAlive` | int (seconds) | 3600 | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. | +| `maxSignalDuration` | int (seconds) | 86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. | + +**Note:** `evaluationWindow` at the top level of `options` is **not used** for sequence rules. Each step and each transition carries its own window. Omit the top-level field or leave it at `0`. + +#### `sequenceDetectionOptions` + +Method-specific sub-object under `options`. + +##### `steps[]` + +Ordered list of chain stages. The first entry is the entry point; the last is the terminal step referenced by the case condition. + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | yes | Unique name identifying the step. Referenced by `stepTransitions[].parent`/`child` and by the case condition. | +| `condition` | string | yes | Condition referencing the corresponding query alias (for example, `user_creation1 > 0`). Uses the same grammar as case conditions. | +| `evaluationWindow` | int (seconds) | yes | Per-step window during which the step's query must match. Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. | + +##### `stepTransitions[]` + +Transitions defining the allowed order of steps and their evaluation windows. + +| Field | Type | Required | Description | +|---|---|---|---| +| `parent` | string | yes | Name of the parent (upstream) step. | +| `child` | string | yes | Name of the child (downstream) step. | +| `evaluationWindow` | int (seconds) | yes | Maximum wait between `parent` firing and `child` firing. Allowed values: same as `steps[].evaluationWindow`. | + +### Constraints + +- **Total chain duration ≤ 24 hours (86400 seconds).** The sum of `stepTransitions[].evaluationWindow` bounds the total chain duration. +- Up to 10 steps per rule. +- Steps must be in a linear sequence. +- `groupByFields` must match across every step's query. +- Do not use `THEN` in the case condition — ordering is enforced by step order. + +### Example payload + +Example of a `sequence_detection` rule that chains three CloudTrail stages — `CreateUser` → admin policy attachment → impact action — all correlated by client IP. + +```json +{ + "name": "CloudTrail CreateUser → policy attachment → impact", + "type": "log_detection", + "isEnabled": true, + "message": "User creation followed by admin policy attachment followed by impact action, all from {{@network.client.ip}}.", + "tags": [ + "source:cloudtrail", + "security:attack", + "tactic:TA0003-persistence" + ], + "options": { + "detectionMethod": "sequence_detection", + "evaluationWindow": 0, + "keepAlive": 3600, + "maxSignalDuration": 86400, + "sequenceDetectionOptions": { + "steps": [ + { "name": "user_creation", "condition": "user_creation1 > 0", "evaluationWindow": 21600 }, + { "name": "policy_attachment", "condition": "policy_attachment1 > 0", "evaluationWindow": 21600 }, + { "name": "mitre_impact", "condition": "mitre_impact1 > 0", "evaluationWindow": 21600 } + ], + "stepTransitions": [ + { "parent": "user_creation", "child": "policy_attachment", "evaluationWindow": 21600 }, + { "parent": "policy_attachment", "child": "mitre_impact", "evaluationWindow": 21600 } + ] + } + }, + "queries": [ + { + "name": "user_creation1", + "query": "source:cloudtrail @evt.name:CreateUser @eventSource:iam.amazonaws.com", + "aggregation": "count", + "dataSource": "logs", + "groupByFields": ["@network.client.ip"], + "distinctFields": [] + }, + { + "name": "policy_attachment1", + "query": "source:cloudtrail @evt.name:(AttachUserPolicy OR AttachRolePolicy OR PutRolePolicy)", + "aggregation": "count", + "dataSource": "logs", + "groupByFields": ["@network.client.ip"], + "distinctFields": [] + }, + { + "name": "mitre_impact1", + "query": "source:cloudtrail @evt.name:(DeleteTrail OR StopLogging OR CreateAccessKey OR DeleteBucket)", + "aggregation": "count", + "dataSource": "logs", + "groupByFields": ["@network.client.ip"], + "distinctFields": [] + } + ], + "cases": [ + { + "status": "info", + "name": "", + "condition": "mitre_impact > 0", + "notifications": [] + } + ] +} +``` + +### Further reading + +- [Create a detection rule (API reference)][4] +- [Log search syntax][3] +- [Security notification variables][2] + ## Further reading {{< partial name="whats-next/whats-next.html" >}} [1]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=sequence +[2]: /security/notifications/variables/ +[3]: /logs/search_syntax/ +[4]: /api/latest/security-monitoring/#create-a-detection-rule diff --git a/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/third_party.md b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/third_party.md new file mode 100644 index 00000000000..56d4bd1a833 --- /dev/null +++ b/content/en/security/cloud_siem/detect_and_monitor/custom_detection_rules/third_party.md @@ -0,0 +1,188 @@ +--- +title: Third Party +description: Learn about how the third party detection method works. +disable_toc: false +--- + +## Overview + +The third party detection method maps vendor-supplied severities — from products like SentinelOne, Jamf Protect, CrowdStrike, Microsoft Defender, or CheckPoint Harmony — onto Datadog signal severities. The rule ingests events that a third-party product has already scored and emits one Datadog signal per matching vendor severity tier. + +See [Create Rule][1] for instructions on how to configure a third party rule. + +## When to use + +Use `third_party` when the upstream product is the authoritative detector and Datadog's job is to re-label the event so it surfaces as a first-class security signal. Typical integrations: + +- EDR vendors (SentinelOne, CrowdStrike, Microsoft Defender) +- MDM and endpoint-posture tools (Jamf Protect) +- Cloud-native detection products whose events already carry a severity attribute (for example, `@evt.severity`, `@severity`, `@alert.severity`) + +If Datadog should compute the severity itself (by counting, cardinality, ordering, or novelty), pick a different detection method. + +## How it works + +A third party rule has two conjunctive filter stages: + +1. **Root queries** (`rootQueries`) filter all incoming events from the vendor source and control how signals are grouped (one signal per distinct `groupByFields` tuple). +2. **Per-severity queries** (`queries`) further filter events by vendor severity level (for example, `@evt.severity:Critical`). + +Every event must match **both** a `rootQuery` AND one of the per-severity queries to generate a signal with the corresponding severity. Events that match a `rootQuery` but no per-severity query fall back to `defaultStatus`. + +## API schema reference + +This section describes the `third_party` rule schema. It is also used by the [Security Monitoring API][3] to create and update rules. + +### Top-level payload fields + +#### Required + +| Field | Type | Description | +|---|---|---| +| `name` | string | The name of the rule. | +| `type` | string enum | The rule type. Use `log_detection` for Cloud SIEM rules. | +| `message` | string | Message for generated signals. Supports Mustache templating (see [Security notification variables][2]). | +| `isEnabled` | bool | Whether the rule is enabled. | +| `options` | object | Contains `detectionMethod` and `thirdPartyRuleOptions`. See [Options](#options). | +| `queries` | array | Per-severity filter queries. See [Queries](#queries). | + +Either `cases` or `thirdPartyCases` is required — see [Cases](#cases). + +#### Optional + +| Field | Type | Description | +|---|---|---| +| `tags` | array\ | Tags for generated signals. | +| `hasExtendedTitle` | bool | Whether the notifications include the triggering group-by values in their title. | + +### Queries + +Per-severity filter queries. Each selects one vendor severity tier. + +| Field | Type | Required | Description | +|---|---|---|---| +| `query` | string | yes | Filter selecting one vendor severity level (for example, `@evt.severity:Critical`). | +| `aggregation` | string enum | yes | Must be `none` for third party rules. | +| `name` | string | no | Name of the query. | + +**Important**: Do not set `groupByFields`, `distinctFields`, or `metrics` on per-severity queries. The server rejects them when paired with `aggregation: "none"`. Signal grouping is controlled by `rootQueries[].groupByFields` instead. + +### Cases + +Two forms are accepted. For new rules, prefer `thirdPartyCases` — the pairing is explicit and survives query reordering. + +#### Canonical: `thirdPartyCases` (replaces `cases`) + +| Field | Type | Required | Description | +|---|---|---|---| +| `name` | string | no | Name of the case. | +| `query` | string | yes | A query to map a third party event to this case. | +| `status` | string enum | yes | Severity of the Security Signal. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `notifications` | array\ | no | Notification targets for each case. | + +#### Alternate: flat `cases` pairing 1-to-1 with `queries` + +A flat `cases` array where each entry sets only `status`. The Nth case inherits the filter from the Nth entry of `queries`. This positional pairing is used by Datadog-shipped rules. + +```json +"cases": [ + { "status": "low" }, + { "status": "medium" }, + { "status": "high" }, + { "status": "critical" } +] +``` + +Reordering one array without the other silently breaks the mapping. + +### Options + +Common `options` fields: + +| Field | Type | Typical | Description | +|---|---|---|---| +| `detectionMethod` | string enum | required | The detection method. Must be `third_party` for this method. | +| `evaluationWindow` | int (seconds) | 1800 | Allowed values: `0`, `60`, `300`, `600`, `900`, `1800`, `3600`, `7200`, `10800`, `21600`. For `third_party`, this field is not used in the classical sense but is accepted. | +| `keepAlive` | int (seconds) | 1800 | Once a signal is generated, the signal remains "open" if a case is matched at least once within this keep-alive window. For `third_party` rules, this field is not used. | +| `maxSignalDuration` | int (seconds) | 86400 | A signal "closes" regardless of case matching once the time exceeds the maximum duration. | +| `decreaseCriticalityBasedOnEnv` | bool | `false` | If `true`, signals in non-production environments have a lower severity. | + +#### `thirdPartyRuleOptions` + +Method-specific sub-object under `options`. + +| Field | Type | Required | Description | +|---|---|---|---| +| `rootQueries` | array of `{ query, groupByFields }` | yes | Queries to be combined (AND) with each per-severity query. Each entry applies its own `query` and its own `groupByFields`, so one rule can group different alert types on different tuples. | +| `signalTitleTemplate` | string | no | A template for the signal title; if omitted, the title is generated based on the case name. Supports attribute and tag variables (see [Security notification variables][2]). | +| `defaultStatus` | string enum | no | Severity used when an event satisfies one of the `rootQueries` but none of the per-severity queries. Allowed values: `info`, `low`, `medium`, `high`, `critical`. | +| `defaultNotifications` | array\ | no | Notification targets for the logs that do not correspond to any of the cases. | + +##### `rootQueries[]` + +| Field | Type | Required | Description | +|---|---|---|---| +| `query` | string | yes | Query to run on logs. Filters all incoming events from the vendor source. | +| `groupByFields` | array\ | yes | Fields to group by. Controls signal de-duplication — one signal per distinct tuple. An empty array disables grouping, so every matching event produces its own signal. | + +### Vendor patterns + +- **Single root query**: Used when all alerts from the vendor should be grouped the same way. Example: SentinelOne with one `rootQueries` entry grouping by `@agentRealtimeInfo.name`. +- **Multiple root queries**: Used when different alert types should be grouped on different tuples. Example: CrowdStrike with one `rootQueries` entry per `@evt.type`, each with its own `groupByFields`. + +### Example payload + +Example of a `third_party` rule that maps SentinelOne endpoint alert severities to Datadog signal severities, grouped by host. + +```json +{ + "name": "SentinelOne Alerts", + "type": "log_detection", + "isEnabled": true, + "message": "{{@evt.name}} on {{@agentRealtimeInfo.name}}.", + "tags": [ + "source:sentinelone", + "security:attack" + ], + "options": { + "detectionMethod": "third_party", + "evaluationWindow": 1800, + "keepAlive": 1800, + "maxSignalDuration": 86400, + "thirdPartyRuleOptions": { + "signalTitleTemplate": "SentinelOne custom alert — {{@evt.name}} on {{@agentRealtimeInfo.name}}", + "defaultStatus": "medium", + "defaultNotifications": [], + "rootQueries": [ + { + "query": "source:sentinelone endpoint:cloud_detection_alerts", + "groupByFields": ["@agentRealtimeInfo.name"] + } + ] + } + }, + "queries": [ + { "query": "@evt.severity:Low", "aggregation": "none" }, + { "query": "@evt.severity:Medium", "aggregation": "none" }, + { "query": "@evt.severity:High", "aggregation": "none" }, + { "query": "@evt.severity:Critical", "aggregation": "none" } + ], + "cases": [ + { "status": "low" }, + { "status": "medium" }, + { "status": "high" }, + { "status": "critical" } + ] +} +``` + +## Further reading + +- [Create a detection rule (API reference)][3] +- [Security notification variables][2] +- [Log search syntax][4] + +[1]: /security/cloud_siem/detect_and_monitor/custom_detection_rules/create_rule/real_time_rule?tab=thirdparty +[2]: /security/notifications/variables/ +[3]: /api/latest/security-monitoring/#create-a-detection-rule +[4]: /logs/search_syntax/ diff --git a/content/en/security/cloud_siem/detect_and_monitor/suppressions.md b/content/en/security/cloud_siem/detect_and_monitor/suppressions.md index 4500e95c176..6828bdc415f 100644 --- a/content/en/security/cloud_siem/detect_and_monitor/suppressions.md +++ b/content/en/security/cloud_siem/detect_and_monitor/suppressions.md @@ -3,4 +3,17 @@ title: Suppressions disable_toc: false --- -{{< include-markdown "security/suppressions" >}} \ No newline at end of file +{{< include-markdown "security/suppressions" >}} + +## API schema note — deprecated `filters` field + +Detection rule payloads accept a legacy `filters` array field for excluding matching events. **This field is deprecated for `log_detection`, `signal_correlation`, and `workload_security` rules.** Do not emit it on new rules. + +Use one of the following instead: + +- **Suppression rules** (recommended) — a separately managed resource at `POST /api/v2/security_monitoring/configuration/suppressions`. Scope to one or more rules, filter by a Datadog query, and optionally set `start_date` and `expiration_date` bounds. Suppressions live and evolve independently of the rule that generates the signal, so a noisy rule can be silenced without re-shipping it. +- **Tighten `queries[].query`** — use when the exclusion is intrinsic to the rule and you want it to travel with the rule payload. + +For the suppression API reference, see the [Security Monitoring API documentation][1]. + +[1]: /api/latest/security-monitoring/#create-a-suppression-rule