Add ProjectDiscovery Cloud integration with changelog datastream #15760

yahyaghani · 2025-10-27T00:19:47Z

Description

This PR implements the ProjectDiscovery Cloud integration with the changelogs and export datastream.

Current Status:

✅ CEL input with offset-based pagination
✅ ECS-compliant field mapping
✅ Request tracing support for debugging
✅ Docker mock server for testing
✅ Pipeline and system tests
✅ Rally benchmark configuration

Implementation Details:

Uses CEL (Common Expression Language) input for efficient API polling
Pulls vulnerability status change events from /v1/scans/vuln/changelogs
Supports configurable batch size, time windows, and collection intervals
Publisher pipeline host enrichment disabled (agentless-ready)

Relates to #15061

TODO:

Implement second datastream for /v1/scans/results/export endpoint (all vulnerability results)
Final documentation and field mapping review

packages/projectdiscovery_cloud/data_stream/vulnerability/agent/stream/stream.yml.hbs

clement-fouque

I haven't been able to ingest data into the stack. I have the following error:

"error": {
      "message": [
        "failed eval: ERROR: <input>:26:32: no such overload\n |       (resp.StatusCode == 200) ?\n | ...............................^",
        "Processor json with tag json_event_original in pipeline logs-projectdiscovery_cloud.changelogs-0.1.1 failed with message: field [original] not present as part of path [event.original]"
      ]
    },

clement-fouque · 2025-11-26T10:17:38Z

packages/projectdiscovery_cloud/data_stream/changelogs/manifest.yml

+      - name: base_url
+        type: text
+        title: ProjectDiscovery Cloud API Base URL
+        description: The base URL for the ProjectDiscovery Cloud API (e.g., https://api.projectdiscovery.io)
+        multi: false
+        required: true
+        show_user: true
+        default: https://api.projectdiscovery.io
+      - name: api_key
+        type: password
+        title: API Key
+        description: The API key for authenticating to ProjectDiscovery Cloud.
+        multi: false
+        required: true
+        show_user: true
+        secret: true
+      - name: team_id
+        type: text
+        title: Team ID
+        description: The Team ID for your ProjectDiscovery Cloud account.
+        multi: false
+        required: true
+        show_user: true


Suggested change

- name: base_url

type: text

title: ProjectDiscovery Cloud API Base URL

description: The base URL for the ProjectDiscovery Cloud API (e.g., https://api.projectdiscovery.io)

multi: false

required: true

show_user: true

default: https://api.projectdiscovery.io

- name: api_key

type: password

title: API Key

description: The API key for authenticating to ProjectDiscovery Cloud.

multi: false

required: true

show_user: true

secret: true

- name: team_id

type: text

title: Team ID

description: The Team ID for your ProjectDiscovery Cloud account.

multi: false

required: true

show_user: true

Those properties are already defined at the “parent” level. As they will be consistent across all data streams, it’s best to define them at the parent level.

clement-fouque · 2025-11-26T10:33:39Z

packages/projectdiscovery_cloud/docs/README.md

+**Minimum versions:**
+- Kibana: `^9.1.0`
+- Elasticsearch: Compatible with Kibana version
+- Elastic subscription: `platinum`


I believe it's false since it should be available in the basic/open-source subscription.

clement-fouque · 2025-11-26T10:35:36Z

packages/projectdiscovery_cloud/docs/README.md

@@ -0,0 +1,499 @@
+# ProjectDiscovery Cloud Integration


Documentation is automatically generated if you define it in packages/XXX/_dev/build/docs/README.md

elastic-vault-github-plugin-prod · 2025-12-09T03:29:42Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

qcorporation

I'll want to modify

integrations/.github/ISSUE_TEMPLATE/integration_bug.yml
integrations/.github/ISSUE_TEMPLATE/integration_feature_request.yml

warning: incomplete review - we'll need to come back to this

qcorporation · 2025-12-15T14:49:14Z

packages/projectdiscovery_cloud/changelog.yml

@@ -0,0 +1,21 @@
+# newer versions go on top


You'll want to collapse this into one entry as there will be one PR to reference to merge into main

qcorporation · 2025-12-15T15:11:03Z

packages/projectdiscovery_cloud/_dev/build/docs/README.md

@@ -0,0 +1,191 @@
+# ProjectDiscovery Cloud Integration


You'll most likely want to follow the documentation template that's been created by our team: https://www.elastic.co/docs/extend/integrations/documentation-guidelines

Reach out to @mjwolf and the Docs II team as he'll have the ability to auto-generate this documentation for you.

qcorporation · 2025-12-15T15:14:00Z

packages/projectdiscovery_cloud/manifest.yml

+  - security
+conditions:
+  kibana:
+    version: "^9.1.0"


Do we want to offer this integration to 8.x stack as well?

qcorporation · 2025-12-15T15:16:12Z

packages/projectdiscovery_cloud/manifest.yml

+
+type: integration
+categories:
+  - security


might want to consider adding -cloud here

qcorporation · 2025-12-15T15:17:34Z

packages/projectdiscovery_cloud/manifest.yml

+    title: ProjectDiscovery Cloud
+    description: Collect vulnerability changelogs and export results from ProjectDiscovery Cloud
+    inputs:
+      - type: cel


does this cel input require ssl configuraiton?

qcorporation · 2025-12-15T15:43:46Z

packages/projectdiscovery_cloud/data_stream/export/fields/fields.yml

+      type: keyword
+    - name: remediation
+      type: text
+    - name: reference


can we use vulnerability.reference?

qcorporation · 2025-12-15T15:44:34Z

packages/projectdiscovery_cloud/data_stream/export/fields/fields.yml

+      type: keyword
+    - name: category
+      type: keyword
+    - name: request


can we use any of the ecs http.request.* fields?

qcorporation · 2025-12-15T15:45:12Z

packages/projectdiscovery_cloud/data_stream/export/fields/fields.yml

+      type: keyword
+    - name: request
+      type: keyword
+    - name: response


can we use any of the http.response.* fields?

qcorporation · 2025-12-15T16:12:38Z

packages/projectdiscovery_cloud/data_stream/changelogs/fields/fields.yml

+- name: projectdiscovery
+  type: group
+  fields:
+    - name: target


could you use destination.* within ecs, possibly destination.address

qcorporation · 2025-12-15T16:13:00Z

packages/projectdiscovery_cloud/data_stream/changelogs/fields/fields.yml

+      type: keyword
+    - name: vuln_hash
+      type: keyword
+    - name: scan_id


can you use vulnerability.report_id

clement-fouque

I completed my review by adding multiple comments. Please let me know if you need clarification.

clement-fouque · 2025-12-15T12:46:45Z

packages/projectdiscovery_cloud/data_stream/changelogs/fields/fields.yml

+# Custom vulnerability fields not in ECS 9.2.0
+- name: vulnerability.status
+  type: keyword
+  description: The state of the vulnerability (e.g., open, closed, resolved).
+- name: vulnerability.scanner.type
+  type: keyword
+  description: The type of vulnerability scanner used (e.g., nuclei).


As mentioned by Dan here:

If you can avoid adding capitalised fields to published products, that would be ideal

I believe they should be removed.

clement-fouque · 2025-12-15T12:51:28Z

packages/projectdiscovery_cloud/data_stream/changelogs/agent/stream/cel.yml.hbs

@@ -0,0 +1,96 @@
+config_version: 2
+interval: {{interval}}
+resource.max_executions: 1000


max_executions has a default value of 1000. I think it must be either removed, either we should add a field in the configuration (example).

clement-fouque · 2025-12-15T12:55:35Z

...ages/projectdiscovery_cloud/data_stream/changelogs/elasticsearch/ingest_pipeline/default.yml

+  - rename:
+      field: json.vuln_status
+      target_field: vulnerability.status
+      ignore_missing: true


This field doesn't exist in vulnerability ECS field. It must be removed.

clement-fouque · 2025-12-15T12:56:05Z

...ages/projectdiscovery_cloud/data_stream/changelogs/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: vulnerability.scanner.type
+      value: nuclei


This field doesn't exist in vulnerability ECS field. It must be removed.

clement-fouque · 2025-12-16T11:05:28Z

packages/projectdiscovery_cloud/data_stream/changelogs/agent/stream/cel.yml.hbs

+    state.with(
+      {
+        "base": state.url.trim_right("/") + "/v1/scans/vuln/changelogs",
+        "q": {


For readability, it would be easier to rename p as post and q as query

clement-fouque · 2025-12-16T15:13:56Z

packages/projectdiscovery_cloud/data_stream/export/fields/ecs.yml

@@ -0,0 +1,33 @@
+- external: ecs


I'm not sure it's required as ECS fields could be inherited (to be confirmed). For example, in the Qualys GAV, cloud ECS fields are not manually defined: https://github.com/elastic/integrations/tree/main/packages/qualys_gav/data_stream/asset/fields

clement-fouque · 2025-12-16T15:15:02Z

packages/projectdiscovery_cloud/data_stream/export/manifest.yml

+        multi: false
+        required: false
+        show_user: true
+        default: low


I believe default should be removed (or set to low,medium,high,critical).

clement-fouque · 2025-12-16T15:36:55Z

packages/projectdiscovery_cloud/data_stream/export/elasticsearch/ingest_pipeline/default.yml

+  - date:
+      field: json.created_at
+      target_field: '@timestamp'
+      formats:
+        - ISO8601
+      if: ctx.json?.created_at != null && ctx.json.created_at != ''
+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
+  - date:
+      field: json.updated_at
+      target_field: '@timestamp'
+      formats:
+        - ISO8601
+      if: ctx.json?.created_at == null && ctx.json?.updated_at != null && ctx.json.updated_at != ''
+      on_failure:
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'


I don't like the behaviour of modifying the @timestamp based on the created_at or updated_at fields. It makes trending impossible, unless we are created a transform that will store daily values.

I would be in favour to delete them in order to store the full export at each interval.

clement-fouque · 2025-12-16T15:39:28Z

...ages/projectdiscovery_cloud/data_stream/changelogs/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: event.module
+      value: projectdiscovery_cloud
+  - set:
+      field: event.dataset
+      value: projectdiscovery_cloud.changelogs


Can you align field names, dataset name and tags to either projectdiscovery or projectdiscovery_cloud ?

clement-fouque · 2025-12-16T15:48:00Z

packages/projectdiscovery_cloud/data_stream/export/manifest.yml

@@ -0,0 +1,89 @@
+title: Collect Vulnerability Results from ProjectDiscovery Cloud


I don't know the root cause but fields nesting is not working on export datastream. It's working on changelogs though.

- Implements changelogs datastream for incremental vulnerability updates - Includes optional target field sanitizer (gated by 'sanitize_target' tag) to clean malformed API responses - Pipeline tests for both pass-through and sanitized cases - Full ECS mapping with vulnerability fields - Supports configurable interval, batch_size, and time_window - Auto-generated documentation in _dev/build/docs/README.md

@mjwolf

…ngelog per Quan’s review (elastic#15760) Summary - Map HTTP bodies to ECS: - http.request.body.content - http.response.body.content - Map target → destination.address with IP/domain derivation - Map scan_id → vulnerability.report_id - Use vulnerability.reference exclusively (remove vendor duplicate) - Use host.hostname exclusively (remove vendor duplicate) - Map vendor tags → ECS tags; remove json.tags - Set vulnerability.scanner.vendor=ProjectDiscovery; remove non-ECS scanner.type - Remove vendor-specific duplicate fields (projectdiscovery.*) - Add cloud category to package manifest - Widen Kibana version support to ^8.18.0 || ^9.0.0 - Collapse changelog into a single 0.1.1 entry (per package guidelines) Deferred (intentional) - SSL/TLS configuration: optional; defer until a concrete need (custom CA, TLS-inspecting proxy, self-signed certs). If required, follow standard CEL pattern (manifest var + resource.ssl wiring). - Documentation template: defer until @mjwolf returns to coordinate auto-generation and template adoption (align badges/structure). - Issue templates: repo-wide infra; Quan indicated these will be handled centrally. Impact - Improves ECS compliance and consistency across data streams - Reduces noise by removing non-ECS and vendor-duplicate fields - Improves package discoverability (cloud category) and broadens compatibility (8.18+) References - Addresses Quan’s requested changes - Changelog entry linked to PR elastic#15760

@mjwolf

…ngelog per Quan’s review (elastic#15760) Summary - Map HTTP bodies to ECS: - http.request.body.content - http.response.body.content - Map target → destination.address with IP/domain derivation - Map scan_id → vulnerability.report_id - Use vulnerability.reference exclusively (remove vendor duplicate) - Use host.hostname exclusively (remove vendor duplicate) - Map vendor tags → ECS tags; remove json.tags - Set vulnerability.scanner.vendor=ProjectDiscovery; remove non-ECS scanner.type - Remove vendor-specific duplicate fields (projectdiscovery.*) - Add cloud category to package manifest - Widen Kibana version support to ^8.18.0 || ^9.0.0 - Collapse changelog into a single 0.1.1 entry (per package guidelines) Deferred (intentional) - SSL/TLS configuration: optional; defer until a concrete need (custom CA, TLS-inspecting proxy, self-signed certs). If required, follow standard CEL pattern (manifest var + resource.ssl wiring). - Documentation template: defer until @mjwolf returns to coordinate auto-generation and template adoption (align badges/structure). - Issue templates: repo-wide infra; Quan indicated these will be handled centrally. Impact - Improves ECS compliance and consistency across data streams - Reduces noise by removing non-ECS and vendor-duplicate fields - Improves package discoverability (cloud category) and broadens compatibility (8.18+) References - Addresses Quan’s requested changes - Changelog entry linked to PR elastic#15760

@mjwolf

…ngelog per Quan’s review (elastic#15760) Summary - Map HTTP bodies to ECS: - http.request.body.content - http.response.body.content - Map target → destination.address with IP/domain derivation - Map scan_id → vulnerability.report_id - Use vulnerability.reference exclusively (remove vendor duplicate) - Use host.hostname exclusively (remove vendor duplicate) - Map vendor tags → ECS tags; remove json.tags - Set vulnerability.scanner.vendor=ProjectDiscovery; remove non-ECS scanner.type - Remove vendor-specific duplicate fields (projectdiscovery.*) - Add cloud category to package manifest - Widen Kibana version support to ^8.18.0 || ^9.0.0 - Collapse changelog into a single 0.1.1 entry (per package guidelines) Deferred (intentional) - SSL/TLS configuration: optional; defer until a concrete need (custom CA, TLS-inspecting proxy, self-signed certs). If required, follow standard CEL pattern (manifest var + resource.ssl wiring). - Documentation template: defer until @mjwolf returns to coordinate auto-generation and template adoption (align badges/structure). - Issue templates: repo-wide infra; Quan indicated these will be handled centrally. Impact - Improves ECS compliance and consistency across data streams - Reduces noise by removing non-ECS and vendor-duplicate fields - Improves package discoverability (cloud category) and broadens compatibility (8.18+) References - Addresses Quan’s requested changes - Changelog entry linked to PR elastic#15760

Implement Clement's requested changes for changelogs data stream: - Switch vulnerability field mappings from `rename` to `set/copy_from` for better transparency - Remove non-ECS `vulnerability.status` field mapping - Add vendor-specific `projectdiscovery.vuln_status` field to preserve status information - Update event message template to use `{{projectdiscovery.vuln_status}}` - Remove `vulnerability.status` field definition from schema (not in ECS spec) - Add `projectdiscovery.vuln_status` keyword field to schema This change ensures better ECS compliance while preserving all vendor-specific data in the projectdiscovery namespace, aligning with the preserve_duplicate_custom_fields pattern.

@timestamp

Address review feedback from Clement Fouque, including critical system test fixes and four enhancement areas. ## System Test Fixes Fixed CEL syntax errors blocking system tests: - Added missing closing parentheses in state.with() calls - Removed trailing commas in CEL object literals - Removed publisher_pipeline.disable_host setting that blocked data routing System tests now pass: changelogs (4 hits), export (1 hit) ## SSL/TLS Configuration Support Added optional SSL configuration for enterprise environments: - New `ssl` variable in both data stream manifests - Supports verification_mode, certificate_authorities, ca_trusted_fingerprint - Default: all commented out, doesn't affect standard HTTPS ## Export Timestamp Semantics Changed export data stream to use ingestion time for @timestamp: - Removed date processors that toggled between created_at/updated_at - Vendor timestamps preserved in projectdiscovery.created_at/updated_at - Enables proper trending for snapshot-style exports ## Field Organization - Removed redundant ecs.yml files (inherited via index templates) - Removed subobjects: false from export manifest (fixes field nesting) - Removed default: low from severity filter (matches "export all" description) ## Test Results - ✅ Package check: Pass - ✅ Pipeline tests: Pass (changelogs 4/4, export 2/2) - ✅ System tests: Pass (4 hits, 1 hit) Fixes elastic#15061 Co-authored-by: Clement Fouque <[email protected]>

…ry-cloud-integration

…pdates Fixes elastic#15061

elasticmachine · 2026-01-05T03:34:52Z

💚 Build Succeeded

Buildkite Build
Commit: d0dc964

History

💔 Build #36083 failed fbd6962
💔 Build #36082 failed 88d1688
💚 Build #36068 succeeded c3b370b
💚 Build #36047 succeeded e3f83fc
💚 Build #36046 succeeded e6ac7cd

yahyaghani requested a review from a team as a code owner October 27, 2025 00:19

yahyaghani marked this pull request as draft October 27, 2025 00:20

efd6 reviewed Oct 27, 2025

View reviewed changes

packages/projectdiscovery_cloud/data_stream/vulnerability/agent/stream/stream.yml.hbs Outdated Show resolved Hide resolved

andrewkroh added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. New Integration Issue or pull request for creating a new integration package. labels Oct 27, 2025

clement-fouque requested changes Nov 26, 2025

View reviewed changes

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch from 5d9cfc9 to 3c4a36e Compare December 9, 2025 01:13

qcorporation reviewed Dec 15, 2025

View reviewed changes

clement-fouque requested changes Dec 16, 2025

View reviewed changes

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch 5 times, most recently from a68ca88 to ca7cc91 Compare December 18, 2025 09:49

yahyaghani changed the title ~~[DRAFT] Add ProjectDiscovery Cloud integration with changelog datastream~~ Add ProjectDiscovery Cloud integration with changelog datastream Dec 18, 2025

yahyaghani marked this pull request as ready for review December 18, 2025 10:27

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch from ca7cc91 to e6ac7cd Compare January 2, 2026 06:30

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch from e6ac7cd to e3f83fc Compare January 2, 2026 06:57

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch from e3f83fc to c3b370b Compare January 3, 2026 04:34

yahyaghani and others added 2 commits January 3, 2026 06:45

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch from 7d39066 to 88d1688 Compare January 4, 2026 23:10

Merge remote-tracking branch 'upstream/main' into feat/projectdiscove…

fbd6962

…ry-cloud-integration

yahyaghani requested a review from clement-fouque January 5, 2026 00:32

build: regenerate policy snapshots after CEL formatting and default u…

d0dc964

…pdates Fixes elastic#15061

yahyaghani force-pushed the feat/projectdiscovery-cloud-integration branch from 2e45cee to d0dc964 Compare January 5, 2026 03:10

yahyaghani requested a review from qcorporation January 5, 2026 10:14

		@@ -0,0 +1,89 @@
		title: Collect Vulnerability Results from ProjectDiscovery Cloud

Add ProjectDiscovery Cloud integration with changelog datastream #15760

Are you sure you want to change the base?

Add ProjectDiscovery Cloud integration with changelog datastream #15760

Conversation

yahyaghani commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Current Status:

Implementation Details:

TODO:

Uh oh!

Uh oh!

clement-fouque left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clement-fouque Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 9, 2025

🚀 Benchmarks report

Uh oh!

qcorporation left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clement-fouque left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jan 5, 2026

💚 Build Succeeded

History

Uh oh!

yahyaghani commented Oct 27, 2025 •

edited

Loading

clement-fouque Nov 26, 2025 •

edited

Loading