From 5094fb10c91150f7ec56bb3578f27fb59926551d Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 24 Oct 2024 12:19:50 +0200 Subject: [PATCH 01/21] docs(auditlogs): add audit logging sig proposal --- projects/audit-logging.md | 109 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 projects/audit-logging.md diff --git a/projects/audit-logging.md b/projects/audit-logging.md new file mode 100644 index 000000000..e6f2cab9c --- /dev/null +++ b/projects/audit-logging.md @@ -0,0 +1,109 @@ + + +# Audit Logging + +## Background and description + + + +Audit logging describes the capability of capturing audit-trail relevant events of a system to meet compliance requirements. Such events may originate from the infrastructure (e.g. a Kubernetes cluster) up to the application-level. It is a capability that is particularly relevant for providers of enterprise software. + +Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). + +Examples of audit logs include: +- permission changes (e.g. of a service account or application user) +- modification of data +- accessing sensitive information +- failed login attempts + +### Current challenges + + + +Audit Logging is currently not within the scope of OpenTelemetry + +- no semantic conventions for audit logs in OTEL +- OTEL APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. +- OTEL collectors may lose audit logs in transit (i.e. no guarantee of delivery) + +### Goals, objectives, and requirements + + + +The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: + +- REQ-CON-01: Semantic conventions for application-level audit logs are defined +- REQ-CON-02: Semantic conventions for infrastructure-level audit logs are defined +- REQ-SRC-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. +- REQ-PIP-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted + +## Deliverables + + + + + +- semantic convention for audit logs +- extend OTEL APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) +- extend OTEL collector for audit logging purposes (in collaboration with the respective SIG) + +## Staffing / Help Wanted + + + +The following vendors are interested in improving this area: +- SAP + +Other vendors are invited to join the discussion. + +### Required staffing + + + + + + + + +* Project lead: SAP (name tbd) +* Sponsors: tbd +* GC liaison: tbd +* Engineers: + * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) +* Maintainers/approvers: tbd + +## Timeline + + + +TBD based on community involvement. + +## Labels + + + +- audit-logging (tbc) + +## Project Board + + + + + + + + + + + +TODO: add link + +## SIG Meetings and Other Info + + + + + + + +TODO: add information \ No newline at end of file From 9337b7f9df3bd9bf3835dbf3ef53f81a0392f1a7 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 24 Oct 2024 12:21:24 +0200 Subject: [PATCH 02/21] docs(auditlogs): re-number requirements --- projects/audit-logging.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index e6f2cab9c..7e6e0afd4 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -32,10 +32,10 @@ Audit Logging is currently not within the scope of OpenTelemetry The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: -- REQ-CON-01: Semantic conventions for application-level audit logs are defined -- REQ-CON-02: Semantic conventions for infrastructure-level audit logs are defined -- REQ-SRC-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. -- REQ-PIP-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-CONV-01: Semantic conventions for application-level audit logs are defined +- REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined +- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. +- REQ-PIPE-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted ## Deliverables From 75f2c579dfc4b897a9eb91fd2e1575c987e62e2c Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 24 Oct 2024 12:22:48 +0200 Subject: [PATCH 03/21] docs(auditlogs): remove template instructions --- projects/audit-logging.md | 41 --------------------------------------- 1 file changed, 41 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 7e6e0afd4..a749f5165 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -1,11 +1,7 @@ - - # Audit Logging ## Background and description - - Audit logging describes the capability of capturing audit-trail relevant events of a system to meet compliance requirements. Such events may originate from the infrastructure (e.g. a Kubernetes cluster) up to the application-level. It is a capability that is particularly relevant for providers of enterprise software. Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). @@ -18,8 +14,6 @@ Examples of audit logs include: ### Current challenges - - Audit Logging is currently not within the scope of OpenTelemetry - no semantic conventions for audit logs in OTEL @@ -28,8 +22,6 @@ Audit Logging is currently not within the scope of OpenTelemetry ### Goals, objectives, and requirements - - The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: - REQ-CONV-01: Semantic conventions for application-level audit logs are defined @@ -39,18 +31,12 @@ The goal of this project is to make OTEL fit for audit logging purposes that mee ## Deliverables - - - - - semantic convention for audit logs - extend OTEL APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) - extend OTEL collector for audit logging purposes (in collaboration with the respective SIG) ## Staffing / Help Wanted - - The following vendors are interested in improving this area: - SAP @@ -58,13 +44,6 @@ Other vendors are invited to join the discussion. ### Required staffing - - - - - - - * Project lead: SAP (name tbd) * Sponsors: tbd * GC liaison: tbd @@ -74,36 +53,16 @@ Other vendors are invited to join the discussion. ## Timeline - - TBD based on community involvement. ## Labels - - - audit-logging (tbc) ## Project Board - - - - - - - - - - TODO: add link ## SIG Meetings and Other Info - - - - - - TODO: add information \ No newline at end of file From f81c2f44c5e5142169f44d542298189c1f7ccf54 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit <2060790+mlenkeit@users.noreply.github.com> Date: Tue, 19 Nov 2024 14:24:44 +0100 Subject: [PATCH 04/21] docs(auditlogs): use OTel over OTEL Co-authored-by: Reiley Yang --- projects/audit-logging.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index a749f5165..8084321e6 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -16,24 +16,24 @@ Examples of audit logs include: Audit Logging is currently not within the scope of OpenTelemetry -- no semantic conventions for audit logs in OTEL -- OTEL APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. -- OTEL collectors may lose audit logs in transit (i.e. no guarantee of delivery) +- no semantic conventions for audit logs in OTel +- OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. +- OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery) ### Goals, objectives, and requirements -The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: +The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: - REQ-CONV-01: Semantic conventions for application-level audit logs are defined - REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined - REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. -- REQ-PIPE-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted ## Deliverables - semantic convention for audit logs -- extend OTEL APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) -- extend OTEL collector for audit logging purposes (in collaboration with the respective SIG) +- extend OTel APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) +- extend OTel collector for audit logging purposes (in collaboration with the respective SIG) ## Staffing / Help Wanted From 65ae32e7b31d2628f345ad2f56396df0c6f7821a Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:40:29 +0100 Subject: [PATCH 05/21] docs(auditlogs): list @reyang as first sponsor --- projects/audit-logging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 8084321e6..d6e3519a2 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -45,7 +45,9 @@ Other vendors are invited to join the discussion. ### Required staffing * Project lead: SAP (name tbd) -* Sponsors: tbd +* Sponsors: + - Reiley Yang (@reyang) + - tbd * GC liaison: tbd * Engineers: * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) From 776b821f8aebcffc8ce41e882375dce0a1e6d430 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:45:22 +0100 Subject: [PATCH 06/21] docs(auditlogs): add Microsoft to interested vendors --- projects/audit-logging.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index d6e3519a2..aaa0c92fd 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -39,6 +39,7 @@ The goal of this project is to make OTel fit for audit logging purposes that mee The following vendors are interested in improving this area: - SAP +- Microsoft Other vendors are invited to join the discussion. From 6dd519d9861a088c816bb8a1d10642555c5a6090 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:45:45 +0100 Subject: [PATCH 07/21] docs(auditlogs): add contacts to vendor list --- projects/audit-logging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index aaa0c92fd..bf720eb67 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -38,8 +38,8 @@ The goal of this project is to make OTel fit for audit logging purposes that mee ## Staffing / Help Wanted The following vendors are interested in improving this area: -- SAP -- Microsoft +- SAP (@mlenkeit, @FWinkler79) +- Microsoft (@reyang) Other vendors are invited to join the discussion. From 2ec002d8de4ce7a74759ba80310837523edeb85b Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:48:52 +0100 Subject: [PATCH 08/21] docs(auditlogs): use consistent punctuation for requirement list --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index bf720eb67..77be9c121 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -26,7 +26,7 @@ The goal of this project is to make OTel fit for audit logging purposes that mee - REQ-CONV-01: Semantic conventions for application-level audit logs are defined - REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined -- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. +- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK - REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted ## Deliverables From d7e265f5eec18374dc3015ba0dd830d0b72e9601 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit <2060790+mlenkeit@users.noreply.github.com> Date: Wed, 20 Nov 2024 16:00:51 +0100 Subject: [PATCH 09/21] docs(auditlogs): minor word change in Challenges chapter --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 77be9c121..0b0a6e4e6 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -14,7 +14,7 @@ Examples of audit logs include: ### Current challenges -Audit Logging is currently not within the scope of OpenTelemetry +OpenTelemetry does not have a good solution for audit logging - no semantic conventions for audit logs in OTel - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. From 405ddb540f0628577d6be167eeec28827b8d6681 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 13:37:34 +0100 Subject: [PATCH 10/21] docs(auditlogs): describe guarantee of delivery in appendix --- projects/audit-logging.md | 40 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 77be9c121..0709e6f32 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -68,4 +68,42 @@ TODO: add link ## SIG Meetings and Other Info -TODO: add information \ No newline at end of file +TODO: add information + +## Appendix + +### Appendix A: Guarantee of Delivery + +In the context of this document, guarantee of delivery describes the ability of delivering audit logs from source to destination through OTel means while ensuring that all such signals arrive at the destination and/or providing the source with a means to handle failed delivery. + +Messaging protocols that support different levels of delivery guarantees may refer to this behavior as _at least once_ or _exactly once_, as opposed to _at most once_. + +We assume that every component that is involved in the delivery of audit logs from source to destination must support guarantee of delivery individually, rather than assuming that this ability can be provided by e.g. only the collector or SDK. + +The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel collector, and a durable storage. The workload acts as the source and produces audit logs via the OTel SDK/API. It writes the data via OTLP to the OTel collector. The OTel collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket. + +The following implications would apply: + +- workload produces an audit-relevant event: + + The workload emits the event via the OTel SDK/API. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc. + +- OTel collector receives the event: + + To ensure that the event is not lost even if the collector process is terminated or crashes, the collector may need to persist the event before acknowledging receipt to the workload or SDK. If the event cannot be persisted, receipt must be rejected. + +- OTel collector exports the event: + + Once the event is exported and the target (i.e. S3) acknowledges receipt, the event can dropped from the collector's persistence. + +- S3 receives the event: + + Acknowledges receipt after persisting the event. + + Note that this is outside the scope of the OTel. More general, when using OTel for audit logging purposes, it's the users (e.g. Ops) responsibility to configure a suitable export target. + +Note that this example may contain implementation details for illustration purposes. The actual implementation may differ as long as the requirements are met. + +The example is kept simple for illustration purposes. Many edge cases need to be discussed by the SIG, such as batch-sending of signals or handling of multiple export targets. + +It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. \ No newline at end of file From 0adb8e5933af2929e0390668f637089111a318e3 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:23:09 +0100 Subject: [PATCH 11/21] docs(auditlogs): add sample audit logs to appendix --- projects/audit-logging.md | 81 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 0709e6f32..3109c2e25 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -106,4 +106,83 @@ Note that this example may contain implementation details for illustration purpo The example is kept simple for illustration purposes. Many edge cases need to be discussed by the SIG, such as batch-sending of signals or handling of multiple export targets. -It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. \ No newline at end of file +It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. + +### Appendix B: Sample Audit Log Events + +The following list contains sample audit log events in a YAML format for better readability and intentionally do not follow any OTel-related schema. + +An event consists of the event name, event-specific data, and general metadata. The individual properties of these events would ideally be reflected in common or audit log-specific semantic conventions. + +- failed login attempts + + ```yaml + event: UserLoginFailure + data: + loginMethod: oidc + failureReason: userLocked + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + application: payroll + tenant: fab54af9-f978-463e-9c02-f92db1afc2b4 + ``` + +- permission changes (e.g. of a service account or application user) + + ```yaml + event: AuthnRoleToUserAdd + data: + user: jane-doe + role: editor + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + application: payroll + tenant: fab54af9-f978-463e-9c02-f92db1afc2b4 + ``` + +- accessing sensitive information + + ```yaml + event: DppDataAccess + data: + channelType: web + channelId: https://payroll.example.com/user/jane-doe/compensation + dataSubjectType: employeeID + dataSubjectId: jane-doe + objectType: compensation + objectId: 1196f42b-8f12-4df0-9b1f-01c98d2c7291 + attribute: salary + value: 50000 + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + application: payroll + tenant: fab54af9-f978-463e-9c02-f92db1afc2b4 + ``` + + +- modification of data + + ```yaml + event: DataModification + data: + objectType: CronJob + objectId: my-sample-cronjob + attribute: schedule + oldValue: 0 0 1 * * # monthly + newValue: 0 0 1 1 * # annually + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + k8sCluster: my-sample-cluster + ``` \ No newline at end of file From a5ef343bafcd9141c3f41f10625fb8e336da1cbb Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:27:23 +0100 Subject: [PATCH 12/21] docs(auditlogs): add links to sample audit logs --- projects/audit-logging.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 3109c2e25..dd4df711a 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -6,11 +6,11 @@ Audit logging describes the capability of capturing audit-trail relevant events Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). -Examples of audit logs include: +Examples of audit logs include: (see [Appendix B: Sample Audit Log Events]) +- failed login attempts - permission changes (e.g. of a service account or application user) -- modification of data - accessing sensitive information -- failed login attempts +- modification of data ### Current challenges @@ -185,4 +185,7 @@ An event consists of the event name, event-specific data, and general metadata. ip: 10.11.12.13 initiator: john-doe k8sCluster: my-sample-cluster - ``` \ No newline at end of file + ``` + + +[Appendix B: Sample Audit Log Events]: #appendix-b-sample-audit-log-events \ No newline at end of file From 711dc46f2cd64bafdfa3982c5ba832c616486ba3 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:27:42 +0100 Subject: [PATCH 13/21] docs(auditlogs): add links to appendix A --- projects/audit-logging.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index dd4df711a..b1ebcfd6d 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -20,6 +20,8 @@ Audit Logging is currently not within the scope of OpenTelemetry - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. - OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery) +See [Appendix A: Guarantee of Delivery] for more details + ### Goals, objectives, and requirements The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: @@ -29,6 +31,8 @@ The goal of this project is to make OTel fit for audit logging purposes that mee - REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK - REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted +See [Appendix A: Guarantee of Delivery] for more details + ## Deliverables - semantic convention for audit logs @@ -188,4 +192,5 @@ An event consists of the event name, event-specific data, and general metadata. ``` +[Appendix A: Guarantee of Delivery]: #appendix-a-guarantee-of-delivery [Appendix B: Sample Audit Log Events]: #appendix-b-sample-audit-log-events \ No newline at end of file From 087865c139a17be07204a5784279cfe2c7d4e01c Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:28:11 +0100 Subject: [PATCH 14/21] docs(auditlogs): use GitHub handle only in staffing list --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index b1ebcfd6d..072321d45 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -51,7 +51,7 @@ Other vendors are invited to join the discussion. * Project lead: SAP (name tbd) * Sponsors: - - Reiley Yang (@reyang) + - @reyang - tbd * GC liaison: tbd * Engineers: From 3876a315eaad74539bc52110f8a430a985acdb55 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:28:51 +0100 Subject: [PATCH 15/21] docs(auditlogs): add svrnm as GC liaison --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 072321d45..da4112ea8 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -53,7 +53,7 @@ Other vendors are invited to join the discussion. * Sponsors: - @reyang - tbd -* GC liaison: tbd +* GC liaison: @svrnm * Engineers: * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) * Maintainers/approvers: tbd From 066501b92566ce770eb79f5633d2c1e5166aa9a5 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Fri, 22 Nov 2024 12:13:50 +0100 Subject: [PATCH 16/21] docs(auditlogs): minor changes in wording --- projects/audit-logging.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 227c866a0..214c5026c 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -6,7 +6,7 @@ Audit logging describes the capability of capturing audit-trail relevant events Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). -Examples of audit logs include: (see [Appendix B: Sample Audit Log Events]) +Examples of audit logs include: (see [Appendix B: Examples of Audit Log Events]) - failed login attempts - permission changes (e.g. of a service account or application user) - accessing sensitive information @@ -18,7 +18,7 @@ OpenTelemetry does not have a good solution for audit logging - no semantic conventions for audit logs in OTel - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. -- OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery) +- OTel Collector instances may lose audit logs in transit (i.e. no guarantee of delivery) See [Appendix A: Guarantee of Delivery] for more details @@ -29,7 +29,7 @@ The goal of this project is to make OTel fit for audit logging purposes that mee - REQ-CONV-01: Semantic conventions for application-level audit logs are defined - REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined - REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK -- REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-PIPE-01: OTel Collector instances must provide guaranteed delivery of audit logs, including when its process is interrupted See [Appendix A: Guarantee of Delivery] for more details @@ -37,7 +37,7 @@ See [Appendix A: Guarantee of Delivery] for more details - semantic convention for audit logs - extend OTel APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) -- extend OTel collector for audit logging purposes (in collaboration with the respective SIG) +- extend OTel Collector for audit logging purposes (in collaboration with the respective SIG) ## Staffing / Help Wanted @@ -54,8 +54,9 @@ Other vendors are invited to join the discussion. - @reyang - tbd * GC liaison: @svrnm -* Engineers: +* Engineers for API/SDK: * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) +* Engineers for OTel Collector: tbd * Maintainers/approvers: tbd ## Timeline @@ -84,23 +85,23 @@ Messaging protocols that support different levels of delivery guarantees may ref We assume that every component that is involved in the delivery of audit logs from source to destination must support guarantee of delivery individually, rather than assuming that this ability can be provided by e.g. only the collector or SDK. -The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel collector, and a durable storage. The workload acts as the source and produces audit logs via the OTel SDK/API. It writes the data via OTLP to the OTel collector. The OTel collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket. +The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel Collector instance, and a durable storage. The workload acts as the source and produces audit logs via the OTel API/SDK. It writes the data via OTLP to the collector. The collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket. The following implications would apply: - workload produces an audit-relevant event: - The workload emits the event via the OTel SDK/API. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc. + The workload emits the event via the OTel API/SDK. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc. -- OTel collector receives the event: +- OTel Collector receives the event: To ensure that the event is not lost even if the collector process is terminated or crashes, the collector may need to persist the event before acknowledging receipt to the workload or SDK. If the event cannot be persisted, receipt must be rejected. -- OTel collector exports the event: +- OTel Collector exports the event: Once the event is exported and the target (i.e. S3) acknowledges receipt, the event can dropped from the collector's persistence. -- S3 receives the event: +- the target (i.e. S3) receives the event: Acknowledges receipt after persisting the event. @@ -112,7 +113,7 @@ The example is kept simple for illustration purposes. Many edge cases need to be It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. -### Appendix B: Sample Audit Log Events +### Appendix B: Examples of Audit Log Events The following list contains sample audit log events in a YAML format for better readability and intentionally do not follow any OTel-related schema. @@ -193,4 +194,4 @@ An event consists of the event name, event-specific data, and general metadata. [Appendix A: Guarantee of Delivery]: #appendix-a-guarantee-of-delivery -[Appendix B: Sample Audit Log Events]: #appendix-b-sample-audit-log-events \ No newline at end of file +[Appendix B: Examples of Audit Log Events]: #appendix-b-examples-of-audit-log-events \ No newline at end of file From 70cbac40fad98ce67683de7fbf960c12c3beaecb Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Fri, 22 Nov 2024 13:00:19 +0100 Subject: [PATCH 17/21] docs(auditlogs): shorten requirement ids to pass spell check --- projects/audit-logging.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 214c5026c..14ab563b7 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -26,10 +26,10 @@ See [Appendix A: Guarantee of Delivery] for more details The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: -- REQ-CONV-01: Semantic conventions for application-level audit logs are defined -- REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined -- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK -- REQ-PIPE-01: OTel Collector instances must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-01: Semantic conventions for application-level audit logs are defined +- REQ-02: Semantic conventions for infrastructure-level audit logs are defined +- REQ-03: Guaranteed delivery of audit logs exported via OpenTelemetry SDK +- REQ-04: OTel Collector instances must provide guaranteed delivery of audit logs, including when its process is interrupted See [Appendix A: Guarantee of Delivery] for more details From 6bc9a5e1617d515da371aeb5b6a196876b588ccd Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit <2060790+mlenkeit@users.noreply.github.com> Date: Thu, 27 Mar 2025 12:59:17 +0100 Subject: [PATCH 18/21] docs(auditlogs): propose project lead Co-authored-by: Hilmar Falkenberg --- projects/audit-logging.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 14ab563b7..0539279aa 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -49,14 +49,14 @@ Other vendors are invited to join the discussion. ### Required staffing -* Project lead: SAP (name tbd) +* Project lead: @hilmarf * Sponsors: - @reyang - tbd * GC liaison: @svrnm -* Engineers for API/SDK: - * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) -* Engineers for OTel Collector: tbd +* Engineers contributing to the SIG: + - @hilmarf + - ... * Maintainers/approvers: tbd ## Timeline @@ -194,4 +194,4 @@ An event consists of the event name, event-specific data, and general metadata. [Appendix A: Guarantee of Delivery]: #appendix-a-guarantee-of-delivery -[Appendix B: Examples of Audit Log Events]: #appendix-b-examples-of-audit-log-events \ No newline at end of file +[Appendix B: Examples of Audit Log Events]: #appendix-b-examples-of-audit-log-events From 41329f3dba6f4d2dd14817ecec12733c047201a1 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit <2060790+mlenkeit@users.noreply.github.com> Date: Tue, 20 May 2025 12:21:04 +0100 Subject: [PATCH 19/21] * docs(auditlogs): describe phases approach --- projects/audit-logging.md | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 0539279aa..7c8500494 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -12,19 +12,25 @@ Examples of audit logs include: (see [Appendix B: Examples of Audit Log Events]) - accessing sensitive information - modification of data +We intend to tackle this in phases: +- In Phase 1 (_in progress_), we are **building an end-to-end prototype** to refine the challenges and requirements for audit logging in OTel and to showcase potential solutions. This is time-boxed until end of September 2025. We are set up to run this without a formal OTel project sign-off. +- In Phase 2, we intend to **contribute functional extensions upstream** back to OTel. We will work towards signing off this project proposal and either join existing SIGs or form a separate one. The results from Phase 1 should help us in the discussions with the maintainers to make our proposed extensions/changes more tangible. +- In Phase 3, we plan to **work on semantic conventions** for audit logging. + ### Current challenges OpenTelemetry does not have a good solution for audit logging - no semantic conventions for audit logs in OTel -- OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. -- OTel Collector instances may lose audit logs in transit (i.e. no guarantee of delivery) +- lack of delivery guarantees in OTel, e.g.: + - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. + - OTel Collector instances may lose audit logs in transit (i.e. no guarantee of delivery) See [Appendix A: Guarantee of Delivery] for more details ### Goals, objectives, and requirements -The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: +The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: (_to be refined by Phase 1_) - REQ-01: Semantic conventions for application-level audit logs are defined - REQ-02: Semantic conventions for infrastructure-level audit logs are defined @@ -35,10 +41,18 @@ See [Appendix A: Guarantee of Delivery] for more details ## Deliverables -- semantic convention for audit logs +Phase 1 (_in progress_, see repo [audit-log-poc-for-otel](https://github.com/apeirora/audit-log-poc-for-otel)): +- end-to-end prototype of a setup that produces audit logs uses OTel to deliver them to a sink. +- list of gaps in OTel with regard to delivering audit logs +- prototype implementations to close selected gaps + +Phase 2: - extend OTel APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) - extend OTel Collector for audit logging purposes (in collaboration with the respective SIG) +Phase 3: +- semantic convention for audit logs + ## Staffing / Help Wanted The following vendors are interested in improving this area: @@ -61,7 +75,9 @@ Other vendors are invited to join the discussion. ## Timeline -TBD based on community involvement. +- Phase 1: until end of September 2025 (approx.) +- Phase 2: tbd +- Phase 3: tbd ## Labels From 03d1de1d49bec649b889ebca5e463f1b27d5f6f1 Mon Sep 17 00:00:00 2001 From: Hilmar Falkenberg Date: Fri, 19 Sep 2025 16:40:25 +0200 Subject: [PATCH 20/21] Add files via upload --- .github/workflows/sync-fork.yaml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 .github/workflows/sync-fork.yaml diff --git a/.github/workflows/sync-fork.yaml b/.github/workflows/sync-fork.yaml new file mode 100644 index 000000000..de520f2bc --- /dev/null +++ b/.github/workflows/sync-fork.yaml @@ -0,0 +1,16 @@ +name: sync-fork +on: + schedule: + - cron: '4 2 * * 4,2' + workflow_dispatch: +jobs: + sync-fork: + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - run: gh repo sync $REPOSITORY -b $BRANCH_NAME + env: + GITHUB_TOKEN: ${{ secrets.SYNC_FORK_TOKEN }} + REPOSITORY: ${{ github.repository }} + BRANCH_NAME: ${{ github.ref_name }} From 2f9813f4e0e62f645bd1f62d45203f3442189a0e Mon Sep 17 00:00:00 2001 From: Hilmar Falkenberg Date: Fri, 19 Sep 2025 16:42:09 +0200 Subject: [PATCH 21/21] Revert "Add files via upload" This reverts commit 03d1de1d49bec649b889ebca5e463f1b27d5f6f1. --- .github/workflows/sync-fork.yaml | 16 ---------------- 1 file changed, 16 deletions(-) delete mode 100644 .github/workflows/sync-fork.yaml diff --git a/.github/workflows/sync-fork.yaml b/.github/workflows/sync-fork.yaml deleted file mode 100644 index de520f2bc..000000000 --- a/.github/workflows/sync-fork.yaml +++ /dev/null @@ -1,16 +0,0 @@ -name: sync-fork -on: - schedule: - - cron: '4 2 * * 4,2' - workflow_dispatch: -jobs: - sync-fork: - runs-on: ubuntu-latest - permissions: - contents: write - steps: - - run: gh repo sync $REPOSITORY -b $BRANCH_NAME - env: - GITHUB_TOKEN: ${{ secrets.SYNC_FORK_TOKEN }} - REPOSITORY: ${{ github.repository }} - BRANCH_NAME: ${{ github.ref_name }}