You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: data/docs/logs-management/guides/pii-scrubbing.mdx
+40-15Lines changed: 40 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,9 @@
1
1
---
2
-
date: 2024-07-05
2
+
date: 2025-11-12
3
3
id: pii-scrubbing
4
-
tags: [SigNoz Cloud]
4
+
tags: [SigNoz Cloud, Self-Host]
5
5
title: Guide to perform PII Scrubbing using SigNoz
6
+
description: Learn how to scrub PII (Personally Identifiable Information) from logs using OpenTelemetry Collector. This guide shows you how to remove sensitive data like emails, SSNs, and credit card numbers from application logs before they reach SigNoz to ensure GDPR and CCPA compliance.
@@ -12,29 +13,43 @@ import GetHelp from '@/components/shared/get-help.md'
12
13
PII scrubbing removes sensitive personal information from data to protect privacy and comply with regulations like GDPR and CCPA. SigNoz simplifies this process for your applications using OpenTelemetry. This guide explains how to implement PII scrubbing in your data pipeline before sending information to SigNoz.
13
14
14
15
## Prerequisite
16
+
15
17
-[SigNoz Cloud](https://signoz.io/teams/) account
16
-
-Logs in log file
18
+
-Application sending logs to OpenTelemetry Collector (via OTLP or any other receivers)
17
19
18
20
## Send application logs to SigNoz
19
21
20
-
You need to configure OpenTelemetry Collector to send your logs in log files to SigNoz, checkout [this documentation](https://signoz.io/docs/userguide/collect_logs_from_file/)
21
-
for detailed instructions on how to do this.
22
+
You can send logs to SigNoz via OpenTelemetry Collector using various methods:
23
+
24
+
-**OTLP:** Direct instrumentation from your application
25
+
-**File Logs:** Using filelog receiver, checkout [this documentation](https://signoz.io/docs/userguide/collect_logs_from_file/)
26
+
- Other OpenTelemetry receivers and protocols
27
+
28
+
For more detailed guides on setting up log collection for your specific use case, refer to the [Send Logs to SigNoz documentation](https://signoz.io/docs/logs-management/send-logs-to-signoz/).
22
29
23
30
## Process of Scrubbing PII Data
24
31
25
32
OpenTelemetry Collector offers a powerful [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md) to filter and modify sensitive data before it reaches SigNoz. We'll use this processor to perform PII scrubbing on our logs.
26
33
27
-
### Sample log
34
+
### Sample log
28
35
29
36
Let's use the following sample log line to demonstrate the PII scrubbing process:
30
37
31
38
```json
32
-
{"timestamp": "2024-07-05T12:34:56Z", "severity": "INFO", "message": "User [email protected] (SSN: 123-45-6789) made a purchase with credit card 4111-1111-1111-1111", "user_id": "12345", "purchase_amount": 99.99}
39
+
{
40
+
"timestamp": "2024-07-05T12:34:56Z",
41
+
"severity": "INFO",
42
+
"message": "User [email protected] (SSN: 123-45-6789) made a purchase with credit card 4111-1111-1111-1111",
43
+
"user_id": "12345",
44
+
"purchase_amount": 99.99
45
+
}
33
46
```
34
47
35
48
This log contains several pieces of PII that we want to scrub: an email address, a social security number (SSN), and a credit card number inside the log body.
49
+
36
50
### Configuring the Transform Processor
37
-
To scrub this PII, we'll use the transform processor in our OpenTelemetry Collector configuration. Trasform processor makes use of [Regular Expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions) for scrubbing your sensitive data. Here's how we can set it up:
51
+
52
+
To scrub this PII, we'll use the transform processor in our OpenTelemetry Collector configuration. Transform processor makes use of [Regular Expressions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions) for scrubbing your sensitive data. Here's how we can set it up:
38
53
39
54
```yaml
40
55
processors:
@@ -58,7 +73,9 @@ Let's break down what each statement does:
58
73
- The third statement identifies credit card number patterns and replaces all digits with asterisks.
59
74
60
75
### Applying the Processor
76
+
61
77
To apply this processor to your logs pipeline, add it to the processors list in your pipeline configuration:
78
+
62
79
```yaml
63
80
service:
64
81
pipelines:
@@ -68,9 +85,12 @@ service:
68
85
processors: [batch, transform]
69
86
exporters: [otlp]
70
87
```
88
+
71
89
### Result
90
+
72
91
After applying this configuration and restarting your OpenTelemetry Collector, the sample log line would be transformed to:
73
-
```yaml
92
+
93
+
```json
74
94
{
75
95
"timestamp": "2024-07-05T12:34:56Z",
76
96
"severity": "INFO",
@@ -79,17 +99,22 @@ After applying this configuration and restarting your OpenTelemetry Collector, t
79
99
"purchase_amount": 99.99
80
100
}
81
101
```
82
-
<figuredata-zoomablealign='center'>
83
-
<imgsrc="/img/docs/logs-management/guides/sample-log-scrubbed-output.webp"alt="Sample log output with PII data scrubbed out"/>
84
-
<figcaption><i>Sample log output with PII data scrubbed out</i></figcaption>
alt="Sample log output with PII data scrubbed out"
107
+
/>
108
+
<figcaption>
109
+
<i>Sample log output with PII data scrubbed out</i>
110
+
</figcaption>
85
111
</figure>
86
112
87
113
As you can see, the email address, SSN, and credit card number have been scrubbed, protecting this sensitive information before it reaches SigNoz.
88
114
Remember to adjust the regular expressions in the `replace_pattern` statements as needed to match the specific formats of PII in your logs. You can add more statements to handle additional types of sensitive data that may appear in your logs.
89
115
90
-
91
-
To know more about the capabilities of trasform processsor, checkout [this documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md).
116
+
To know more about the capabilities of transform processor, checkout [this documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md).
0 commit comments