Skip to content

docs: update Kafka MSK IAM authentication configuration #2294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kalavt wants to merge 3 commits into
base: master
Choose a base branch
from
Open

docs: update Kafka MSK IAM authentication configuration #2294

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bbd73d3
kafka: Add AWS MSK IAM auth support
kalavt Dec 12, 2025
a767991
docs: fix typo in Kafka output plugin description
kalavt Dec 12, 2025
21db1cd
Apply suggestions from code review
alexakreizinger Dec 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 56 additions & 36 deletions pipeline/inputs/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,73 +136,93 @@

## AWS MSK IAM authentication

Fluent Bit v4.0.4 and later supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This lets you securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
Starting with version 4.0.4, Fluent Bit supports AWS IAM authentication for Amazon MSK clusters. This allows you to use your AWS credentials and IAM policies to control access to Kafka topics.

Check warning on line 139 in pipeline/inputs/kafka.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.WordList] Use 'lets you' instead of 'allows you to'.

Raw Output:
{"message": "[FluentBit.WordList] Use 'lets you' instead of 'allows you to'.", "location": {"path": "pipeline/inputs/kafka.md", "range": {"start": {"line": 139, "column": 103}}}, "severity": "INFO"}

### Build requirements
### Prerequisites

If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
- Access to an AWS MSK cluster with IAM authentication enabled
- Valid AWS credentials (IAM role, access keys, or instance profile)
- Network connectivity to your MSK brokers

- The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
### Configuration parameters [#config-aws]

### Runtime requirements
| Property | Description | Default |
| -------- | ----------- | ------- |
| `rdkafka.sasl.mechanism` | Set to `aws_msk_iam` to enable MSK IAM authentication | _none_ |
| `aws_region` | AWS region (optional, automatically detected from broker hostname for standard MSK endpoints) | auto-detected |

- **Network Access:** Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
- **AWS Credentials:** Provide these AWS credentials using any supported AWS method. These credentials are discovered by default when `aws_msk_iam` flag is enabled.
- IAM roles (recommended for EC2, ECS, or EKS)
- Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
- AWS credentials file (`~/.aws/credentials`)
- Instance metadata service (IMDS)
- **IAM Permissions:** The credentials must allow access to the target MSK cluster, as shown in the following example policy.
### Basic configuration

### Configuration parameters [#config-aws]
For most use cases, simply set `rdkafka.sasl.mechanism` to `aws_msk_iam`:

Check warning on line 156 in pipeline/inputs/kafka.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "simply" that imply ease of use, because the user may find this action difficult.

Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like \"simply\" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/inputs/kafka.md", "range": {"start": {"line": 156, "column": 21}}}, "severity": "WARNING"}

```yaml
pipeline:
inputs:
- name: kafka
brokers: boot-abc123.c1.kafka-serverless.us-east-1.amazonaws.com:9098
topics: my-topic
rdkafka.sasl.mechanism: aws_msk_iam
```
| Property | Description | Required |
| -------- | ----------- | -------- |
| `aws_msk_iam` | If `true`, enables AWS MSK IAM authentication. Possible values: `true`, `false`. | `false` |
| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction. This value is required if `aws_msk_iam` is `true`. | _none_ |
The AWS region is automatically detected from the broker hostname for standard MSK endpoints.
### Configuration example
**Note:** When using `aws_msk_iam`, Fluent Bit automatically sets `rdkafka.security.protocol` to `SASL_SSL`. You don't need to configure it manually.

### Using custom DNS or PrivateLink

If you're using custom DNS names or PrivateLink aliases, specify the `aws_region` parameter:

```yaml
pipeline:
inputs:
- name: kafka
brokers: my-cluster.abcdef.c1.kafka.us-east-1.amazonaws.com:9098
brokers: my-kafka-endpoint.example.com:9098
topics: my-topic
aws_msk_iam: true
aws_msk_iam_cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcdef-1234-5678-9012-abcdefghijkl-s3

outputs:
- name: stdout
match: '*'
rdkafka.sasl.mechanism: aws_msk_iam
aws_region: us-east-1
```

### Example AWS IAM policy
### AWS credentials

{% hint style="info" %}
Fluent Bit uses the standard AWS credentials chain to authenticate:

IAM policies and permissions can be complex and might vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, consult your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
2. AWS credentials file (`~/.aws/credentials`)
3. IAM instance profile (recommended for EC2)
4. IAM task role (recommended for ECS)
5. IAM service account (recommended for EKS)

{% endhint %}
### Required IAM permissions

The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
Your AWS credentials need the following permissions to consume from MSK topics:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"kafka-cluster:*",
"kafka-cluster:DescribeCluster",
"kafka-cluster:ReadData",
"kafka-cluster:Connect",
"kafka-cluster:DescribeTopic",
"kafka-cluster:Connect"
"kafka-cluster:ReadData",
"kafka-cluster:DescribeGroup",
"kafka-cluster:AlterGroup"
],
"Resource": "*"
"Resource": [
"arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/CLUSTER_UUID",
"arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/CLUSTER_UUID/my-topic",
"arn:aws:kafka:REGION:ACCOUNT:group/CLUSTER_NAME/CLUSTER_UUID/fluent-bit"
]
}
]
}
```

Replace `REGION`, `ACCOUNT`, `CLUSTER_NAME`, `CLUSTER_UUID`, and topic/group names with your actual values.

**Note:** The `CLUSTER_UUID` segment is required in all topic and group ARNs. You can find your cluster's UUID in the MSK console or by describing the cluster with the AWS CLI.

Check warning on line 224 in pipeline/inputs/kafka.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'ARNs'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'ARNs'?", "location": {"path": "pipeline/inputs/kafka.md", "range": {"start": {"line": 224, "column": 73}}}, "severity": "INFO"}

{% hint style="info" %}
For detailed IAM policy configuration, consult your AWS administrator or refer to the [AWS MSK documentation](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html).
{% endhint %}
144 changes: 101 additions & 43 deletions pipeline/outputs/kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The _Kafka Producer_ output plugin lets you ingest your records into an [Apache Kafka](https://kafka.apache.org/) service. This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka).

In Fluent Bit 4.0.4 and later, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.
In Fluent Bit 4.0.4 and later, the Kafka output plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.

## Configuration parameters

Expand Down Expand Up @@ -237,83 +237,141 @@

## AWS MSK IAM authentication

Fluent Bit 4.0.4 and later supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM for the Kafka output plugin. This lets you securely send data to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.
Starting with version 4.0.4, Fluent Bit supports AWS IAM authentication for Amazon MSK clusters. This allows you to use your AWS credentials and IAM policies to control access to Kafka topics.

Check warning on line 240 in pipeline/outputs/kafka.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.WordList] Use 'lets you' instead of 'allows you to'.

Raw Output:
{"message": "[FluentBit.WordList] Use 'lets you' instead of 'allows you to'.", "location": {"path": "pipeline/outputs/kafka.md", "range": {"start": {"line": 240, "column": 103}}}, "severity": "INFO"}

### Prerequisites

If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:
- Access to an AWS MSK cluster with IAM authentication enabled
- Valid AWS credentials (IAM role, access keys, or instance profile)
- Network connectivity to your MSK brokers

- Build Requirements
### Configuration parameters

The packages `libsasl2` and `libsasl2-dev` must be installed on your build environment.
| Property | Description | Default |
| -------- | ----------- | ------- |
| `rdkafka.sasl.mechanism` | Set to `aws_msk_iam` to enable MSK IAM authentication | _none_ |
| `aws_region` | AWS region (optional, automatically detected from broker hostname for standard MSK endpoints) | auto-detected |

- Runtime Requirements:
### Basic configuration

- Network Access: Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
- AWS Credentials: Provide credentials using any supported AWS method:
- IAM roles (recommended for EC2, ECS, or EKS)
- Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
- AWS credentials file (`~/.aws/credentials`)
- Instance metadata service (IMDS)
For most use cases, simply set `rdkafka.sasl.mechanism` to `aws_msk_iam`:

Check warning on line 257 in pipeline/outputs/kafka.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Simplicity] Avoid words like "simply" that imply ease of use, because the user may find this action difficult.

Raw Output:
{"message": "[FluentBit.Simplicity] Avoid words like \"simply\" that imply ease of use, because the user may find this action difficult.", "location": {"path": "pipeline/outputs/kafka.md", "range": {"start": {"line": 257, "column": 21}}}, "severity": "WARNING"}

These credentials are discovered by default when `aws_msk_iam` flag is enabled.
{% tabs %}
{% tab title="fluent-bit.yaml" %}

- IAM Permissions: The credentials must allow access to the target MSK cluster.
```yaml
pipeline:
inputs:
- name: cpu

### AWS MSK IAM configuration parameters
outputs:
- name: kafka
match: '*'
brokers: b-1.mycluster.kafka.us-east-1.amazonaws.com:9098
topics: my-topic
rdkafka.sasl.mechanism: aws_msk_iam
```
This plugin supports the following parameters:
{% endtab %}
{% tab title="fluent-bit.conf" %}
| Property | Description | Type | Default |
|---------------------------|-----------------------------------------------------|---------|-------------------------------|
| `aws_msk_iam` | Optional. Enable AWS MSK IAM authentication. | Boolean | `false` |
| `aws_msk_iam_cluster_arn` | Full ARN of the MSK cluster for region extraction. Required if `aws_msk_iam` is set. | String | _none_ |
```text
[INPUT]
Name cpu

### Configuration example
[OUTPUT]
Name kafka
Match *
Brokers b-1.mycluster.kafka.us-east-1.amazonaws.com:9098
Topics my-topic
rdkafka.sasl.mechanism aws_msk_iam
```

{% endtab %}
{% endtabs %}

The AWS region is automatically detected from the broker hostname for standard MSK endpoints.

**Note:** When using `aws_msk_iam`, Fluent Bit automatically sets `rdkafka.security.protocol` to `SASL_SSL`. You don't need to configure it manually.

### Using custom DNS or PrivateLink

If you're using custom DNS names or PrivateLink aliases, specify the `aws_region` parameter:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: random
- name: cpu

outputs:
- name: kafka
match: '*'
brokers: my-cluster.abcdef.c1.kafka.us-east-1.amazonaws.com:9098
brokers: my-kafka-endpoint.example.com:9098
topics: my-topic
aws_msk_iam: true
aws_msk_iam_cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcdef-1234-5678-9012-abcdefghijkl-s3
rdkafka.sasl.mechanism: aws_msk_iam
aws_region: us-east-1
```
{% endtab %}
{% tab title="fluent-bit.conf" %}
```text
[INPUT]
Name cpu

[OUTPUT]
Name kafka
Match *
Brokers my-kafka-endpoint.example.com:9098
Topics my-topic
rdkafka.sasl.mechanism aws_msk_iam
aws_region us-east-1
```

{% endtab %}
{% endtabs %}

### AWS IAM policy
### AWS credentials

IAM policies and permissions can be complex and can vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, consult with your AWS administrator or an AWS expert who is familiar with MSK and IAM security.
Fluent Bit uses the standard AWS credentials chain to authenticate:

The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:
1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
2. AWS credentials file (`~/.aws/credentials`)
3. IAM instance profile (recommended for EC2)
4. IAM task role (recommended for ECS)
5. IAM service account (recommended for EKS)

### Required IAM permissions

Your AWS credentials need the following permissions to produce to MSK topics:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"kafka-cluster:*",
"kafka-cluster:DescribeCluster",
"kafka-cluster:ReadData",
"kafka-cluster:DescribeTopic",
"kafka-cluster:Connect"
],
"Resource": "*"
}
]
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kafka-cluster:Connect",
"kafka-cluster:DescribeTopic",
"kafka-cluster:WriteData"
],
"Resource": [
"arn:aws:kafka:REGION:ACCOUNT:cluster/CLUSTER_NAME/CLUSTER_UUID",
"arn:aws:kafka:REGION:ACCOUNT:topic/CLUSTER_NAME/CLUSTER_UUID/my-topic"
]
}
]
}
```

Replace `REGION`, `ACCOUNT`, `CLUSTER_NAME`, `CLUSTER_UUID`, and topic name with your actual values.

**Note:** The `CLUSTER_UUID` segment is required in all topic ARNs. You can find your cluster's UUID in the MSK console or by describing the cluster with the AWS CLI.

Check warning on line 373 in pipeline/outputs/kafka.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'ARNs'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'ARNs'?", "location": {"path": "pipeline/outputs/kafka.md", "range": {"start": {"line": 373, "column": 63}}}, "severity": "INFO"}

{% hint style="info" %}
For detailed IAM policy configuration, consult your AWS administrator or refer to the [AWS MSK documentation](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html).
{% endhint %}