Lambda GitLab Loki

An AWS Lambda function that receives GitLab webhook payloads and forwards them to Grafana Cloud Loki for centralized logging and monitoring. This enables comprehensive observability of your GitLab CI/CD pipeline events in Grafana.

Features

• 🚀 Serverless GitLab webhook processing with AWS Lambda
• 📊 Direct integration with Grafana Cloud Loki
• 🔄 Automatic conversion of GitLab webhook payloads to Loki log format
• 🎯 Real-time pipeline monitoring and alerting capabilities
• 💰 Cost-effective pay-per-use model
• 🔒 Secure credential management with AWS environment variables

Prerequisites

• AWS Account with Lambda and API Gateway access
• Grafana Cloud account with Loki access
• GitLab project with webhook configuration permissions
• Python 3.9+ (for local development/testing)

Setup

1. Grafana Cloud Configuration

1. Get your Grafana Cloud credentials:
   • Log into your Grafana Cloud account
   • Navigate to "My Account" → "Security" → "API Keys"
   • Create a new API key with logs:write permissions
   • Note your Grafana Cloud Logs URL (typically https://logs-prod-xxx.grafana.net/loki/api/v1/push)
   • Note your Grafana Cloud username

2. AWS Lambda Deployment

Option A: AWS Console Deployment

1. Create the Lambda function:

   # Create a deployment package
   zip lambda-gitlab-loki.zip lambda_function.py

2. In AWS Console:

   • Go to AWS Lambda → Create function
   • Choose "Author from scratch"
   • Function name: gitlab-loki-webhook
   • Runtime: Python 3.9 or later
   • Upload the zip file created above

3. Set environment variables:

   • GRAFANA_CLOUD_USERNAME: Your Grafana Cloud username
   • GRAFANA_CLOUD_PASSWORD: Your Grafana Cloud API key
   • GRAFANA_CLOUD_LOGS_URL: Your Grafana Cloud Loki push URL

Option B: AWS CLI Deployment

# Create IAM role for Lambda
aws iam create-role --role-name lambda-gitlab-loki-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {"Service": "lambda.amazonaws.com"},
        "Action": "sts:AssumeRole"
      }
    ]
  }'

# Attach basic execution policy
aws iam attach-role-policy \
  --role-name lambda-gitlab-loki-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

# Create deployment package
zip lambda-gitlab-loki.zip lambda_function.py

# Create Lambda function
aws lambda create-function \
  --function-name gitlab-loki-webhook \
  --runtime python3.9 \
  --role arn:aws:iam::YOUR_ACCOUNT_ID:role/lambda-gitlab-loki-role \
  --handler lambda_function.lambda_handler \
  --zip-file fileb://lambda-gitlab-loki.zip

# Set environment variables
aws lambda update-function-configuration \
  --function-name gitlab-loki-webhook \
  --environment Variables='{
    "GRAFANA_CLOUD_USERNAME":"your-username",
    "GRAFANA_CLOUD_PASSWORD":"your-api-key",
    "GRAFANA_CLOUD_LOGS_URL":"https://logs-prod-xxx.grafana.net/loki/api/v1/push"
  }'

3. API Gateway Setup

Using AWS Console:

1. Create API Gateway:

   • Go to API Gateway → Create API
   • Choose "REST API" → Build
   • API name: gitlab-webhook-api

2. Create Resource and Method:

   • Actions → Create Resource
   • Resource name: webhook
   • Actions → Create Method → POST
   • Integration type: Lambda Function
   • Lambda Function: gitlab-loki-webhook
   • Enable Lambda Proxy integration: ✅

3. Deploy API:

   • Actions → Deploy API
   • Deployment stage: prod
   • Note the Invoke URL (e.g., https://abc123.execute-api.us-east-1.amazonaws.com/prod/webhook)

Using AWS CLI:

# Create API Gateway
API_ID=$(aws apigateway create-rest-api --name gitlab-webhook-api --query 'id' --output text)

# Get root resource ID
ROOT_ID=$(aws apigateway get-resources --rest-api-id $API_ID --query 'items[0].id' --output text)

# Create webhook resource
RESOURCE_ID=$(aws apigateway create-resource \
  --rest-api-id $API_ID \
  --parent-id $ROOT_ID \
  --path-part webhook \
  --query 'id' --output text)

# Create POST method
aws apigateway put-method \
  --rest-api-id $API_ID \
  --resource-id $RESOURCE_ID \
  --http-method POST \
  --authorization-type NONE

# Set up Lambda integration
aws apigateway put-integration \
  --rest-api-id $API_ID \
  --resource-id $RESOURCE_ID \
  --http-method POST \
  --type AWS_PROXY \
  --integration-http-method POST \
  --uri "arn:aws:apigateway:REGION:lambda:path/2015-03-31/functions/arn:aws:lambda:REGION:ACCOUNT_ID:function:gitlab-loki-webhook/invocations"

# Grant API Gateway permission to invoke Lambda
aws lambda add-permission \
  --function-name gitlab-loki-webhook \
  --statement-id apigateway-invoke \
  --action lambda:InvokeFunction \
  --principal apigateway.amazonaws.com

# Deploy API
aws apigateway create-deployment \
  --rest-api-id $API_ID \
  --stage-name prod

4. GitLab Webhook Configuration

1. In your GitLab project:

   • Go to Settings → Webhooks
   • URL: Your API Gateway endpoint (e.g., https://abc123.execute-api.us-east-1.amazonaws.com/prod/webhook)
   • Secret Token: (optional, but recommended for security)

2. Select trigger events:

   • ✅ Push events
   • ✅ Pipeline events
   • ✅ Job events
   • ✅ Merge request events
   • ✅ Release events
   • (Select others based on your monitoring needs)

3. SSL verification: ✅ Enable SSL verification

4. Test the webhook:

   • Click "Test" → "Push events"
   • Check AWS CloudWatch Logs for the Lambda function
   • Verify logs appear in Grafana Cloud Loki

Usage

Once configured, the system will automatically:

1. Receive GitLab webhooks at your API Gateway endpoint
2. Transform payloads into Loki-compatible log format
3. Forward logs to Grafana Cloud Loki with timestamp and job labels
4. Enable querying in Grafana with LogQL

Querying Logs in Grafana

Use LogQL queries to explore your GitLab events:

# All GitLab webhook events
{job="gitlab-webhook"}

# Pipeline events only
{job="gitlab-webhook"} | json | object_kind="pipeline"

# Failed pipelines
{job="gitlab-webhook"} | json | object_kind="pipeline" | status="failed"

# Specific project events
{job="gitlab-webhook"} | json | project_name="my-project"

# Recent push events
{job="gitlab-webhook"} | json | object_kind="push" | project_name=~".*"

Example Dashboards

Create Grafana dashboards to visualize:

• Pipeline success/failure rates over time
• Deployment frequency
• Build duration trends
• Push activity by branch
• Merge request metrics

Troubleshooting

Common Issues

1. "Webhook delivery failed"

   • Check API Gateway endpoint URL
   • Verify Lambda function is deployed and has correct permissions
   • Check CloudWatch Logs for Lambda errors

2. "Logs not appearing in Loki"

   • Verify Grafana Cloud credentials
   • Check Loki URL format and endpoint
   • Ensure API key has logs:write permissions

3. "Internal Server Error (500)"

   • Check Lambda environment variables are set correctly
   • Review CloudWatch Logs for detailed error messages
   • Verify payload format compatibility

Debugging

1. Enable detailed logging:

   # Update Lambda function with debug logging
   aws lambda update-function-configuration \
     --function-name gitlab-loki-webhook \
     --environment Variables='{
       "GRAFANA_CLOUD_USERNAME":"your-username",
       "GRAFANA_CLOUD_PASSWORD":"your-api-key", 
       "GRAFANA_CLOUD_LOGS_URL":"your-url",
       "DEBUG":"true"
     }'

2. Test locally:

   # Create test_payload.json with GitLab webhook sample
   # Run locally for debugging
   python3 -c "
   import json
   from lambda_function import lambda_handler
   
   with open('test_payload.json') as f:
       test_event = {'body': f.read()}
   
   result = lambda_handler(test_event, {})
   print(result)
   "

3. Monitor CloudWatch:

   • AWS Console → CloudWatch → Log groups → /aws/lambda/gitlab-loki-webhook
   • View real-time logs during webhook testing