Workflow CLI Getting Started

Workflow CLI Getting Started

This guide walks you through your first OpenSearch migration using the Workflow CLI. You'll learn how to configure, submit, and monitor a migration workflow.

Prerequisites

Before starting, ensure:

• You have access to the migration console on your Kubernetes cluster
• Argo Workflows is installed and running
• Source and target clusters are accessible
• You've reviewed the Workflow CLI Overview

Step 1: Access the Migration Console

The Migration Console is a Kubernetes pod where you run the Workflow CLI. First, ensure kubectl is configured for your cluster.

For EKS deployments, configure kubectl:

aws eks update-kubeconfig --region <REGION> --name migration-eks-cluster-<STAGE>-<REGION>

Connect to the Migration Console pod:

kubectl exec -it migration-console-0 -n ma -- /bin/bash

You should see a welcome message similar to:

Welcome to the Migration Assistant Console

To get started, run:
  console --help

For workflow commands, run:
  workflow --help

Step 2: Edit the configuration

Customize the configuration for your clusters:

workflow configure edit

This opens your default editor. Update these sections:

Source cluster:

sourceClusters:
  my-es-source:
    endpoint: https://source.example.com:9200
    version: "7.10.2"
    authConfig:
      basic:
        username: admin
        password: your-password

Target cluster:

targetClusters:
  my-os-target:
    endpoint: https://target.example.com:9200
    version: "2.11.0"
    authConfig:
      basic:
        username: admin
        password: your-password

Migration configuration:

migrations:
  - sourceCluster: my-es-source
    targetCluster: my-os-target
    snapshotMigrations:
      - indices: ["logs-*", "metrics-*"]
        metadataMigration:
          enabled: true
        documentBackfill:
          enabled: true

Save and exit (in vi: Esc, then :wq, then Enter).

Step 3: Verify cluster connectivity

Before submitting the workflow, verify that the source and target clusters are reachable:

console clusters connection-check

You should see successful connection status for both clusters. If there are connectivity issues, check your cluster endpoints and credentials in the configuration.

Step 4: Submit the workflow

Submit your migration workflow:

workflow submit

You'll see output like:

Initializing workflow from session: default
Workflow initialized with prefix: a1b2c3d4-e5f6-7890-abcd-ef1234567890
Submitting workflow to namespace: ma
Workflow submitted successfully
  Name: migration-xwz9r
  Prefix: a1b2c3d4-e5f6-7890-abcd-ef1234567890
  Namespace: ma

Important: Note the workflow name for monitoring.

Step 5: Monitor workflow progress

Check the status of your workflow:

workflow status

You'll see a tree view showing progress:

[*] Workflow: migration-xwz9r
  Phase: Running
  Started: 2024-01-15T10:30:00Z

📋 Workflow Steps
├── ✓ Initialize (Succeeded)
├── ▶ Create Snapshot (Running)
│   ├── ✓ Validate Indexes (Succeeded)
│   └── ▶ Snapshot Creation (Running)
├── ○ Register Snapshot (Pending)
├── ○ Migrate Metadata (Pending)
└── ○ Restore Documents (Pending)

Status symbols:

Symbol	Meaning
✓	Step completed successfully
▶	Step currently running
○	Step pending (waiting to run)
✗	Step failed
⟳	Step waiting for approval

Step 6: Wait for completion

The workflow runs asynchronously. You can monitor it in several ways:

Option A: Check manually

workflow status

Option B: Auto-refresh with watch

watch -n 10 workflow status

(Refreshes every 10 seconds. Press Ctrl+C to exit.)

Option C: Submit with wait flag

workflow submit --wait --timeout 3600

Step 7: Handle approvals (if required)

If the workflow requires manual approval, you'll see:

├── ⟳ Manual Approval Gate - WAITING FOR APPROVAL

Approve to continue:

workflow approve

Then check status again:

workflow status

Step 8: Verify success

When complete, you'll see:

[+] Workflow: migration-xwz9r
  Phase: Succeeded
  Started: 2024-01-15T10:30:00Z
  Finished: 2024-01-15T11:45:00Z

📋 Workflow Steps
├── ✓ Initialize (Succeeded)
├── ✓ Create Snapshot (Succeeded)
├── ✓ Register Snapshot (Succeeded)
├── ✓ Migrate Metadata (Succeeded)
└── ✓ Restore Documents (Succeeded)

Verify data in your target cluster to confirm the migration succeeded.

Common scenarios

Migrate specific indexes only

migrations:
  - sourceCluster: my-source
    targetCluster: my-target
    snapshotMigrations:
      - indices: ["user-data-*", "application-logs-2024-*"]
        metadataMigration:
          enabled: true
        documentBackfill:
          enabled: true

Metadata only (no documents)

migrations:
  - sourceCluster: my-source
    targetCluster: my-target
    snapshotMigrations:
      - indices: ["*"]
        metadataMigration:
          enabled: true
        documentBackfill:
          enabled: false

Multiple index groups

migrations:
  - sourceCluster: my-source
    targetCluster: my-target
    snapshotMigrations:
      # Critical data with metadata
      - indices: ["orders-*", "customers-*"]
        metadataMigration:
          enabled: true
        documentBackfill:
          enabled: true
      
      # Logs without metadata
      - indices: ["logs-*"]
        metadataMigration:
          enabled: false
        documentBackfill:
          enabled: true

Troubleshooting

Configuration won't save

Problem: Editor closes but configuration isn't saved.

Solution:

• Save the file before exiting (vi: Esc → :wq → Enter)
• Check for YAML syntax errors

Workflow fails to submit

Problem: Error: "No workflow configuration found"

Solution:

# Check if configuration exists
workflow configure view

# If empty, create one
workflow configure edit

Workflow stuck in pending

Problem: All steps show "Pending"

Solution:

1. Check workflow templates are deployed:

   kubectl get workflowtemplates -n ma

2. Check pod status:

   kubectl get pods -n ma

3. View workflow logs:

   workflow output

Quick reference

# Configuration commands
workflow configure sample --load    # Start with sample
workflow configure edit            # Edit configuration
workflow configure view            # Show configuration
workflow configure clear           # Clear configuration

# Execution commands
workflow submit                    # Submit workflow
workflow submit --wait             # Submit and wait for completion
workflow status                    # Show all workflows
workflow status                    # Show specific workflow
workflow output                    # View workflow logs

# Management commands
workflow approve                   # Approve manual steps
workflow stop                      # Stop a running workflow

Next steps

Now that you've completed your first migration:

1. Verify data in your target cluster
2. Learn about Backfill Workflow for more details on document migration
3. Review the Workflow CLI Overview for architectural details
4. Check Troubleshooting if you encounter issues

Getting help

If you encounter issues:

1. Check command help:

   workflow --help
   workflow configure --help
   workflow submit --help

2. Enable verbose logging:

   workflow -vv status

3. Check workflow logs:

   workflow output

4. Provide feedback through the OpenSearch Migrations repository