RDSC-5587 Update CLI public docs and changelog

content/integrate/redis-data-integration/data-pipelines/deploy.md

@@ -22,9 +22,9 @@ The sections below explain how to deploy a pipeline after you have created the r
 ## Set secrets
 
 Before you deploy your pipeline, you must set the authentication secrets for the
-source and target databases. Each secret has a name that you can pass to the
+source and target databases. Each secret has a name that you pass to the
 [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
-command (VM deployment) or the `rdi-secret.sh` script (K8s deployment) to set the secret value. 
+command to set the secret value.
 You can then refer to these secrets in the `config.yaml` file using the syntax "`${SECRET_NAME}`"
 (the sample
 [config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#example" >}})
@@ -53,10 +53,10 @@ secrets are only relevant for TLS/mTLS connections.
 {{< embed-md "rdi-tls-secrets.md" >}}
 {{< /note >}}
 
-### Set secrets for VM deployment
+### Set secrets with the CLI
 
 Use [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
-to set secrets for a VM deployment. 
+to set secrets for any installation type (VM, Kubernetes, or Redis Cloud).
 
 The specific command lines for source secrets are as follows:
 
@@ -92,62 +92,42 @@ redis-di set-secret TARGET_DB_KEY /path/to/myclient.key
 redis-di set-secret TARGET_DB_KEY_PASSWORD yourKeyPassword
 ```
 
-### Set secrets for K8s/Helm deployment using the rdi-secret.sh script
+By default, `set-secret` waits for the pipeline to apply the change before returning. When you set
+several secrets at once, set all but the last one with `--wait=false` to avoid a timeout while the
+pipeline is only partially updated. See [Wait for changes to complete](#wait) below for details.
 
-Use the `rdi-secret.sh` script to set secrets for a K8s/Helm deployment. To use this script, unzip the archive that contains the RDI Helm chart and navigate to the resulting folder. The `rdi-secret.sh` script is located in the `scripts` subfolder. The general pattern for using this script is:
+### Manage secrets with the CLI
 
-```bash
-scripts/rdi-secret.sh set <SECRET-NAME> <SECRET-VALUE>
-```
-
-The script also lets you retrieve a specific secret or list all the secrets that have been set:
+Along with `set-secret`, the CLI has commands to list, inspect, and delete secrets. Because the API
+never returns secret values, these commands show only the secret keys and whether they are set, not
+the stored values.
 
 ```bash
-# Get specific secret
-scripts/rdi-secret.sh get <SECRET-NAME>
-
-# List all secrets
-scripts/rdi-secret.sh list
-```
+# List all the secrets of a pipeline and whether each one is set
+redis-di list-secrets
 
-The specific command lines for source secrets are as follows:
+# Show a single secret and whether it is set
+redis-di describe-secret SOURCE_DB_PASSWORD
 
-```bash
-# For username and password
-scripts/rdi-secret.sh set SOURCE_DB_USERNAME yourUsername
-scripts/rdi-secret.sh set SOURCE_DB_PASSWORD yourPassword
-
-# With source TLS, in addition to the above
-scripts/rdi-secret.sh set SOURCE_DB_CACERT /path/to/myca.crt
-
-# With source mTLS, in addition to the above
-scripts/rdi-secret.sh set SOURCE_DB_CERT /path/to/myclient.crt
-scripts/rdi-secret.sh set SOURCE_DB_KEY /path/to/myclient.key
-# Use this only if SOURCE_DB_KEY is password-protected
-scripts/rdi-secret.sh set SOURCE_DB_KEY_PASSWORD yourKeyPassword 
+# Delete a secret (prompts for confirmation unless you add --force)
+redis-di delete-secret SOURCE_DB_CACERT
 ```
 
-The corresponding command lines for target secrets are:
-
-```bash
-# For username and password
-scripts/rdi-secret.sh set TARGET_DB_USERNAME yourUsername
-scripts/rdi-secret.sh set TARGET_DB_PASSWORD yourPassword
-
-# With target TLS, in addition to the above
-scripts/rdi-secret.sh set TARGET_DB_CACERT /path/to/myca.crt
-
-# With target mTLS, in addition to the above
-scripts/rdi-secret.sh set TARGET_DB_CERT /path/to/myclient.crt
-scripts/rdi-secret.sh set TARGET_DB_KEY /path/to/myclient.key
-# Use this only if TARGET_DB_KEY is password-protected
-scripts/rdi-secret.sh set TARGET_DB_KEY_PASSWORD yourKeyPassword
-```
+See the reference pages for
+[`list-secrets`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-secrets" >}}),
+[`get-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-get-secret" >}}),
+[`describe-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe-secret" >}}),
+and [`delete-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete-secret" >}})
+for the full list of options.
 
 ### Set secrets for K8s/Helm deployment using Kubectl command
 
-In some scenarios, you may prefer to use [`kubectl create secret generic`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_generic/)
-to set secrets for a K8s/Helm deployment. The general pattern of the commands is:
+{{< note >}}It is strongly recommended to manage secrets with the `redis-di` CLI rather than with
+`kubectl` directly. The CLI applies the correct labels automatically, validates the secret keys, and
+works the same way across all installation types.{{< /note >}}
+
+For a Kubernetes/Helm deployment, you can also use [`kubectl create secret generic`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_generic/)
+to set secrets instead of the CLI. The general pattern of the commands is:
 
 ```bash
 kubectl create secret generic <DB> \
@@ -247,16 +227,158 @@ kubectl create secret generic target-db-ssl --namespace=rdi \
 
 Note that the certificate paths contained in the secrets `SOURCE_DB_CACERT`, `SOURCE_DB_CERT`, and `SOURCE_DB_KEY` (for the source database) and `TARGET_DB_CACERT`, `TARGET_DB_CERT`, and `TARGET_DB_KEY` (for the target database) are internal to RDI, so you *must* use the values shown in the example above. You should only change the certificate paths when you create the `source-db-ssl` and `target-db-ssl` secrets.
 
+Secrets that you create directly with `kubectl` must also be labeled so that the RDI operator
+discovers them as pipeline secrets. Each secret needs the following labels, where the
+`app.kubernetes.io/instance` label is the pipeline name (`default` for the default pipeline):
+
+| Label | Value |
+| :-- | :-- |
+| `app.kubernetes.io/name` | `pipeline` |
+| `app.kubernetes.io/instance` | `default` |
+| `product` | `rdi` |
+
+Apply the labels to each secret with [`kubectl label`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_label/):
+
+```bash
+kubectl label secret source-db --namespace=rdi --overwrite \
+  app.kubernetes.io/name=pipeline \
+  app.kubernetes.io/instance=default \
+  product=rdi
+kubectl label secret target-db --namespace=rdi --overwrite \
+  app.kubernetes.io/name=pipeline \
+  app.kubernetes.io/instance=default \
+  product=rdi
+
+# With source TLS or mTLS
+kubectl label secret source-db-ssl --namespace=rdi --overwrite \
+  app.kubernetes.io/name=pipeline \
+  app.kubernetes.io/instance=default \
+  product=rdi
+
+# With target TLS or mTLS
+kubectl label secret target-db-ssl --namespace=rdi --overwrite \
+  app.kubernetes.io/name=pipeline \
+  app.kubernetes.io/instance=default \
+  product=rdi
+```
+
 ## Deploy a pipeline
 
 When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}), you are
-ready to deploy. Use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
-to configure and deploy pipelines for both VM and K8s installations.
-
-For VM installations, you can also use the
+ready to deploy. Use the
 [`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}})
 command to deploy a pipeline:
 
 ```bash
 redis-di deploy --dir <path to pipeline folder>
-```
+```
+
+RDI first validates the configuration and then deploys it if it is correct. You can control the
+validation and what happens after deployment with the following options:
+
+- `--dry-run`: Validate the configuration without deploying it. Off by default.
+- `--validate-tables`: Validate the configuration against the source and target databases, for
+  example that the tables it references exist. On by default; pass `--validate-tables=false` to skip
+  this check, which is useful when the databases are not reachable at deploy time.
+- `--validate-cdc`: Additionally validate that the source database is correctly configured for
+  [change data capture (CDC)]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}).
+  Off by default; enable it with `--validate-cdc`.
+- `--start`: Start the pipeline as soon as it is deployed. On by default; pass `--start=false` to
+  deploy the pipeline without starting it, then start it later with
+  [`redis-di start`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-start" >}}).
+
+See the [`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}})
+reference page for the full list of options.
+
+You can also use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
+to configure and deploy pipelines for both VM and K8s installations.
+
+## Display the pipeline status
+
+Once a pipeline is deployed, use the
+[`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}})
+command (also available as `redis-di status`) to display its status. This combines the pipeline
+configuration with its runtime status, showing its overall state, its sources and targets, its jobs
+and components, and its per-stream statistics and performance metrics.
+
+```bash
+redis-di describe
+```
+
+To watch the status update live, pair the command with `watch`:
+
+```bash
+watch -n 1 redis-di describe
+```
+
+For a shorter overview, [`redis-di list`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list" >}})
+prints a one-line summary of the pipeline, and
+[`redis-di get`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-get" >}})
+does the same for a single pipeline. See the
+[`redis-di describe`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-describe" >}})
+reference page for details.
+
+## Start and stop a pipeline
+
+Use [`redis-di stop`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-stop" >}})
+to pause a running pipeline and
+[`redis-di start`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-start" >}})
+to resume it. Stopping a pipeline halts data processing without deleting the pipeline or its
+configuration, so you can start it again later from where it left off.
+
+```bash
+redis-di stop
+redis-di start
+```
+
+## Reset a pipeline
+
+Use [`redis-di reset`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-reset" >}})
+to return a pipeline to initial full-sync mode. This reloads a fresh
+[snapshot]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) of the source
+data and then resumes change data capture (CDC), which is useful when the source and target have
+drifted out of sync.
+
+```bash
+redis-di reset
+```
+
+## Undeploy a pipeline
+
+To remove a pipeline, use the
+[`redis-di delete`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-delete" >}})
+command. This stops the pipeline and deletes it, along with its configuration and status, from RDI.
+The secrets you set for the pipeline are not affected.
+
+```bash
+redis-di delete <pipeline>
+```
+
+Because deleting a pipeline is destructive, the command asks for confirmation unless you add the
+`--force` option. If you omit the pipeline name, the `default` pipeline is deleted.
+
+## Wait for changes to complete {#wait}
+
+The commands that change a pipeline's state, namely `deploy`, `delete`, `start`, `stop`, `reset`,
+`set-secret`, and `delete-secret`, do not return as soon as the API accepts the request. By default,
+they wait for the pipeline to finish transitioning to the expected state, polling its status until it
+succeeds, reaches an error, or the `--timeout` (2 minutes by default) elapses. This is usually what
+you want: the command reflects the real outcome, so a script can rely on the change having taken
+effect and can fail fast if it did not.
+
+In some cases, though, a pipeline needs *several* changes before it can transition to a healthy state,
+and waiting after each individual change would time out. The clearest example is rotating both the
+username and the password of a database: if you set only the username with the default `--wait=true`,
+the pipeline tries to reconnect with the new username and the old password, fails, and the command
+times out after two minutes with the pipeline in a broken state.
+
+To avoid this, set all the related secrets, or at least all of them except the last, with
+`--wait=false`, so the pipeline applies them together and only the final command waits for it to
+become healthy:
+
+```bash
+redis-di set-secret SOURCE_DB_USERNAME newUsername --wait=false
+redis-di set-secret SOURCE_DB_PASSWORD newPassword
+```
+
+The same applies to any set of changes that are only valid together.

content/integrate/redis-data-integration/data-pipelines/rejected-records.md

@@ -108,8 +108,10 @@ for Redis Cloud, or use the appropriate self-managed RDI reset workflow.
 
 ## CLI and API access
 
-For self-managed RDI, use the [`redis-di get-rejected`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-get-rejected" >}})
-command to inspect rejected records.
+For self-managed RDI, use the [`redis-di list-dlqs`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlqs" >}})
+command to see the dead-letter queues and the
+[`redis-di list-dlq-records`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records" >}})
+command (also available as `redis-di get-rejected`) to inspect the rejected records of a queue.
 
 For Redis Cloud RDI, connect to the RDI database and inspect the corresponding
 DLQ streams directly when you need details that are not shown in the Redis Cloud

content/integrate/redis-data-integration/faq.md

@@ -100,7 +100,7 @@ job then RDI can't transform the data. When this happens, RDI will store the ori
 in a "dead letter queue" along with a message to say why it was rejected. The dead letter
 queue is stored as a capped stream in the RDI staging database. You can see its contents
 with Redis Insight or with the
-[`redis-di get-rejected`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-get-rejected" >}})
+[`redis-di list-dlq-records`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-list-dlq-records" >}})
 command from the CLI.
 
 See [Rejected records]({{< relref "/integrate/redis-data-integration/data-pipelines/rejected-records" >}}) for more information about DLQ.

content/integrate/redis-data-integration/installation/install-k8s.md

@@ -64,6 +64,8 @@ Complete the following steps before installing the RDI Helm chart:
 -   If you want to use a private image registry,
     [prepare it with the RDI images](#using-a-private-image-registry).
 
+-   [Download the RDI CLI](#download-the-rdi-cli), which you use to deploy and manage pipelines.
+
 ### Create the RDI database
 
 RDI uses a database on your Redis Enterprise cluster to store its state
@@ -146,6 +148,41 @@ To pull images from a private image registry, you must provide the image pull se
 -   [Google Kubernetes Engine (GKE)](https://cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images)
 -   [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration?tabs=azure-cli)
 
+### Download the RDI CLI
+
+You manage RDI with the [`redis-di` CLI]({{< relref "/integrate/redis-data-integration/reference/cli" >}}),
+which you use to deploy pipelines, set secrets, and inspect status. Unlike the VM installation, which
+bundles the CLI, a Kubernetes installation requires you to download it separately from the Redis
+download center.
+
+The CLI is currently built for the following platforms. Download the binary that matches the operating
+system and architecture of the machine you will run it from. On Linux and macOS, if you are not sure
+which to choose, run `uname -sm`: `Linux x86_64` is Linux amd64, `Linux aarch64` is Linux arm64,
+`Darwin x86_64` is macOS on Intel, and `Darwin arm64` is macOS on Apple silicon.
+
+| Platform | Download |
+| :-- | :-- |
+| Linux, x86-64 (amd64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/linux-amd64/redis-di) |
+| Linux, ARM64 (aarch64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/linux-arm64/redis-di) |
+| macOS, Intel (amd64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/darwin-amd64/redis-di) |
+| macOS, Apple silicon (arm64) | [`redis-di`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/darwin-arm64/redis-di) |
+| Windows, x86-64 (amd64) | [`redis-di.exe`](https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/{{< rdi-version >}}/bin/windows-amd64/redis-di.exe) |
+
+For example, to download the CLI for Linux amd64, make it executable, and put it on your `PATH`:
+
+```bash
+export RDI_VERSION={{< rdi-version >}}
+wget https://redis-enterprise-software-downloads.s3.amazonaws.com/redis-di/cli/$RDI_VERSION/bin/linux-amd64/redis-di
+chmod +x redis-di
+sudo mv redis-di /usr/local/bin/
+```
+
+{{< note >}}The macOS and Windows binaries are not currently signed or notarized, so the operating
+system may block them the first time you run them. On macOS, allow the binary to run in
+**System Settings > Privacy & Security**, or remove the quarantine attribute with
+`xattr -d com.apple.quarantine ./redis-di`, and then run it again. On Windows, if Microsoft Defender
+SmartScreen blocks it, choose **More info > Run anyway**.{{< /note >}}
+
 ## Supported versions of Kubernetes and OpenShift
 
 {{< embed-md "rdi-k8s-reqs.md" >}}
@@ -350,12 +387,13 @@ kubectl get pod -n rdi
 NAME                      READY  STATUS  	RESTARTS  AGE
 collector-api-<id>        1/1    Running  0         29m
 rdi-api-<id>              1/1 	 Running 	0      	  29m
-rdi-metric-exporter-<id>  1/1    Running 	0      	  29m
 rdi-operator-<id>         1/1 	 Running 	0      	  29m
 rdi-reloader-<id>         1/1 	 Running 	0      	  29m
 ```
 
-You can verify that the RDI API works by adding a connection to the RDI API server to
+You can verify that the RDI API works by running
+[`redis-di info`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-info" >}})
+against it, or by adding a connection to the RDI API server to
 [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}).
 
 ## Using ingress controllers
@@ -390,13 +428,13 @@ section to learn how to do this.
 ## Deploy a pipeline
 
 When the Helm installation is complete and you have prepared the source database for CDC,
-you are ready to start using RDI.
-Use [Redis Insight]({{< relref "/develop/tools/insight" >}}) to
+you are ready to start using RDI. See the guides on how to
 [configure]({{< relref "/integrate/redis-data-integration/data-pipelines" >}}) and
 [deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}})
-your pipeline (see
+RDI pipelines for more information. You can also configure and deploy a pipeline
+using [Redis Insight]({{< relref "/develop/tools/insight" >}}). See
 [RDI in Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
-for full details on how to do this).
+for full details on how to connect to RDI and deploy pipelines.
 
 ## Uninstall RDI