OADP Must-gather

New Markdown UI for summarized results

In order to better facilitate the quick analsysis of customer issues, OADP now offers a markown summary of the collected information. The oadp-must-gather-summary.md file can be found under the clusters directory in the oadp-must-gather results.

Example screenshot #1

Example screenshot #2

Developer Setup

To test OADP Must-gather, run

go run cmd/main.go -h
go run cmd/main.go

To test OADP Must-gather with oc adm must-gather, run

podman build -t ttl.sh/oadp/must-gather-$(git rev-parse --short HEAD)-$(echo $RANDOM):1h -f Dockerfile . --platform=<cluster-architecture>
podman push <this-image>
oc adm must-gather --image=<this-image> -- /usr/bin/gather -h
oc adm must-gather --image=<this-image>

OADP Must-gather is also tested through OADP E2E tests, being run after test cases and checking if summary does not contain errors and all required objects were collected.

To test omg tool, create omg.Dockerfile file

FROM python:3.10.12-slim-bullseye

WORKDIR /test-omg
COPY ./ ./
RUN pip install o-must-gather

and run

podman build -t omg-container -f omg.Dockerfile .
podman run -ti --rm omg-container bash
# inside container
omg use must-gather/clusters/
omg get backup -n <namespace> # and other OADP resources

To update OADP Must-gather go.mod dependencies, run

make update-deps

This defaults to the oadp-dev branch. It pulls from the following sources:

• github.com/openshift/oadp-operator — OADP operator API types (DPA, CloudStorage, etc.)
• github.com/migtools/oadp-non-admin — Non-admin API types
• github.com/migtools/oadp-vm-file-restore — VM file restore API types

As new CRD sources are added to the must-gather, they should also be added to the update-deps Makefile target.

Update it often. It must be updated prior to releases.

Possible necessary updates over the time

go get github.com/openshift/oc@<branch-or-commit>
go mod tidy
go mod verify

OADP release

Prior to each release, OADP Must-gather must be updated.

To update dependencies and version constants in one step, run

make prepare-release BRANCH=<release-branch> VERSION=<version>

For example:

make prepare-release BRANCH=oadp-1.6 VERSION=1.6

This runs update-deps against the release branch and sets mustGatherVersion and mustGatherImage in pkg/cli.go.

Note: The velero fork version in the replace directive of go.mod must still be verified manually to match the oadp-operator.

Deprecated folder

Scripts under deprecated/ folder are for backwards compatibility with old OADP Must-gather shell script. Users should use new OADP Must-gather Go script, as highlighted in product documentation.

Standards

OADP Must-gather must comply with https://github.com/openshift/enhancements/blob/995b620cb04c030bf62c908e188472fe7031a704/enhancements/oc/must-gather.md?plain=1#L94-L104

1. Must have a zero-arg, executable file at /usr/bin/gather that does your default gathering

OADP Must-gather binary can be called without args. All OADP Must-gather binary args are optional

2. Must produce data to be copied back at /must-gather. The data must not contain any sensitive data. We don't string PII information, only secret information.

OADP Must-gather collected data is stored at /must-gather folder in the same path the binary was called.

Most of the data is collected through oc adm inspect command (including Secrets). The other data are cluster information, OADP related information (CRDs and CRs) and storage information (StorageClasses, VolumeSnapshotClasses and CSIDrivers CRDs and CRs). These objects should not contain any sensitive data.

3. Must produce a text /must-gather/version that indicates the product (first line) and the version (second line, major.minor.micro-qualifier), so that programmatic analysis can be developed.

OADP Must-gather stores version information in /must-gather/version file

Example content of the file

OpenShift API for Data Protection (OADP) Must-gather
oadp-dev-branch

4. Should honor the user-provided values for --since and --since-time, which are passed to plugins via environment variables named MUST_GATHER_SINCE and MUST_GATHER_SINCE_TIME, respectively.

TODO oc adm inspect command is called through Go code. But both since and since-time are private. Need to change this in https://github.com/openshift/oc/blob/ae1bd9e4a75b8ab617a569e5c8e1a0d7285a16f6/pkg/cli/admin/inspect/inspect.go#L118-L119 to allow usage in OADP Must-gather