nvidia repos failing on re-sync - Error: Task failed: ''NoneType' object has no attribute 'artifact_set_sha256''

[hbrown-uiowa]

Version
pulpcore 3.89.1
pulp_deb 3.7.0
Installed via container: quay.io/pulp/pulp:3.89.1

Describe the bug
Repeated syncs of the nvidia cuda repos fail with 'NoneType' object has no attribute 'artifact_set_sha256''

To Reproduce

Create the repo:

NAME="nvidia-cuda-24.04-amd64"
pulp deb remote create --name=$NAME --url="https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/" --distribution /
pulp deb repository create --name=$NAME --remote=$NAME
pulp deb repository sync --name=$NAME
PUB_HREF=$(pulp deb publication -t verbatim create --repository=$NAME | jq -r '.pulp_href')
pulp deb distribution create --name=${NAME}-bimonthly --base-path=${NAME}-bimonthly --publication $PUB_HREF

Note that the /pulp/content/$NAME/Packages file exists

Re-run a sync:

pulp deb repository sync --name=$NAME 
PUB_HREF=$(pulp deb publication -t verbatim create --repository=$NAME | jq -r '.pulp_href')
pulp deb distribution update --name=${NAME}-bimonthly --base-path=${NAME}-bimonthly --publication $PUB_HREF

Note that /pulp/content/$NAME/Packages has disappeared.

Re-run the sync pulp deb repository sync --name=$NAME and get the sync fail with 'NoneType' object has no attribute 'artifact_set_sha256''

Expected behavior
A new sync/publication/distribution update would not delete the Packages file and would complete successfully.

Additional context
Re-doing the sync as

pulp deb repository sync --name=$NAME --no-optimize

makes the repository usable again until the next sync happens that does not use --no-optimize

[quba42]

Interesting. I don't think we changed much with pulp_deb (other than maybe ACS support, but I don't think that would cause this). Maybe something changed in the repo? It would be interesting to re-test using an older pulp_deb version to see if that is also affected.

[hbrown-uiowa]

The Packages file disappearing occurs on pulpcore 3.82.1 and pulp_deb 3.5.2, but subsequent syncs don't fail. Ultimately, with no Packages file, the repo isn't usable.

[hbrown-uiowa]

I'm also seeing the same behavior when I try it with pulpcore 3.65.0 and pulp_deb 3.4.0. I'm wondering if there's a sync with no new debs to download that it somehow short circuits and removes the Packages file and then never regenerates it.

[quba42]

Since I am unsure when I will get around to experimenting myself, I will start by giving some high level thoughts and pointers:

First, a quick summary of what --optimize does: It checks if the sha256 checksum of a release file or Packages file has not changed since the last sync, and if it has not, it skips processing of that Packages file during the sync. It is weird that not processing something would lead to it being removed from the repo.

My best guess is that there is some unexpected interaction with independent features like mirror=True sync option. mirror sync is supposed to have the effect to remove things from the repo that are no longer part of the upstream repo. Are you using mirror=True for your syncs? If yes, try switching to mirror=False to check if that "fixes" the issue. Still would not explain why only this one repository is suddenly affected.

The other thing I can say is that I remember nvidia repos in particular being a particularly extreme example of a cobbled together flat repo format that just barely meets any APT repo specifications. I would not be at all surprised if they found another creative way to add a new idiosyncrasy to their repo metadata.

[hbrown-uiowa]

As far as I can tell, mirror has always been False and my debugging has confirmed that it was False.

Thus far, initial sync (or a --no-optimize sync) is clean and then the second sync with --optimize as the default causes the Packages file to disappear and then when I check the repository version:

# pulp deb repository version show --repository=$NAME | jq -r '.content_summary.removed'
{
  "deb.package_index": {
    "count": 1,
    "href": "/pulp/api/v3/content/deb/package_indices/?repository_version_removed=/pulp/api/v3/repositories/deb/apt/019b0f3f-edca-7661-83e1-7fe334fb1cf8/versions/14/"
  }
}

The Releases file just references the Packages file.

I will continue to try and figure out where things are breaking down.

[hbrown-uiowa]

This is still an issue under pulpcore 3.113.0 and pulpdeb 3.9.0

[hbrown-uiowa]

I still need to test this, but it is possible that when --optimize is set and it is a flat repo, then the code at lines 717-723 of pulp_deb/app/tasks/synchronizing.py tries to call _readd_previous_package_indices with stored_distribution set to "flat-repo" and then the function at lines 1359-1362 is filtering with relative_path__contains="flat-repo". The filter doesn't match anything, so there isn't a Packages file to re-add and when the repo sync finishes it does the delta between the old and new and sees the Packages file isn't there and drops it.

[hbrown-uiowa]

It looks like the problematic line is this:
PackageIndex.objects.filter(relative_path__contains=distribution) in the function def _readd_previous_package_indices(previous_version, new_version, distribution): in pulp_deb/app/tasks/synchronizing.py

When it is a flat-repo and --optimize is set, the distribution argument is set to flat-repo but the relative_path of the PackageIndex.object is set to Packages and never matches, so the PackageIndex content type never gets added back during the sync.

After the optimized sync, pulp deb repository version show --repository=$NAME then has:

  "content_summary": {
    "added": {},
    "removed": {
      "deb.package_index": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/package_indices/?repository_version_removed=/pulp/api/v3/repositories/deb/apt/019eace3-67b7-7789-89f7-caec349b7e89/versions/2/"
      }
    },`

and the next publication of the repository removes it.

The sync no longer fails with the original error message on pulpcore 3.113.0 and pulpdeb 3.9.0, but the repo is not usable because the Packages file is gone.

[quba42]

I had some time to re-read and re-test here.

I cannot reproduce the failed re-sync, but I can reproduce the issue that a (optimized) re-sync will remove the upstream Package index from the flat repo even though it should not do that.

This is definitely a bug. It is a bug that only affects verbatim publications, which makes it relatively low priority for us (the active plugin maintainers).

I will update the issue labels accordingly for now.

[quba42]

The workaround here is to use structured publications or else non-optimized syncs.