CSAF Importer never completes task - Episode 2

[gildub]

Although this might look like a clone of #2133, it isn't.

The symptoms are the same but the reason is different.

In this case we're facing an issue between trustd and csaf-walker.
It seems for a network failure reason or equivalent, a file which exists in the URL tree is not fetched.
For example rhsa-2023_5095.json.sha256 is not fetched and csaf-walker automatically tries to fetch the sibling sha512 file.
Since, the file (i.e rhsa-2023_5095.json.sha512) doesn't exist, the walker returns an error to trustd importer which fails.

This happens at random times, more likely because of networks latency, but in such way that the CSAF importer never finishes getting 100% of the files (and assuming #2133 has been addressed).

[ctron]

This is not about the existence of the file, but about the failure of the request. If the file does not exist (404) it will be fine. If the file cannot be retrieved due to another error code which is 5xx, it will retry. If the server responds with another 4xx failure, it will error.

It is important to error, as any request error (other than 404) might hide a checksum file required for validating the content.

[gildub]

@ctron,

It's impossible from the logs (in default INFO mode) to tell the error code. Maybe the error codes should be propagated as well to help troubleshooting.

The first ERROR from trustify_module_importer indicates :
... error=Visitor error: Retrieval error: source error: Fetch error: Request error: error sending request for url (..../rhsa-2023_5095.json.sha512)
This is a chain of errors where the last one effectively mentions request error.
From that, you're saying : Yeah this is a request error, the file cannot be retrieved.
That's fair enough from the walker viewpoint.

Meanwhile the module importer never asked for that file, the walker made that choice to go after the .sha512 file because it didn't find the .sha256

So if we go back just before that first error, the rhsa-2023_5095.json.sha256 was supposed to be fetched but no error was propagated. If that's confirmed then we have an issue with the walker.

What do you think ?

[ctron]

Meanwhile the module importer never asked for that file, the walker made that choice to go after the .sha512 file because it didn't find the .sha256

No, it always tries to fetch both. In parallel.

[ctron]

https://github.com/scm-rs/csaf-walker/blob/3b7afca57f305e5c973e7bb7085bf2d89466929d/csaf/src/source/http.rs#L181-L219

[gildub]

@ctron, I understand they're fetched concurrently but I wonder how to properly inform the user the .sha256 which does exists on the server side didn't get retrieved ?

[ctron]

If the file cannot be found (404) it's considered optional. The user is not informed. If the file cannot be fetched because of any other reason, the user is informed by means of an error.

[helio-frota]

I saw two different errors: