Java 20 deprecates all constructors of the class java.net.URL

[rzo1]

similar to crawler-commons/crawler-commons#522

We should take care of it in StormCrawler too.

[TamimEhsan]

@rzo1 can I take this one?

[rzo1]

Feel free. You might want to have a look at crawler-commons/crawler-commons#526 and related PRs. Might give a first impression / idea.

[TamimEhsan]

@rzo1 there seems to be a lot going on there! Probably better if I start working after those PRs are merged.

[rzo1]

Think it's fine to start. We don't have a lot of occurences, afair.

[rzo1]

@TamimEhsan Do you still want to work on this? Crawler Commons 1.6 is now released.

[rzo1]

I pushed an initial migration on a separate branch (1597). We’re now seeing some failures in BasicURLNormalizerTest because URI enforces RFC 3986 compliance strictly, whereas URL is more tolerant of invalid input. We need to decide how we want to handle this going forward. In practice, a crawler may encounter malformed URIs but still attempt to normalize them into a valid URL. The open question is how much recovery or sanitization we want to perform versus simply failing when an invalid URI is encountered. WDYT @sebastian-nagel @jnioche ?