Update of metadata during a replication fails when base path is not matched in all partitions

[patduin]

Sometimes replication fails with an error like:

Caused by: com.hotels.bdp.circustrain.api.CircusTrainException: Partition path 's3://bucket/db/table/1539359466542/acquisition_instant=20181012T155106Z' does not seem to belong to data source path 's3://bucket/table'

Or something similar. This is caused by a check in the code that tries to create the target partition location based on the source location layout. If possible this should be fixed in the metadata of the source table. But since the data is actually copied it would be nice if Circus Train can make an estimate on what the path should look like.

The underlying issue is that we have several places where we check try to estimate a relative path based on the base table location. This means that if partitions don't share a base location we cannot replicate it.

We should improve Circus Train to allow replication even if this is the case.
We need to rewrite the Copiers and HdfsSnapshotLocationManager to generate the partition locations that need to be replicated + the matching partition name (we can use Hive's Warehouse class for that). A Copier can use that information to generate the output location for the data.

This implies we don't copy the input folders anymore from the source so for instance given this:

Table partition by column year:String.

Base: s3://base/db/table/
Partition1 = s3://base/db/table/year=2020/
Partition2 = s3://base2/table/year=2021/

This would result in:

Replication base location= s3:/target/db/table/
Partition1 = s3:/target/db/table/year=2020/
Partition2 = s3:/target/db/table//year=2021/

But also:

Base: s3://base/db/table/
Partition1 = s3://base/db/table/a/b/c/year=2020/

This would result in:

Replication base location= s3:/target/db/table/
Partition1 = s3://target/db/table/year=2020/

^ Currently we can replicate this and we keep a/b/c in the path, this would change.

[massdosage]

We should see whether we can make this behaviour configurable for backwards compatibility.

Do you want to add the event ids to the above paths or leave them out for simplicity? A good use case to test this out would be a two step replication, currently the final target would have two event ids in the location paths, with this change it would only have one (that of the second replication).