Skip to content

Circus Train should also sync the external schema for Avro Tables when replication mode is METADATA_UPDATE #144

@abhimanyugupta07

Description

@abhimanyugupta07

As a user of CT,

I want that the Circus Train should also sync the external schema for an Avro Table when the replication mode is either METADATA_UPDATE or METADATA_MIRROR.

Context
At the moment, when the replication mode is METADATA_UPDATE or METADATA_MIRROR, the table location is not needed to be provided in the config file which means that the null check on https://github.com/HotelsDotCom/circus-train/blob/master/circus-train-avro/src/main/java/com/hotels/bdp/circustrain/avro/transformation/AbstractAvroSerDeTransformation.java#L50 returns empty and the Avro Transform is not triggered which in turn will not copy the schema file to replica.

This was discovered while working on: #131

Related PR: #141

Possible Solution
CT can detect if the table location is not provided and in the Avro transform, it can get the table location from the target HMS.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions