The current Avro writer and reader implementations are not that efficient because we need to use avro::GenericDatum as the intermediate layer:
It would be more efficient if we can get rid of conversion from/to avro::GenericDatum by directly using the encoder/decoder in the Avro writer/reader. It can greatly improve the performance of manifest file I/O.
The current Avro writer and reader implementations are not that efficient because we need to use
avro::GenericDatumas the intermediate layer:Avro writer:
ArrowArray->avro::GenericDatum->avro::EncoderAvro reader:
avro::Decoder->avro::GenericDatum->ArrowArray(feat: eliminate GenericDatum in Avro reader for performance #374)It would be more efficient if we can get rid of conversion from/to
avro::GenericDatumby directly using the encoder/decoder in the Avro writer/reader. It can greatly improve the performance of manifest file I/O.