S3Source and Outputer

[nsmith-]

Ready to go

[Dr15Jones]

I got up to the Source. Will stop until next week.

[Dr15Jones]

I think a mark down file describing the organization of the data within Ceph would be helpful.

[Dr15Jones]

Here is a slightly radical idea for you. You do not actually require each data product stripe to contain the events in the same order. You only care that within a given block of a stripe the blocks for all data products hold the same set of events across all the stripes.

With that in mind, you would not need to wait until all data products have serialized for an event before writing them out to the stripes. You just need to know for each given event which block the event belongs with and then for a data product's block within that stripe you have to also record where each event begins. That recording could be as simple as storing ever increasing counter which matches to the counter stored in the stripe containing the event's meta data.