Dealing with external data updates for slimming down s3 buckets

[horsto]

I have a computed table that stores tif stacks as attach@my_store objects in a storage bucket.
The size of these are getting out of hand (and their costs).

I want to reduce the size of my external store (my_store) by

• temporarily downloading all of those "massive" objects
• compressing those objects locally
• re-uploading, or "re-attaching" them to the original records in place (I don't want to deal with deleting, re-inserting downstream records)

How do I handle this safely, or more specifically my question is:
What happens when I do an update1() on each record in that table that has those attach@my_store objects. Does the blob in external storage get overwritten, does it end up under a different name in external storage, how can I safely clean those old, massive blobs (my original goal) from the bucket?

[horsto]

I am aware that my plan violates data integrity principles of downstream records / calculated results, but I am willing to live with that since I suspect the divergences to be minimal after compression and I can, at a later time point, start to recalculate downstream results.

[dimitri-yatsenko]

Consider doing this as part of datajoint 2.0 migration. The is a new feature of codecs for objects. What is your data type. What compression are you using? you may be able to covert the objects into zarr for compression and lazy loading.

[horsto]

Indeed, converting to zarr makes a lot of sense. I am very much looking forward to that feature.
Ok, then I think waiting for dj 2.0 is the right decision!