Switch datastore to single-region?

[snarfed]

Our datastore (technically Firestore in Datastore mode) is currently multi-region (https://cloud.google.com/datastore/docs/locations). Multi-region pricing is double single-region for operations (eg reads and writes) and 20% higher for storage. Multi-region has marginally better availability and durability guarantees, but only in the .001% range, which is way overkill for us. We should probably switch to single region.

Sadly we can't change our existing database's regionality, so we'd have to make a new one, copy over all the data, and then switch our app to point to it. Afaict that would cost $240 (via clone) or $400 (via writes) up front, but would then save us ~$630/mo. Will take some delicate ops work, I'd probably propose we take planned downtime to make it simpler/safer, but not a huge time commitment otherwise, probably just a few days.

Our app is in us-central-1, so we'd probably put the single-region datastore there. One open question is that https://cloud.google.com/datastore/docs/locations#location-r says:

This location does not support App Engine. If you plan to use App Engine, you should choose a different location.

I think that's either talking about a regional endpoint, which I don't think we'd need, or it's out of date, but I'm not sure. Need to test first.

More background on cost optimization: #1149

[Ryuno-Ki]

When planning this also consider what might happen with the region going down (I'm sure you noticed what AWS outage caused recently) as well as an unilateral termination because the owner decides to adhere to some orange head's bidding. (Speculative but not totally unplausible).

What would a failover to another region look like? Can we have backups stored outside the US?

[snarfed]

Good questions! And good ideas...but they're pretty far beyond our requirements and SLOs right now. Maybe if/when our budget gets to 10x+ its current size. 😁