MongoDB replication instrumentation

[rkistner]

This instruments time spent in various parts of the replication process for MongoDB, and logs with each change stream batch. Example log output:

info: [powersync_4] Processed batch of 6000 changes / 4055192 bytes in 320ms {"bytes":4055192,"count":6000,"duration":320,"evaluating_duration":55,"flush_duration":135,"lock_delay":0,"parse_duration":21,"retry_delay":0,"wait_for_change_stream":61}

The goal here is to help diagnose slow replication issues on a high level. When replication is slower than expected, we want to know whether the bottleneck is (1) the source db or network, (2) the storage db, or (3) replication process CPU. The stats here help identify that.

This includes:

1. duration: Roughly equal total duration between batches.
2. wait_for_change_stream: Time spent waiting for the next batch on the change stream, including the source db waiting for more changes, scanning the oplog, processing the change stream pipeline, and network transfer time.
3. parse_duration: Time spent converting from raw change stream buffers to input for sync config.
4. evaluating_duration: Time spent evaluating sync queries.
5. flush_duration: Time spent writing to the storage database.
6. lock_delay: Time spent waiting for other replication jobs within the same process.
7. retry_delay: Time spent waiting for other replication jobs in different processes.

Fields 2-7 above give a rough breakdown of the total duration, but it is not an exhaustive list - there is more CPU overhead not explicitly tracked.

Other source databases are not currently tracking any of this. Postgres storage also doesn't report the storage-specific stats currently.

[changeset-bot]

🦋 Changeset detected

Latest commit: f1f4c0f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR