Buffered writes cause memory issues and delayed output in plugin execution

[caterryan]

Bug Report

During plugin execution, calls to influxdb3_local.write and influxdb3_local.write_to_db are buffered and deferred until the plugin completes, instead of writing incrementally. Similarly, log lines (influxdb3_local.log, warn, error) are emitted only after the plugin finishes. This behavior causes OOM errors and delayed feedback, which is particularly problematic for long-running or high-volume plugins. Writes and logs should ideally flush during execution rather than after completion.

Expected Behavior

influxdb3_local.write and influxdb3_local.write_to_db should write during plugin execution. Writes should not wait until the plugin completes.

Log lines (influxdb3_local log, warn, error) are written during plugin execution, not after the plugin completes.

Current Behavior

influxdb3_local.write and influxdb3_local.write_to_db are buffered and written after the plugin completes. This results in OOM when running on small nodes or when working with large data volumes.

Similar to writing, log lines (influxdb3_local log, warn, error) are written after the plugin completes. This prevents use cases such as long-running plugins from reporting on their status.

Possible Solutions

1. Write buffer on WAL flush.
2. Write directly to the /query endpoint
3. Do nothing and accept this behavior as a limitation

Context

Write

• influxdb3_local.write delegates to the Rust method PyPluginCallApi::write, which just calls LineBuilder.build() and pushes the resulting line protocol string into the in-memory PluginReturnState.write_back_lines buffer without touching storage.
• Every plugin invocation owns a fresh PluginReturnState; once the Python entry function returns, the runtime swaps out that state and hands it back to the caller.
• The Rust runner immediately feeds that state into handle_return_state, which joins the buffered lines and issues a single write_buffer.write_lp call against the plugin’s database; it also iterates any cross-database writes collected via write_to_db and flushes those too.
• Because handle_return_state runs (and is awaited) right after the Python function completes—across WAL, schedule, and request triggers—the actual database writes happen only after your plugin finishes executing, not at each write() call.

Logs

logs are emitted in three ways

1. to the host process
2. to SysEventStore
3. to PluginReturnState.log_lines

The first two should log immediately, but PluginReturnState.log_lines are not written until the plugin finishes.

[philjb]

I was gonna say this might be a good chunk of work and require new apis and/or reworking how the plugin system but i might be easier to call rust code from python plugins than i expected. Having the plugin call the http api would be a lot of overhead for IPC writes.

[pauldix]

Yeah, these should just call Rust functions immediately rather than buffering the state and only calling the functions after it returns. I'l be logging a separate issue for writes that will have them potentially calling the HTTP API, which we'll need to do when running the processing engine from non-ingest nodes.