add resync and poll #32734
add resync and poll #32734
Conversation
| context: dg.AssetExecutionContext, fivetran: FivetranWorkspace | ||
| ): | ||
| # Perform a full historical resync of all tables in the connector | ||
| yield from fivetran.resync_and_poll(context=context) |
There was a problem hiding this comment.
Oh this is super helpful.
I wonder if in the example we should supply a config object and then conditionally call resync_and_poll if someone provides a resync=True flag, that way we can have one set of assets.
There was a problem hiding this comment.
yeah I think that makes sense -- I was following the pattern from our old Fivetran resource but it's fair to say perhaps you want to alter the yield depending on config.
There was a problem hiding this comment.
@cmpadden respun with config on sync_and_poll
There was a problem hiding this comment.
I like where this landed, originally I was thinking you'd have the user define the config and have something like:
if config.resync:
yield from fivetran.resync_and_poll(context=context)
else:
yield from fivetran.sync_and_poll(context=context)
But defining the configuration for them is much cleaner and "batteries included".
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| # Mocked FivetranClient.resync_and_poll returns None if the connector is paused | ||
| result = materialize( | ||
| [my_fivetran_assets], | ||
| resources={"fivetran": workspace}, | ||
| ) | ||
| assert result.success | ||
| asset_materializations = [ | ||
| event | ||
| for event in result.events_for_node(connector_id) | ||
| if event.event_type_value == "ASSET_MATERIALIZATION" | ||
| ] | ||
| assert len(asset_materializations) == 0 | ||
|
|
||
| captured = capsys.readouterr() | ||
| assert re.search( | ||
| r"dagster - WARNING - (?s:.)+ - The connector with ID (?s:.)+ has not been resynced.", | ||
| captured.err, | ||
| ) |
There was a problem hiding this comment.
Test logic error: This test case does not pass resync=True in the run_config, so it will call _sync_and_poll() which invokes client.sync_and_poll(), not client.resync_and_poll(). However, the resync_and_poll fixture only mocks FivetranClient.resync_and_poll(). This means:
- The mocked return value of
None(for paused connector) will never be returned - The test will actually call the real
sync_and_pollmethod (or fail if no mock exists for it) - The assertion checking for the "has not been resynced" warning message will fail
Fix:
result = materialize(
[my_fivetran_assets],
resources={"fivetran": workspace},
run_config={"ops": {connector_id: {"config": {"resync": True}}}},
)| # Mocked FivetranClient.resync_and_poll returns None if the connector is paused | |
| result = materialize( | |
| [my_fivetran_assets], | |
| resources={"fivetran": workspace}, | |
| ) | |
| assert result.success | |
| asset_materializations = [ | |
| event | |
| for event in result.events_for_node(connector_id) | |
| if event.event_type_value == "ASSET_MATERIALIZATION" | |
| ] | |
| assert len(asset_materializations) == 0 | |
| captured = capsys.readouterr() | |
| assert re.search( | |
| r"dagster - WARNING - (?s:.)+ - The connector with ID (?s:.)+ has not been resynced.", | |
| captured.err, | |
| ) | |
| # Mocked FivetranClient.resync_and_poll returns None if the connector is paused | |
| result = materialize( | |
| [my_fivetran_assets], | |
| resources={"fivetran": workspace}, | |
| run_config={"ops": {connector_id: {"config": {"resync": True}}}}, | |
| ) | |
| assert result.success | |
| asset_materializations = [ | |
| event | |
| for event in result.events_for_node(connector_id) | |
| if event.event_type_value == "ASSET_MATERIALIZATION" | |
| ] | |
| assert len(asset_materializations) == 0 | |
| captured = capsys.readouterr() | |
| assert re.search( | |
| r"dagster - WARNING - (?s:.)+ - The connector with ID (?s:.)+ has not been resynced.", | |
| captured.err, | |
| ) | |
Spotted by Graphite Agent
Is this helpful? React 👍 or 👎 to let us know.
There was a problem hiding this comment.
Pending a green docs / docs_snippets build, I say we go ahead and merge this.
The oss-internal-compatibility error is unrelated.
[2025-11-14T21:49:16Z] src/connections/ConnectionStatus.tsx:1:19 - error TS2305: Module '"@dagster-io/ui-components"' has no exported member 'Intent'.
[2025-11-14T21:49:16Z]
[2025-11-14T21:49:16Z] 1 import {IconName, Intent} from '@dagster-io/ui-components';
Summary & Motivation
Resolves #32458
How I Tested These Changes
added tests, pytest, and BK
Changelog
Added a
resync_and_pollmethod to FivetranWorkspace