test(bigquery): add long-running heavy poller test - #6406
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds a new integration test, job_service_poller_heavy, to test the BigQuery job poller with a delayed query. Feedback was provided regarding the use of a tight busy loop in the BigQuery script, which can exceed the statement execution limit and cause test failures; using a temporary JavaScript UDF is recommended instead.
| let query = r#" | ||
| DECLARE DELAY_TIME DATETIME; | ||
| DECLARE WAIT STRING; | ||
| SET WAIT = 'TRUE'; | ||
| SET DELAY_TIME = DATETIME_ADD(CURRENT_DATETIME, INTERVAL 5 SECOND); | ||
|
|
||
| WHILE WAIT = 'TRUE' DO | ||
| IF (DELAY_TIME < CURRENT_DATETIME) THEN | ||
| SET WAIT = 'FALSE'; | ||
| END IF; | ||
| END WHILE; | ||
| "#; |
There was a problem hiding this comment.
Using a tight busy loop in BigQuery scripting (e.g., WHILE WAIT = 'TRUE' DO ...) will quickly exceed the BigQuery scripting limit of 1,000 executed statements per query, causing the test to fail with a Resources exceeded error.
To implement a reliable delay without hitting the statement limit, you can use a temporary JavaScript UDF that performs the busy-wait. This executes as a single statement in BigQuery.
let query = r#"
CREATE TEMP FUNCTION wait(ms INT64) RETURNS STRING LANGUAGE js AS '
const start = Date.now();
while (Date.now() - start < ms) {}
return "done";
';
SELECT wait(5000);
"#;There was a problem hiding this comment.
https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/datetime_functions#current_datetime - Based on this, I think current datetime will not update over the course of the function so this will not work as expected
There was a problem hiding this comment.
Thank you, TIL. I borrowed this from the Python SDK 😢
To make the execution time deterministic, I think we can switch to a massive Cartesian cross-join (which is what the Go team does).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6406 +/- ##
==========================================
- Coverage 96.18% 96.18% -0.01%
==========================================
Files 288 288
Lines 75424 75424
==========================================
- Hits 72548 72547 -1
- Misses 2876 2877 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add an integration test that proves the
JobPollerLRO backoff logic is invoked.Typical testing queries (e.g.,
SELECT 1) execute so fast that the initialinsert_jobREST call fast-paths and returnsDONEinstantly. This suppresses the inner.poll()loop from back-off sleeping, leaving the LRO logic untested.Follow-up to #6232