UnRetryableError bigquery_load

[toyaser]

Environments

• fluentd version: v1.15.0-1.0
• plugin version: 3.0.0

Configuration

<match audit-logs-bigquery.**>
  @type bigquery_load

  <buffer>
    path /mnt/audit-logs-bigquery.*.buffer
    flush_interval 10s
    total_limit_size 1g
  </buffer>

  auth_method json_key
  json_key "#{ENV['FLUENT_BIGQUERY_JSON_KEY']}"
  project "#{ENV['FLUENT_BIGQUERY_PROJECT']}"
  dataset "#{ENV['FLUENT_BIGQUERY_DATASET']}"
  table "#{ENV['FLUENT_BIGQUERY_TABLE']}"
  @log_level "{'info'}"
</match>

Expected Behavior

No Error

Actual Behavior

I believe all the logs are being written as expected, but I am seeing this error many times, I am not sure I understand what the error is and was wondering if there is something wrong on my end, or with the plugin. I found this old issue here, but its with the insert and I need to use the load as the insert does not support the JSON data type.

Log (if you have)

2022-10-15 19:07:57.816134286 +0000 fluent.error: {"error":"#<Fluent::BigQuery::UnRetryableError: notFound: Not found: Job project-id-01:job_GrVSui91crPvhn2DqmBvVCc2upYA>","chunk":"5eb1774d7fd7dc5d0c6f454e4d9a7839","project_id":"project-id-01","dataset_id":"dataset","table_id":"table_name","job_id":"job_GrVSui91crPvhn2DqmBvVCc2upYA","message":"failed to poll load job error_class=Fluent::BigQuery::UnRetryableError error=\"notFound: Not found: Job project-id-01:job_GrVSui91crPvhn2DqmBvVCc2upYA\" chunk=\"5eb1774d7fd7dc5d0c6f454e4d9a7839\" project_id=\"project-id-01\" dataset_id=\"dataset\" table_id=\"table_name\" job_id=\"job_GrVSui91crPvhn2DqmBvVCc2upYA\""}

[joker1007]

I cannot re-produce the error on my environment.
I'm sorry. I don't know the detail of the error.

I added sample fluent.conf recently. It may help you.
https://github.com/fluent-plugins-nursery/fluent-plugin-bigquery/blob/master/integration/fluent.conf

And, It may also be a good to check the permissions of the Google account.

[kschouw]

@toyaser I ran into a similar issue.

After many hours the solution was to specify the location, I know the docs mention that EU and US locations don't need this set but for some reason europe-west3 didn't register as EU in my case.

So just double check your dataset location and use:

location {some_location}

Hope it helps.

[joker1007]

@kschouw
Thank you very much!

Bigquery API connects US location as a default behavior.
And so, Job API cannot find the load job in EU location, maybe.
If you use EU location, location config must be set.

[toyaser]

@kschouw 🙇 Thank you so much! you are correct.

I added

location us-east1

The error is now gone!!!

@joker1007 as you can see, my location is US and I am still seeing the error. maybe the default is not really working?

[joker1007]

@toyaser

Thank you for the report!!
In current Bigquery API, maybe location parameter needs to be set.
I will fix the default configuration parameter and README.