Rate Limit Error when trying to create BigQuery Tables

[srasul]

When starting a new instance of logflare with Supabase self-hosted, we are seeing this error message during the startup of analytics:

supabase-analytics  | 12:45:53.801 [error] BigQuery dataset create error: 1_default: Exceeded rate limits: too many dataset metadata update operations for this dataset. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
supabase-analytics  | 
supabase-analytics  | 12:45:53.805 [error] BigQuery dataset create error: 1_default: Exceeded rate limits: too many dataset metadata update operations for this dataset. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
supabase-analytics  | 
supabase-analytics  | 12:45:53.808 [error] BigQuery dataset create error: 1_default: Exceeded rate limits: too many dataset metadata update operations for this dataset. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
supabase-analytics  | 
supabase-analytics  | 12:45:53.828 [error] BigQuery dataset create error: 1_default: Exceeded rate limits: too many dataset metadata update operations for this dataset. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas

After looking at code, it seems at this stage we are trying to create the BQ tables. And looking at the link in the error message, it seems that there is a rate-limit of 5 operations per 10 seconds:

Is there a way to add a small delay for these operations so that we don't hit a rate-limit?

We are using Logflare v1.8.2

[srasul]

according to:

https://cloud.google.com/bigquery/docs/troubleshoot-quotas#ts-maximum-update-table-metadata-limit

It seems this limit cannot be updated so one of the solutions is to add a delay between these operations so as not to hit this rate-limit

[Ziinc]

@srasul Thanks for the report, is this issue preventing you from starting up the supabase stack still?

[JLAcostaEC]

The same thing is happening to me...

@srasul Thanks for the report, is this issue preventing you from starting up the supabase stack still?

Well, basically, it doesn't cause any problems when starting supabase, but the logs return the response that the table <UUID> doesn't exist in the BQ DB.

The solution is to shut down the supabase container and restart it. It will then create the missing tables. This is a bit annoying, and I've had it happen that sometimes, when restarting the container, it seems to use a that doesn't exist. So I have to delete all the data from the Postgres DB (and BQ) and restart the container.

What I realized is that if I manually create the DB named 1_default, then this problem doesn't occur; logflare manages to create all the tables without any problems.

But... same question:

Is there a way to add a small delay for these operations so that we don't hit a rate-limit?