Vector receives 401 Unauthorized from Logflare in self-hosted Kubernetes setup despite correct API key

[xmh1011]

Description

In a self-hosted Supabase environment deployed on Kubernetes, the vector service consistently fails to send logs to the logflare (analytics) service due to a 401 Unauthorized error.

This issue persists even when manually sending requests with curl directly to the logflare pod using the correct LOGFLARE_PUBLIC_ACCESS_TOKEN. The environment variables for the API key have been verified to be correctly set and matching in both the vector and logflare pods.

The root cause appears to be related to logflare running in its default multi-tenant mode (LOGFLARE_SINGLE_TENANT=false), where the generic /api/logs endpoint seems to be protected and requires additional project-specific context that vector might not be providing by default. A similar authentication issue is also observed in the standard docker-compose self-hosted setup.

Environment

• Deployment Method: Kubernetes (via official/community Helm chart)
• Image Versions:
  • timberio/vector:[0.49.0-alpine]
  • supabase/logflare:[1.19.0]
• Key Configuration:
  • LOGFLARE_SUPABASE_MODE: true (default)
  • LOGFLARE_SINGLE_TENANT: true (default)
    Here is my supabase:

NAME                                                        READY   STATUS    RESTARTS        AGE   IP              NODE           NOMINATED NODE   READINESS GATES
supabase215710662251257856-analytics-58dc7889f6-8g5dc       1/1     Running   0               37m   10.225.12.195   10.1.200.4     <none>           <none>
supabase215710662251257856-auth-75dc86dbb8-d2xzn            1/1     Running   0               37m   10.225.12.221   10.0.100.134   <none>           <none>
supabase215710662251257856-edge-function-85cb878b7f-5w9ng   1/1     Running   0               37m   10.0.100.99     10.0.100.9     <none>           <none>
supabase215710662251257856-imgproxy-7bbffcf487-gvxbg        1/1     Running   0               37m   10.0.100.103    10.0.100.129   <none>           <none>
supabase215710662251257856-kong-776689645f-rbc7q            1/1     Running   0               37m   10.225.12.216   10.1.200.4     <none>           <none>
supabase215710662251257856-meta-65cb5cd8f8-l6h58            1/1     Running   0               37m   10.0.100.101    10.0.100.129   <none>           <none>
supabase215710662251257856-realtime-5888d55766-849vf        1/1     Running   0               37m   10.225.12.219   10.0.100.134   <none>           <none>
supabase215710662251257856-rest-55f768f669-9rp86            1/1     Running   2 (6m42s ago)   37m   10.225.12.218   10.1.200.5     <none>           <none>
supabase215710662251257856-storage-7ff74b9c75-x7dnz         1/1     Running   0               37m   10.225.12.214   10.1.200.5     <none>           <none>
supabase215710662251257856-studio-64c566bbd9-h77wk          1/1     Running   0               36m   10.0.100.109    10.0.100.129   <none>           <none>
supabase215710662251257856-vector-4l7v4                     1/1     Running   0               37m   10.225.12.235   10.1.200.5     <none>           <none>
supabase215710662251257856-vector-5ntxt                     1/1     Running   0               37m   10.0.100.83     10.0.100.9     <none>           <none>
supabase215710662251257856-vector-j4kwn                     1/1     Running   0               37m   10.0.100.75     10.0.100.10    <none>           <none>
supabase215710662251257856-vector-kkk2v                     1/1     Running   0               37m   10.225.12.234   10.0.100.134   <none>           <none>
supabase215710662251257856-vector-lfgfb                     1/1     Running   0               37m   10.0.100.64     10.0.100.8     <none>           <none>
supabase215710662251257856-vector-v9hkj                     1/1     Running   0               37m   10.0.100.105    10.0.100.129   <none>           <none>
supabase215710662251257856-vector-vjxf5                     1/1     Running   0               37m   10.225.12.238   10.1.200.4     <none>           <none>

Steps to Reproduce

1. Deploy a Supabase instance on a Kubernetes cluster.
2. Allow the vector and logflare pods to run.
3. Observe the logs from the vector pod using kubectl logs -f <vector-pod-name>.
4. Notice the recurring 401 Unauthorized errors.

Expected Behavior

vector should successfully authenticate with logflare using the provided LOGFLARE_PUBLIC_ACCESS_TOKEN and send logs without any authorization errors. The sink should report successful requests.

Actual Behavior

vector is unable to send logs and repeatedly logs a 401 Unauthorized error.

1. Vector Pod Logs:
The following error message is logged by the vector container every 10-15 seconds:

2025-08-18T03:27:34.317820Z ERROR sink{component_kind="sink" component_id=logflare_unmatched component_type=http}:request{request_id=58}: vector::sinks::util::retries: Not retriable; dropping the request. reason="Http status: 401 Unauthorized" internal_log_rate_limit=true
2025-08-18T03:27:44.416921Z ERROR sink{component_kind="sink" component_id=logflare_unmatched component_type=http}:request{request_id=60}: vector::sinks::util::retries: Not retriable; dropping the request. reason="Http status: 401 Unauthorized" internal_log_rate_limit=true
2025-08-18T03:27:55.326211Z ERROR sink{component_kind="sink" component_id=logflare_unmatched component_type=http}:request{request_id=62}: vector::sinks::util::retries: Not retriable; dropping the request. reason="Http status: 401 Unauthorized" internal_log_rate_limit=true

2. Manual curl Request:
A manual POST request to the logflare pod's IP address and port also fails with the same error:

# Request sent to the Logflare pod
curl -X POST \
-H "Content-Type: application/json" \
-H "x-api-key: a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b" \
-d '{"events": [{"message": "This is a manual test log", "timestamp": "'$(date -u +"%Y-%m-%dT%H:%M:%S.%NZ")'"}]}' \
'http://<logflare-pod-ip>:4000/api/logs?source_name=manual.test'

# Response
{"error":"Unauthorized"}

The logflare is healthy.

curl http://10.225.12.195:4000/health

# Response

{"status":"ok","nodes":["logflare@127.0.0.1"],"proc_count":4010,"memory_utilization":"ok","repo_uptime":"78296","caches":{"Elixir.Logflare.Auth.Cache":"ok","Elixir.Logflare.Backends.Cache":"ok","Elixir.Logflare.Billing.Cache":"ok","Elixir.Logflare.Endpoints.Cache":"ok","Elixir.Logflare.Logs.LogEvents.Cache":"ok","Elixir.Logflare.Partners.Cache":"ok","Elixir.Logflare.SourceSchemas.Cache":"ok","Elixir.Logflare.Sources.Cache":"ok","Elixir.Logflare.TeamUsers.Cache":"ok","Elixir.Logflare.Users.Cache":"ok"},"nodes_count":1,"this_node":"logflare@127.0.0.1"}

Troubleshooting & Verification

The LOGFLARE_PUBLIC_ACCESS_TOKEN has been confirmed to be present and identical in both the logflare (analytics) and vector pod environments. This rules out a simple misconfiguration or a missing secret.

Verification in logflare pod:

$ kubectl exec -it supabase-analytics-pod-xxxx -- /bin/sh
# echo $LOGFLARE_PUBLIC_ACCESS_TOKEN
a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b

Verification in vector pod:

$ kubectl exec -it supabase-vector-pod-yyyy -- /bin/sh
/ # echo $LOGFLARE_PUBLIC_ACCESS_TOKEN
a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b

The token value used in the manual curl test matches the value of these environment variables. The 401 response indicates network connectivity is working, and the issue is at the application level.

Of course. Here is an optimized version of that final section, making it a powerful conclusion to your GitHub issue. It clearly demonstrates that the problem is not isolated to Kubernetes.

Confirmation in Docker Self-Host Environment

To confirm this is not a Kubernetes-specific issue, the exact same authentication failure was reproduced in a standard docker-compose self-hosted environment.

Despite the supabase-analytics container having the correct LOGFLARE_PUBLIC_ACCESS_TOKEN, a curl request from the host machine to http://localhost:4000 is still rejected with a 401 Unauthorized error.

1. Verifying the Token in the Docker Container:

# Verifying the token inside the running supabase-analytics container
$ docker exec -it supabase-analytics sh
# echo $LOGFLARE_PUBLIC_ACCESS_TOKEN
a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b

2. Failing Request from the Host Machine:
(Note: The request below uses the correct x-api-key header, which fails as expected.)

# Sending a request from the host machine fails
$ curl -X POST \
-H "Content-Type: application/json" \
-H "x-api-key: a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b" \
-d '{"events": [{"message": "This is a test log from my docker setup", "timestamp": "'$(date -u +"%Y-%m-%dT%H:%M:%S.%NZ")'"}]}' \
'http://localhost:4000/api/logs?source_name=docker.test'

{"error":"Unauthorized"}

This strongly suggests the issue lies within the application's core authentication logic when running in its default multi-tenant mode, independent of the deployment environment (Kubernetes or Docker).

But in docker self-host environment, vector can write logs to logflare.

[Ziinc]

Thanks for the report.
Can you check if the access tokens are created under the analytics oauth_access_tokens table?

[xmh1011]

Thanks for the report. Can you check if the access tokens are created under the analytics oauth_access_tokens table?

Yes

supabase216847585812422656=# select * from _analytics.oauth_access_tokens;
 id | application_id | resource_owner_id |                              token                               | refresh_token | expires_in | revoked_at |
 scopes  | previous_refresh_token |     inserted_at     |     updated_at      |          description          
----+----------------+-------------------+------------------------------------------------------------------+---------------+------------+------------+
---------+------------------------+---------------------+---------------------+-------------------------------
  1 |                |                 1 | a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b |               |            |            |
 public  |                        | 2025-08-21 09:15:04 | 2025-08-21 09:15:04 | LOGFLARE_PUBLIC_ACCESS_TOKEN
  2 |                |                 1 | 1fe43200adc404724cfdead88c4716393d86e414ccf83d3db879515e1a10e999 |               |            |            |
 private |                        | 2025-08-21 09:15:04 | 2025-08-21 09:15:04 | LOGFLARE_PRIVATE_ACCESS_TOKEN
(2 rows)

[Ziinc]

could you paste in your full docker configuration?

also, can you try to ingest into the pod directly with this curl

# Request sent to the Logflare pod
curl -X POST \
-H "Content-Type: application/json" \
-H "x-api-key: a62353e9a704ec12a0d26fde198f8b53ff4b55e19dfe78dd8ee0c49edd46131b" \
-d '{"message": "This is a manual test log"}' \
'http://<logflare-pod-ip>:4000/api/logs?source_name=manual.test'

i assume you were able to manually create the manual.test source in the ui,
and can you see if there is any difference when using the private access token to ingest?