Skip to content

Conversation

@didlawowo
Copy link

Summary

This PR adds authentication support to OpenAIHTTPBackend, enabling GuideLLM to work with OpenAI-compatible servers that require authentication on all endpoints (including /health).

Problem

When using GuideLLM with LiteLLM proxy or similar OpenAI-compatible servers that require authentication, the benchmark fails during backend validation because the /health endpoint check doesn't include authentication headers:

RuntimeError: Backend validation request failed. Could not connect to the server or validate the backend configuration.

The OpenAISettings class already defines api_key, bearer_token, and headers fields, but these were never used in the actual HTTP requests.

Solution

  • Add api_key, bearer_token, and headers parameters to OpenAIHTTPBackend.__init__()
  • Read default values from settings.openai when not explicitly provided
  • Apply authentication headers to:
    • Backend validation requests (/health endpoint)
    • Available models requests (/v1/models endpoint)
    • All generation requests (merged with request-specific headers)

Usage

# Via environment variables (recommended)
export GUIDELLM__OPENAI__API_KEY=sk-xxx
guidellm benchmark run --target http://litellm:4000 ...

# Via backend kwargs
guidellm benchmark run --target http://litellm:4000 \
  --backend-kwargs '{"api_key": "sk-xxx"}' ...

Priority Order for Headers

  1. Request-specific headers (highest priority)
  2. Explicit api_key/bearer_token/headers parameters
  3. Settings from settings.openai (lowest priority)

Testing

  • Tested with LiteLLM proxy requiring master key authentication
  • /health endpoint now returns 200 instead of 401
  • Benchmarks complete successfully against authenticated endpoints

Related

This makes the documented example in the docstring actually work:

backend = OpenAIHTTPBackend(
    target="http://localhost:8000",
    model="gpt-3.5-turbo",
    api_key="your-api-key"  # This param was in the example but didn't exist!
)

Add api_key, bearer_token, and headers parameters to OpenAIHTTPBackend
to enable authentication with OpenAI-compatible servers that require it.

Changes:
- Add api_key, bearer_token, and headers params to __init__
- Read default values from settings.openai when not explicitly provided
- Apply auth headers to backend validation (/health endpoint)
- Apply auth headers to available_models() requests
- Merge default headers with request-specific headers in resolve()

This fixes issues when using GuideLLM with LiteLLM proxy or other
OpenAI-compatible servers that require authentication on all endpoints
including /health.

Usage:
  # Via environment variables (recommended)
  export GUIDELLM__OPENAI__API_KEY=sk-xxx
  guidellm benchmark run --target http://litellm:4000 ...

  # Via backend kwargs
  guidellm benchmark run --target http://litellm:4000 \
    --backend-kwargs '{"api_key": "sk-xxx"}' ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant