CheckThat AI Python SDK

A Python SDK for the CheckThat AI platform's unified LLM API with built-in fact-checking and claim normalization capabilities.

Features

🔄 Unified LLM Access: Access all the latest models from OpenAI, Anthropic, Google Gemini, xAI, and Together AI through a single API
🔍 Claim Normalization: Standardize and structure claims for analysis
✅ Fact-Checking: Built-in claim verification and evidence sourcing
🔌 OpenAI Compatible: Drop-in replacement for OpenAI Python SDK
⚡ Async Support: Full async/await support for high-performance applications
🛡️ Type Safety: Complete type hints for better development experience
🆕 Always Up-to-Date: Access to the newest models as soon as they're released

Installation

pip install checkthat-ai

Quick Start

Basic Usage

import os
from checkthat_ai import CheckThatAI

# Initialize the client - API key is required
api_key = os.environ.get("OPENAI_API_KEY")  # Must provide an API key
client = CheckThatAI(api_key=api_key)

# IMPORTANT: Always check available models first
models = client.models.list()
print("Available models:", models)

# Use exactly like OpenAI's client with latest models
response = client.chat.completions.create(
    model="gpt-5-2025-08-07",  # Use latest models
    messages=[
        {"role": "user", "content": "What is the capital of France?"}
    ]
)

print(response.choices[0].message.content)

Async Usage

import asyncio
from checkthat_ai import AsyncCheckThatAI

async def main():
    client = AsyncCheckThatAI(api_key="your-api-key")
    
    # Check available models first
    models = await client.models.list()
    
    response = await client.chat.completions.create(
        model="gpt-5-2025-08-07",  # Use latest available models
        messages=[
            {"role": "user", "content": "What is the capital of France?"}
        ]
    )
    
    print(response.choices[0].message.content)

asyncio.run(main())

Streaming Responses

# First, check what models support streaming
models = client.models.list()

response = client.chat.completions.create(
    model="gpt-5-2025-08-07",  # Use latest models
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Complete Working Example

Here's a comprehensive example showing various SDK features:

import os
import asyncio
from typing import List
from pydantic import BaseModel, Field
from checkthat_ai import CheckThatAI, AsyncCheckThatAI

# Define structured response models
class FactualResponse(BaseModel):
    answer: str = Field(description="The main answer to the question")
    confidence: float = Field(description="Confidence score between 0 and 1")
    sources: List[str] = Field(description="List of authoritative sources")
    key_claims: List[str] = Field(description="Main claims made in the response")

def basic_example():
    """Basic structured output example."""
    client = CheckThatAI(api_key=os.getenv("OPENAI_API_KEY"))
    
    # Always check available models first
    models = client.models.list()
    print("Available models:", len(models["models_list"]), "providers")
    
    # Using the parse() method with latest models for structured output
    completion = client.chat.completions.parse(
        model="gpt-5-2025-08-07",  # Use latest available models
        messages=[
            {"role": "user", "content": "What is the primary cause of climate change?"}
        ],
        response_format=FactualResponse
    )
    
    # Access the parsed response directly
    parsed = completion.choices[0].message.parsed
    print(f"Answer: {parsed.answer}")
    print(f"Confidence: {parsed.confidence}")
    print(f"Sources: {', '.join(parsed.sources)}")

def enhanced_example():
    """Example with claim refinement."""
    client = CheckThatAI(api_key=os.getenv("OPENAI_API_KEY"))
    
    completion = client.chat.completions.parse(
        model="o4-mini-2025-04-16",  # Use latest efficient models
        messages=[{
            "role": "user", 
            "content": "Explain the health effects of air pollution in urban areas"
        }],
        response_format=FactualResponse,
        refine_claims=True,  # Enable claim refinement
        refine_model="gpt-5-2025-08-07",  # Use best model for refinement
        refine_threshold=0.7
    )
    
    parsed = completion.choices[0].message.parsed
    print(f"Enhanced response with {len(parsed.key_claims)} refined claims")

async def async_example():
    """Async usage example."""
    client = AsyncCheckThatAI(api_key=os.getenv("OPENAI_API_KEY"))
    
    # Check available models asynchronously
    models = await client.models.list()
    
    completion = await client.chat.completions.parse(
        model="claude-sonnet-4-20250514",  # Use latest Anthropic models
        messages=[{
            "role": "user",
            "content": "What are the latest developments in renewable energy?"
        }],
        response_format=FactualResponse,
        refine_claims=True
    )
    
    parsed = completion.choices[0].message.parsed
    print(f"Async response: {parsed.answer[:100]}...")

if __name__ == "__main__":
    basic_example()
    enhanced_example()
    asyncio.run(async_example())

Structured Output Generation

Generate structured, type-safe responses using Pydantic models with the dedicated parse() method:

from checkthat_ai import CheckThatAI
from pydantic import BaseModel, Field
from typing import List

class MathStep(BaseModel):
    step_number: int = Field(description="The step number")
    explanation: str = Field(description="What happens in this step")
    equation: str = Field(description="The mathematical equation")

class MathSolution(BaseModel):
    problem: str = Field(description="The original problem")
    steps: List[MathStep] = Field(description="Step-by-step solution")
    final_answer: str = Field(description="The final answer")

client = CheckThatAI(api_key="your-api-key")

# Use the parse() method for structured outputs
response = client.chat.completions.parse(
    model="gpt-5-2025-08-07",  # Use latest models that support structured outputs
    messages=[
        {"role": "system", "content": "You are a helpful math tutor."},
        {"role": "user", "content": "Solve: 2x + 5 = 13"}
    ],
    response_format=MathSolution  # Pydantic model for structured output
)

# Access the parsed response directly
solution = response.choices[0].message.parsed
print(f"Problem: {solution.problem}")
print(f"Answer: {solution.final_answer}")
for step in solution.steps:
    print(f"Step {step.step_number}: {step.explanation}")

CheckThat AI Features

The SDK includes several enhancements beyond standard OpenAI compatibility:

Claim Refinement

Iteratively improve response accuracy through claim refinement:

response = client.chat.completions.parse(
    model="gpt-5-2025-08-07",  # Use latest models for refinement
    messages=[...],
    response_format=YourModel,
    refine_claims=True,      # Enable refinement
    refine_model="gpt-5-2025-08-07",   # Use best model for refinement process
    refine_threshold=0.7,    # Quality threshold (0.0-1.0)
    refine_max_iters=3       # Maximum refinement iterations
)

Available Evaluation Metrics

The SDK supports various quality evaluation metrics:

G-Eval: General evaluation framework
Bias: Bias detection and analysis
Hallucinations: Hallucination detection
Hallucination Coverage: Coverage analysis of hallucinations
Factual Accuracy: Fact-checking accuracy
Relevance: Content relevance assessment
Coherence: Response coherence evaluation

# Access available metrics programmatically
from checkthat_ai._types import AVAILABLE_EVAL_METRICS
print("Available metrics:", AVAILABLE_EVAL_METRICS)

Alternative: JSON Schema Format

You can also use JSON schema format directly with the regular create() method:

response = client.chat.completions.create(
    model="o4-mini-2025-04-16",  # Use latest models
    messages=[...],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "math_solution",
            "schema": {
                "type": "object",
                "properties": {
                    "problem": {"type": "string"},
                    "final_answer": {"type": "string"}
                },
                "required": ["problem", "final_answer"]
            }
        }
    }
)

# Parse the JSON response manually
import json
solution_data = json.loads(response.choices[0].message.content)

Structured Output Features

Type Safety: Full Pydantic model validation with the parse() method
IDE Support: Auto-completion and type hints
Flexible Schemas: Support for nested objects, lists, and optional fields
Two Approaches: Use parse() for Pydantic models or create() with JSON schema
Simple Integration: Automatic JSON schema generation from Pydantic models
Async Support: Available for both sync and async clients
Backend Routing: Automatically routed to appropriate structured output processing
Claim Refinement: Optional iterative improvement of generated content

Backend Implementation Guide

The SDK automatically handles Pydantic model conversion, so the backend needs to implement two endpoints with routing logic:

1. Two Endpoints: `/chat/completions`

# Backend routing logic
def chat_completions_handler(request):
    if 'response_format' in request:
        # Structured output request - route to appropriate handler
        if request.get('parse_mode', False):  # Using parse() method
            return handle_structured_output_parse(request)
        else:  # Using create() with JSON schema
            return handle_structured_output_create(request)
    else:
        # Regular chat request
        return handle_regular_chat(request)

2. Structured Output Request Format

For parse() method, the backend receives:

{
    "model": "gpt-5-2025-08-07",
    "messages": [
        {"role": "user", "content": "What is 2+2?"}
    ],
    "response_format": {
        "type": "json_schema",
        "json_schema": {
            "name": "my_response",
            "schema": {
                "type": "object",
                "properties": {
                    "answer": {"type": "string"},
                    "confidence": {"type": "number"}
                },
                "required": ["answer", "confidence"]
            }
        }
    },
    "refine_claims": true,
    "refine_model": "gpt-5-2025-08-07",
    "api_key": "user-provided-key"
}

3. Response Format

Return standard ChatCompletion format:

{
    "id": "chatcmpl-...",
    "object": "chat.completion",
    "created": 1234567890,
    "model": "gpt-5-2025-08-07",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "{\"answer\": \"4\", \"confidence\": 0.95}"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {...}
}

4. Key Backend Requirements

JSON Schema Validation: Validate and enforce the provided schema
Model Support: Route to appropriate LLM that supports structured outputs
Error Handling: Return appropriate errors for invalid schemas
API Key Routing: Extract api_key from request and route to appropriate provider
Claim Refinement: Support optional refine_claims and related parameters
CheckThat Features: Handle CheckThat-specific parameters like evaluation metrics

5. CORS Configuration

For same-host development, configure CORS to allow your frontend origin:

# Example CORS configuration (FastAPI/Flask)
from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=[
        "http://localhost:3000",  # React dev server
        "http://localhost:8080",  # Vue dev server
        "http://127.0.0.1:3000", # Alternative localhost
        "http://127.0.0.1:8080", # Alternative localhost
        "http://localhost:5173",  # Vite dev server
    ],
    allow_credentials=True,
    allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
    allow_headers=["*"],
)

For production, specify your actual domain:

allow_origins=[
    "https://yourdomain.com",
    "https://www.yourdomain.com",
    "https://api.yourdomain.com",  # If using separate API subdomain
]

Supported Models

The SDK provides access to all the latest models from multiple providers:

OpenAI: GPT-5, GPT-5 nano, o3, o4-mini, GPT-4o, and more
Anthropic: Claude Sonnet 4, Sonnet Opus 4.1, Claude 3.5 Sonnet, and more
Google: Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 1.5 Pro, and more
xAI: Grok 4, Grok 3, Grok 3 Mini, and more
Together AI: Llama 3.3 70B, Deepseek R1 Distill Llama 70B, and more

⚠️ Important: Always Check Available Models

Before using any model, query the /v1/models endpoint to get the current list of available models:

# Get the most up-to-date list of available models
models = client.models.list()
print("Available models:")
for provider in models["models_list"]:
    print(f"\n{provider['provider']}:")
    for model in provider["available_models"]:
        print(f"  - {model['name']} ({model['model_id']})")

Model availability depends on:

Your API keys and provider access
Current provider service status
Regional availability
Your subscription tier with each provider

The CheckThat AI platform stays up-to-date with the latest model releases from all supported providers.

API Reference

CheckThatAI Client

client = CheckThatAI(
    api_key="your-api-key",           # Required: Your API key (must be provided)
    base_url="https://api.checkthat-ai.com/v1",  # Optional: Custom base URL
    timeout=30.0,                     # Optional: Request timeout
    max_retries=3,                    # Optional: Max retry attempts
)

Chat Completions

Compatible with OpenAI's chat completions API, with additional CheckThat AI features:

# Regular chat completion
response = client.chat.completions.create(
    model="gpt-5-2025-08-07",  # Use latest models
    messages=[...],
    temperature=0.7,
    max_tokens=1000,
    stream=False,
    # ... other OpenAI parameters
)

# Structured output with parse() method
response = client.chat.completions.parse(
    model="claude-sonnet-4-20250514",  # Use latest available models
    messages=[...],
    response_format=YourPydanticModel,
    refine_claims=True,  # CheckThat AI enhancement
    refine_model="gpt-5-2025-08-07",  # Best model for refinement
    # ... other parameters
)

Model Information

# List all available models with details
models = client.models.list()
for provider_data in models["models_list"]:
    print(f"Provider: {provider_data['provider']}")
    for model in provider_data["available_models"]:
        print(f"  {model['name']}: {model['model_id']}")

# Get specific model information
model_info = client.models.retrieve("gpt-5-2025-08-07")
print(f"Model details: {model_info}")

sample response

{
  "models_list": [
    {
      "provider": "OpenAI",
      "available_models": [
        {
          "name": "GPT-5",
          "model_id": "gpt-5-2025-08-07"
        },
        {
          "name": "GPT-4o",
          "model_id": "gpt-4o-2024-11-20"
        },
        {
          "name": "o4-mini",
          "model_id": "o4-mini-2025-04-16"
        }
      ]
    },
    {
      "provider": "Anthropic", 
      "available_models": [
        {
          "name": "Claude Sonnet 4",
          "model_id": "claude-sonnet-4-20250514"
        },
        {
          "name": "Claude 3.7 Sonnet",
          "model_id": "claude-3-7-sonnet-20241022"
        }
      ]
    }
  ]
}

Authentication

CheckThat AI requires you to provide API keys for the LLM providers you want to use. Set your API keys as environment variables:

# Set the API key for your chosen provider
export OPENAI_API_KEY="your-openai-key"
export ANTHROPIC_API_KEY="your-anthropic-key" 
export GEMINI_API_KEY="your-gemini-key"
export XAI_API_KEY="your-xai-key"
export TOGETHER_API_KEY="your-together-key"

Then pass the appropriate API key to the client:

import os

# Use the API key for your chosen provider
api_key = os.getenv("OPENAI_API_KEY")  # or whichever provider you're using
client = CheckThatAI(api_key=api_key)

The SDK will route your requests to the appropriate LLM provider based on the model you select.

Error Handling

The SDK uses OpenAI-compatible exception types and adds CheckThat-specific errors:

from openai import OpenAIError, RateLimitError, APITimeoutError
from checkthat_ai._exceptions import InvalidModelError, InvalidResponseFormatError

try:
    response = client.chat.completions.create(
        model="gpt-5-2025-08-07",  # Use latest models
        messages=[{"role": "user", "content": "Hello!"}]
    )
except InvalidModelError as e:
    print(f"Invalid model: {e}")
    # List available models
    models = client.models.list()
    print("Available models:")
    for provider in models["models_list"]:
        print(f"{provider['provider']}: {len(provider['available_models'])} models")
except InvalidResponseFormatError as e:
    print(f"Response format error: {e}")
except RateLimitError:
    print("Rate limit exceeded")
except APITimeoutError:
    print("Request timed out")
except OpenAIError as e:
    print(f"API error: {e}")

Examples

For comprehensive examples demonstrating advanced features like claim refinement and evaluation metrics, check the examples/ directory:

enhanced_structured_output_example.py - Complete demo of all SDK features

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

📧 Email: kadapalanikhil@gmail.com
🐛 Issues: GitHub Issues
🌐 Website: checkthat-ai.com

Changelog

See CHANGELOG.md for a history of changes.

Name		Name	Last commit message	Last commit date
Latest commit History 72 Commits
.github/workflows		.github/workflows
checkthat_ai		checkthat_ai
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
pyproject.toml		pyproject.toml

License

checkthat-ai/checkthat-py

Folders and files

Latest commit

History

Repository files navigation

CheckThat AI Python SDK

Features

Installation

Quick Start

Basic Usage

Async Usage

Streaming Responses

Complete Working Example

Structured Output Generation

CheckThat AI Features

Claim Refinement

Available Evaluation Metrics

Alternative: JSON Schema Format

Structured Output Features

Backend Implementation Guide

1. Two Endpoints: /chat/completions

2. Structured Output Request Format

3. Response Format

4. Key Backend Requirements

5. CORS Configuration

Supported Models

⚠️ Important: Always Check Available Models

API Reference

CheckThatAI Client

Chat Completions

Model Information

Authentication

Error Handling

Examples

Contributing

License

Support

Changelog

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 2

Packages 0

Contributors 2

Uh oh!

Languages

1. Two Endpoints: `/chat/completions`

Packages