Pydantic evals library review

15-08-2025

• https://github.com/pydantic/genai-prices
• toml definition - https://github.com/pydantic/genai-prices/blob/main/packages/python/pyproject.toml
• simple high-level api with types + overloading + imports - https://github.com/pydantic/genai-prices/blob/main/packages/python/genai_prices/init.py
• simple cli no frills but functional and keeps the import flow consistent - https://github.com/pydantic/genai-prices/blob/main/packages/python/genai_prices/_cli.py
• handling data caching / updates on demand - https://github.com/pydantic/genai-prices/blob/main/packages/python/genai_prices/update_prices.py
• defining types in once place to be used throughout, quite like this, minimal implementation details - https://github.com/pydantic/genai-prices/blob/main/packages/python/genai_prices/types.py
• handling auto-generated files within a package, something we do ourselves - https://github.com/pydantic/genai-prices/blob/main/packages/python/genai_prices/data.py

Example with openai and the responses endpoint:

• https://platform.openai.com/docs/api-reference/responses/object#responses/object-usage
• https://github.com/pydantic/genai-prices/blob/main/packages/python/genai_prices/data.py#L1702

response_data = {
    "model": "gpt-5",
    "usage": {"input_tokens": 1000, "input_tokens_details": {"cached_tokens": 133300}, "output_tokens": 100},
}
extracted_usage = extract_usage(response_data, provider_id="openai", api_flavor="responses")
price = extracted_usage.calc_price()
print(price.total_price)