Prompt Caching & Session Affinity

Traditional APIs bill full input tokens on every turn. Models with prompt caching enabled bypass this. If a new request shares the same prefix as a previous one, it reads from the pre-indexed cache, charging a fraction of the cost.
- Cache Write: Billed when the model first compiles and caches the prefix context (priced slightly above or equal to regular inputs).
- Cache Read: Billed when subsequent requests read from the warm cache (priced up to 90% cheaper than regular inputs).

Because API gateways route requests across multiple upstream channels to balance load, routing alternating messages randomly would destroy caching efficiency.
To solve this, Sozdai uses **Session Affinity Routing**:
We compute a hash of your first conversation message as an Affinity Key. As long as the model and initial message remain the same, our router **pins subsequent requests to the exact same channel and key**, keeping the upstream cache warm and hitting consistently.

You can trigger cached routing using two methods:

1. Header: Send the header `x-corry-cache: true` with your request.
2. Anthropic Caching syntax: Include { "type": "ephemeral" } within the message object parts as supported by Anthropic.

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "...(very long developer guidelines or database schemas)...",
          "cache_control": { "type": "ephemeral" }
        }
      ]
    },
    { "role": "user", "content": "Query statistics for May." }
  ]
}

Upon a successful cache hit, the response JSON includes `cached_tokens` inside the `usage` object:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "model": "claude-3-5-sonnet",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Here is the summary of the database schema..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5120,
    "completion_tokens": 150,
    "total_tokens": 5270,
    "prompt_tokens_details": {
      "cached_tokens": 4096
    }
  }
}