HiveCache — Semantic Caching

HiveCache is ModelHive's proprietary dual-embedding semantic caching engine. Instead of matching requests by exact text, HiveCache understands the meaning behind each prompt — so similar questions get instant cached responses, dramatically cutting costs and latency.

Why HiveCache?

Without Cache	With HiveCache
Every request hits the LLM provider	Similar requests are served instantly
Full cost per request	Cached responses cost a fraction of a live LLM call
1-3s latency per call	< 5ms for cache hits
No learning from past requests	Gets smarter over time

How It Works

1. You send a request — Your app calls the ModelHive API as usual
2. Dual-embedding analysis — HiveCache generates embeddings for both the query (last user message) and the conversation context, then computes a DualScore
3. Smart matching — If a semantically similar request was already answered, the cached response is returned instantly
4. Transparent — Cache hits are flagged in response headers so you always know what happened

The scoring formula combines query similarity and context similarity:

DualScore = α × cos(query_a, query_b) + β × cos(context_a, context_b)

Where α = 0.7 (query weight) and β = 0.3 (context weight) by default.

Intelligent Routing Bands

HiveCache doesn't just do "hit or miss" — it uses a multi-band system based on semantic similarity scores:

Band	Score Threshold	What Happens	Latency
DIRECT	≥ 0.95	Exact or near-exact match — cached response returned verbatim	~3ms
REUSE	≥ 0.90	Meaning is the same, wording differs — cached response returned as-is	~5ms
TWEAK	≥ 0.85	Similar but different — adapted via a cheap LLM (cost-benefit gated)	~500ms
MISS	< 0.85	New topic — full LLM generation, result cached for future use	1-3s

TWEAK Band Cost-Benefit Analysis

The TWEAK band performs a real-time cost-benefit analysis using actual provider pricing. It only runs the adaptation model if the savings from reusing the cached response exceed the cost of the tweak call.

Request Headers

Control HiveCache behavior with request headers:

Header	Values	Behavior
x-hive-cache	"false"	Skip cache lookup (still stores the response for future hits)
x-hive-cache-meta	any string	Opaque client metadata attached to the cached entry (for tracing/benchmarking)

Response Headers

Every response includes HiveCache status headers:

Header	Example	Description
x-hivecache-status	DIRECT	Band classification: DIRECT, REUSE, TWEAK, or MISS
x-hivecache-score	0.9632	The DualScore similarity value
x-hivecache-tokens-saved	1847	Number of tokens saved by serving from cache
x-hivecache-matched-meta	req-abc123	The x-hive-cache-meta of the matched cached entry

Usage Examples

Python

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-modelhive-key",
    base_url="https://api.modelhive.ai/v1"
)

# Normal request — HiveCache is active by default
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is machine learning?"}]
)

# Check cache status from response headers (via httpx)
# x-hivecache-status: DIRECT | REUSE | TWEAK | MISS

cURL

# Standard request — cache active by default
curl -X POST https://api.modelhive.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-your-modelhive-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "What is machine learning?"}]
  }'

# Response includes:
# x-hivecache-status: REUSE
# x-hivecache-score: 0.9412
# x-hivecache-tokens-saved: 523

Bypass Cache

# Force a fresh LLM response (result still gets cached)
curl -X POST https://api.modelhive.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-your-modelhive-key" \
  -H "Content-Type: application/json" \
  -H "x-hive-cache: false" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "What is machine learning?"}]
  }'

With Metadata

# Attach tracing metadata for cache analytics
curl -X POST https://api.modelhive.ai/v1/chat/completions \
  -H "Authorization: Bearer sk-your-modelhive-key" \
  -H "Content-Type: application/json" \
  -H "x-hive-cache-meta: session-42-turn-7" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "What is machine learning?"}]
  }'

Cache Bypass Conditions

HiveCache automatically skips caching in these cases:

Condition	Reason
x-hive-cache: false header	Explicit opt-out
Request contains tools field	Tool/agent responses are context-dependent
Team has cache disabled	Per-tenant CacheEnabled = false
Non-chat endpoints	Only /v1/chat/completions and /v1/messages are cached

Supported Formats

HiveCache works transparently with both OpenAI and Anthropic request formats:

Endpoint	Format
POST /v1/chat/completions	OpenAI-compatible
POST /v1/messages	Anthropic Messages API

Cross-Format Isolation

Anthropic models are namespaced internally (e.g., @anthropic/claude-3.5-sonnet) to prevent cache collisions between OpenAI and Anthropic format responses.

Streaming Support

HiveCache fully supports streaming ("stream": true). Cached responses are replayed as Server-Sent Events (SSE) exactly as the original stream was received — your streaming client code works identically for cache hits and misses.

Cache Analytics API

Monitor cache performance programmatically:

Method	Endpoint	Description
GET	/api/v1/cache/stats	Entry count, memory usage, status
GET	/api/v1/cache/metrics?limit=100	Per-request metrics (band, score, tokens saved, latency)
POST	/api/v1/cache/flush	Flush all cache entries

Cost Sharing

When HiveCache saves money by serving a cached response instead of calling the LLM, you get a share of those savings credited back to your wallet. The exact percentage depends on your subscription tier.

tip

HiveCache is most effective for applications with repetitive query patterns — customer support bots, FAQ systems, code assistants, and content generation workflows typically see the highest hit rates.