Question 1

What is Vachi?

Accepted Answer

Vachi is an LLM gateway for AI agents. It sits between your agent and the model provider (Anthropic, OpenAI, Google) and applies token distillation to every request before it reaches the model. The model still gets the same job and returns the same answer. The payload is smaller, so your model bill grows slower.

Question 2

What's Vachi doing that I can't do myself with better prompt engineering?

Accepted Answer

Distillation runs on every request automatically, in places hand-tuned prompts cannot reach: the long tail of agent traffic where context grows turn by turn, where tool definitions and retrieved snippets repeat, where the cost driver is not your prompt but the cumulative payload. We do not replace prompt engineering. We compose with it. Your prompts get smaller payloads to ride on.

Question 3

How is this different from prompt caching, semantic caching, or model routing?

Accepted Answer

Prompt caching reuses an exact-match prefix on the provider side. It is useful, and we sit on top of it. Semantic caching returns a guessed answer if a similar question was asked before. We do not do that. We never return a cached or guessed response. Model routing sends your request to a weaker, cheaper model. We do not do that either. The model you picked is the model that runs. Our claim is narrower than any of these: most of the tokens in a typical agent request are not load-bearing for the answer, and the right place to fix that is the payload, not the model.

Question 4

What is token distillation?

Accepted Answer

It runs on the request payload before the model sees it, not on the model itself. It is not model training and not model distillation. We weigh how much each token contributes to the next answer and rebuild the payload around the tokens that carry real weight. The bits we defer can be retrieved on demand if the model asks for them again later in the session. The model receives a leaner input and returns the same answer.

Question 5

How does distillation actually reduce my model burn?

Accepted Answer

The cost driver in agent traffic is input token volume, and most of those tokens are not useful for the next answer (repeated context, stale tool output, prior turns that have already been summarized). We condense the payload so the model only pays attention to what matters, and we shape the payload to improve the hit rate on the provider's own prompt cache. The model still does the same job. Fewer tokens go in, the same answer comes out.

Question 6

What is probabilistic caching?

Accepted Answer

It is a runtime optimization on the payload, not on the model's answer. When parts of a request look likely to repeat in the next call, Vachi pre-shapes the distilled payload so the next request is built faster. It never guesses, caches, or reuses a model response. Every request still goes to the model and comes back as a fresh answer.

Question 7

How does pricing work?

Accepted Answer

We take 33% of the savings we measurably generate on your traffic. No flat fee, no per-seat charge, no minimum. Bring your own key, cancel anytime.

Question 8

What happens in a month where Vachi does not save me money?

Accepted Answer

You owe us nothing for that month. That is the deal.

Question 9

How long does a POC take, and what does it cost me to try?

Accepted Answer

Setup is one base-URL change in your agent config. The first proof point shows up the moment you run a real workload through it. Our dashboard streams every request live and shows the token burn the model would have charged, the actual distilled burn, and a multiplier showing how much more work you got out of the same spend. You usually have a real signal inside the first coding session. The dollar cost is the model spend you would have paid anyway, plus our share of the savings we measure (zero in months we do not save you money).

Question 10

How do I verify the savings myself? Can I run a traffic split?

Accepted Answer

Yes. You can flip a distillation-off mode per request so the same payload reaches the model untouched, then compare the two requests side by side. You can also run the simpler version: send a fraction of your agent traffic through Vachi and the rest direct, and compare the bills at the end of the week. We will help you set up whichever shape fits your stack.

Question 11

What is the smallest meaningful pilot you have seen?

Accepted Answer

A single developer running a real coding session through their normal tool (Cursor IDE today) is enough to see a clean savings signal inside the first hour. Production agent traffic takes longer to read, because the savings curve depends on session length and how much context repeats. For a production read, a few days of normal traffic is usually enough.

Question 12

If Vachi is down or slow, how do I bypass without redeploying?

Accepted Answer

Swap your base URL back to the provider's endpoint and your agent is talking to the model directly again. That is the only change. Some teams keep both URLs behind a config flag for exactly this reason.

Question 13

How do you make sure distillation does not degrade my output?

Accepted Answer

Distillation operates on the parts of the payload that do not carry weight for the next answer. The bits we defer can be retrieved on demand if the model needs them later in the session. In practice we have not seen quality degrade. We are still cautious. We test on real workloads with human review before turning distillation on for your traffic, and we keep it off any request that does not look like a clean candidate.

Question 14

How do you measure quality before turning distillation on for my traffic?

Accepted Answer

Today the check is supervised. We replay a slice of your real workload through Vachi and read the outputs side by side with the direct-to-provider baseline, and we collect qualitative feedback from your operators during the pilot. We are working on a published eval suite using industry-standard benchmarks. For now the trust gate is your team's judgment on your own traffic, not a number we publish.

Question 15

Are tool calls, function calls, and structured outputs preserved end-to-end?

Accepted Answer

Yes. Distillation operates on the conversational payload, not on the structured pieces an agent depends on. Tool definitions, tool_use and tool_result blocks, function schemas, and structured-output JSON pass through unchanged.

Question 16

Do you ever touch my system prompt?

Accepted Answer

No. The system prompt is forwarded verbatim.

Question 17

What about streaming responses and multimodal (image, audio) inputs?

Accepted Answer

Streaming works. SSE responses stream back to your agent the same way the provider would deliver them. Multimodal inputs pass through to providers that accept them. Distillation does not touch the binary parts of a multimodal request.

Question 18

What is the impact at p95 / p99, not just average?

Accepted Answer

We do not publish a tail-latency number yet. We have not measured enough production traffic to put one in print and be willing to defend it. What we can say: the network hop Vachi adds is small, the payload that reaches the model is smaller, and the model has fewer input tokens to read before it starts generating. Over a session, the time we save on the model side often more than offsets the extra hop. Vachi is also engineered to minimize runtime overhead through predictive preprocessing. We will measure your tails during the POC and you can see them on the same dashboard we use.

Question 19

Where does Vachi run?

Accepted Answer

Our hosted gateway runs in US data centers today. We can stand up presence in another region if your traffic needs it, and we can also host inside a partner's cloud if that fits better. If you cannot have payloads transit a hosted gateway at all, see the VPC option in the security section.

Question 20

What happens at provider rate limits or 5xx errors?

Accepted Answer

On provider rate limits we honor the retry-after the provider returns and retry transparently for you, up to a bounded number of attempts. On other upstream errors we cannot recover from, we surface the original error code back to your agent so your existing retry and idempotency logic decides what to do. We never silently swallow a failure.

Question 21

Where does my data go? What touches the public internet?

Accepted Answer

With the hosted gateway, your agent calls Vachi over HTTPS, Vachi calls the provider over HTTPS, and the response streams back the same way. Vachi never stores your prompts or the model's responses, and we never train on your data. If you cannot have your payloads transit a third-party network at all, run Vachi inside your VPC (see below).

Question 22

Do you log prompts? Can I turn logging off?

Accepted Answer

No. We only log usage metrics (token counts, model, status code) and errors. We never log the request body or the model response. We do keep the last-seen payload in an ephemeral cache for a short TTL so the next likely request runs faster, the same way the provider's own prompt caching does. That cache expires on its own and is never used for training.

Question 23

We need to keep traffic inside our network. What does VPC deployment look like?

Accepted Answer

Vachi ships as a Docker image you run inside your own VPC. You own the container, you monitor the network pipes, and your model traffic never leaves your perimeter. We keep a thin control plane to push gateway updates and surface aggregate performance and error rates back to you. That channel carries operational telemetry only, never your payloads. We work through the specifics with your security team during the pilot.

Question 24

Do I have to rewrite any code?

Accepted Answer

No. Point your agent's base URL at the Vachi gateway (https://gateway.vachiai.com/v1), add your Vachi key, and your existing client code keeps working.

Question 25

Which models are supported?

Accepted Answer

All the frontier flagships and their sibling models: Anthropic (Claude Opus 4.8 and the rest of the Claude family), OpenAI (GPT 5.5 and the rest of the GPT family), Google (Gemini 3.1 Pro and the rest of the Gemini family).

Question 26

Which tools work today, which do not, and why?

Accepted Answer

Anything that lets you set a custom base URL works: Cursor IDE, Cline, Aider, OpenClaw, LangChain, LlamaIndex, and custom scripts that hit an OpenAI-compatible endpoint. Tools that lock you to a fixed provider list (WindSurf, Cursor CLI today) cannot point at Vachi at all. Tools that use a non-standard request shape can connect but get smaller savings because our optimization assumes the standard spec. Cursor IDE is where most of our users run today, and is the fastest place to see a clean signal.

Question 27

How does BYOK (bring your own key) work?

Accepted Answer

You add your Anthropic, OpenAI, or Google API key in the Vachi dashboard. When your agent makes a request, Vachi distills it and calls the provider using your key. The model bill stays on your provider account. Vachi only sees token counts and routing metadata.

Question 28

What happens if I have not added a provider key yet?

Accepted Answer

Vachi returns a clear error pointing at the missing key. It never silently swaps in a different model.

Question 29

How do you handle new model releases?

Accepted Answer

As soon as the provider ships a new model, you can name it in your requests. We do not gate access to new releases. You choose when to switch.

Frequently asked questions

Start here

How it works

Pricing

POC and verification

Quality

Latency

Data and security

Setup and integration

Still weighing it for your agents?