LLMBDA
A decision layer between an application and its model providers. It picks how each request should actually run.
Almost nobody decides how a request should run. They decide once, then run everything that way.
A team wires up a RAG pipeline, picks the strongest model they can get, and sends it the full retrieved context on every call. It works, so it ships. What that hides is that most requests didn’t need that model, and almost none needed that much context. The retrieved chunks hold the answer plus a few thousand tokens that have nothing to do with the question, and you pay for all of it, on every call.
The existing tools each cover one slice. Routers pick where inference runs but take the prompt as given. Prompt optimizers tune a prompt once, offline, then freeze it. Providers optimize inside their own stack and have no reason to send you elsewhere. Nobody owns the actual decision: which model, which prompt, and which strategy for this request.
Treat execution as a decision, and the decision as something you can learn.
LLMBDA catches the request before it reaches a provider and picks a configuration: how hard to compress the context, whether to restructure it, and which model to send it to. Formally it’s a joint scoring policy over compression × refinement × target model, set up as an offline contextual bandit. The context is the workload signature, the arms are the configurations, and the reward is quality with a cost penalty.
Offline matters here. Learning on live traffic means exploring on real users, and a bad arm is a wrong answer that someone acted on. So the policy is fit against logged outcomes instead. Exploration happens on recorded traffic, and a policy only gets promoted once it clears the quality target. Compression goes through a fine-tuned LLMLingua token classifier that drops low-information tokens and keeps 99% accuracy on CRAG.
Cheaper and better, on the same task.
The benchmark is a policy-eligibility decision over noisy retrieved context: exact clauses to cite, a numeric cost share to compute, strict JSON out. Twelve runs per arm, same judge.
| METRIC | RAW + PRO | LLMBDA + FLASH | DELTA |
|---|---|---|---|
| inference cost | $0.006346 | $0.001585 | −75% |
| judged quality | 34 / 40 | 40 / 40 | +6 |
| latency | 18,102 ms | 16,496 ms | −1,606 ms |
| input tokens | 1,217 | 1,034 | −15% |
| context compression | — | 34% | — |
The quality number surprised me. I expected compression to cost a little accuracy and buy a lot of savings. Instead the judged score went up, from 34/40 to 40/40. My best read is that stripping the irrelevant material out of the retrieved context removed distractors the larger model kept getting pulled toward. The smaller model got a cleaner problem and got it right more often.
One task family, one provider, twelve runs per arm. That’s enough to show the mechanism works and nowhere near enough to call it a general result. Before I’d claim more, I’d want: