Small changes worth making

The first thing is that if you were using a large model on a low effort setting, you owe it to yourself to try Luna on max. It is just so efficient for any task that you can determine doesn’t need maximum intelligence. If you already have identified those tasks by setting your effort to low, swap that model out instead.

Another easy change is to just ask more questions. This is the Jevons Paradox in action - improvements in technology immediately lead to more applications. That could look like testing 5 hypotheses at once before hearing back from an agent, but with models this fast, iterating into deeper levels of detail becomes far easier. Maybe you pre-fetch answers to relevant customer questions rather than wait for them to pick from a list first.

Shifting our approach

Once AI is fast, the ROI of a faster and lower latency data platform jumps. In agentic workflows, the request to the LLM has long been the total time bottleneck. If one turn of the agent took 10s of seconds, the benefits of a faster database just weren’t impactful.

Analytical databases can be 1000x faster than transactional ones if your workload is analytically shaped. Many agent questions are. With this new revision of Luna, your transactional database just became your user experience bottleneck.

Likewise, if your analytical store takes 30 seconds to spin up, a low latency DB could have answered 10 agent questions in that time. Agent workloads are bursty, so a low latency serverless approach makes sense. Especially if you are designing customer facing agentic experiences, you’ll feel the difference with a responsive analytical engine now. What new product features or even whole businesses are possible once an agent can provide data driven answers at this speed?

The models are still only as good as the context they’re given. Building a context layer with all the details of your specific business or domain remains incredibly high leverage. Now though, it pays dividends to be more detailed in that context so that a weaker model can interpret it. When weak models couldn’t write accurate SQL, context just needed to be good enough for the strongest of models. Putting in that extra time documenting your domain can slash costs and speed up answers.

Your data team needs more evals

In the data world, historically our tests were data quality checks. Often we only check if easily computable invariants hold (no duplicate customer ids, no NULL order prices, every order joins to a real product id). For agentic analysis tasks, we need more than just SQL checks. We need natural language questions and the correct answer based on the data. We then evaluate if an agent can take the question and use business context and a database connection to answer it correctly.

Running those evals just got 5x cheaper.

One natural opportunity is to use those savings to run evals far more often. We can explore how each new model performs and even tune settings within models, where before perhaps we accepted the defaults from a single lab.

We could even catch cases where model intelligence fluctuates. Model performance is a function of the underlying model weights, but also the infrastructure used to serve it. As capacity gets tight, model intelligence is reduced, and a well calibrated eval system could catch that. Your business could jump to a more optimal model, whether that is in the days before a new model release or during the highest traffic times of the day.

The model is only one factor though. It is valuable to measure how each new piece of organization context helps (or hurts!). OpenAI’s own data team found that catching regressions in their context layer was a huge value of their eval framework. Running evals weekly just won’t provide enough signal to build accurate organizational knowledge.

The business itself is constantly changing too! If you add a new discount program, your agents may not be able to deduce how to calculate revenue correctly anymore. Catch those logical bugs before you share that graph to the board!

Another truism is that we need to design our eval workflows to be frontier lab agnostic. We use OpenRouter in a custom harness for our evals so that within hours of launch we can run any of the latest models from all the frontier labs. We get the most benefit in this modern AI economy if we can develop intellectual property at the harness or context layers and treat the model as a commodity. By keeping switching costs low and avoiding lock-in, we can jump to the next top model as soon as it provides a return on investment.