“Which model is the best?” We are asked this very often, and there is no single answer. Public rankings change every few months, every provider regularly announces a new version, and the model that tops a general benchmark is not necessarily the best one for classifying your support tickets or summarizing your contracts.

The better question is: which model does your job best, within your constraints? This article presents the criteria we use to decide, and explains why the ability to switch models matters almost as much as the initial choice.

The main model families

Without going into specific versions, which change too quickly to be quoted in a lasting article, today’s landscape is organized around a few families:

  • OpenAI models, available through an API or major clouds, with a range from very capable models to lighter, faster versions.
  • Anthropic’s Claude models, also offered in several sizes, through an API or cloud platforms.
  • Mistral models, from a European company offering models hosted in Europe as well as open models.
  • Open models, whose weights are published and which can be hosted on your own infrastructure or with a provider of your choice.

Each family usually offers a “top-tier” model and smaller ones. This choice of size is often more decisive for your project than the choice of provider.

The criteria that really matter

  • Quality on your task: a model can excel at complex reasoning and bring no gain on a simple classification task. What matters is its performance on your real cases, not its rank in a general leaderboard.
  • Latency: for an assistant used live by a customer or an agent, a few seconds of waiting make a real difference. For an overnight batch, it matters less.
  • Usage cost: models are billed by the volume of text processed. A large model can cost far more than a small one for the same job, and the gap grows with volume.
  • Context length: how much text the model can take into account at once. Useful for analyzing long documents, but a large context does not replace good document retrieval.
  • Confidentiality and hosting: contract terms, no reuse of data, hosting region, self-hosting options. For sensitive data, this criterion can rule out some options straight away.
  • Vendor lock-in: the more your system is built around one provider’s specifics, the more costly it will be to switch the day a better model, a better offer or a regulatory constraint requires it.
The best model is not the one that wins the rankings. It is the one that does your job, at the right quality, at the right cost, within your constraints.

Small or large model?

The reflex is often to take the most powerful model available, “to be safe”. It is rarely the right call. Many business tasks (classifying, extracting fields, rephrasing, summarizing a short text) are handled very well by smaller models that are faster and cheaper.

An effective approach is to combine several models in one system: a small model for simple, high-volume steps, and a large model reserved for steps that need deeper reasoning. That is often where the biggest cost and speed gains are found.

Test on your own cases

The only reliable way to choose is to compare candidates on a test set built from your real data: a few dozen to a few hundred representative examples, with expected answers, including hard cases and trick cases.

The same test set is run through each candidate model, with the same instructions, and you compare answer quality, response time and estimated cost at real volume. This takes a few days, but it replaces weeks of debate based on impressions or generic comparisons. And it will be reused every time a new model is released.

Keep the ability to switch

The model market moves fast: new versions come out regularly, prices change, some models are retired. A well-designed system must be able to change models without being rewritten.

  • An abstraction layer: the application calls an internal interface, which in turn calls the provider. Switching models then means changing a configuration, not the business code.
  • Portable instructions and output formats: avoid depending on features specific to a single provider when a standard alternative exists.
  • The test set as a safety net: before any model change, rerun the test set to check that quality is at least equivalent.

A concrete example

A services company wants to automate the analysis of requests received through a web form: identify the type of request, extract key information, then draft a first reply for the most complex cases. The first prototype uses a large model for everything.

The test set shows that classification and extraction are done just as well by a small model, faster and much cheaper at volume. The large model is kept only for drafting complex replies, which concern only part of the requests. Since some requests contain personal data, the company chooses contracted offers with European hosting for both steps. Everything goes through an abstraction layer: a few months later, a new, better model is tested and adopted in a day, without touching the rest of the system.

Our recommendation

Do not choose a model on its reputation: choose it on your data. Start by defining your non-negotiable constraints (confidentiality, hosting, latency), build a representative test set, compare two or three candidates, and pick the smallest model that reaches the required quality. Then build your system so you can switch easily: in this field, today’s right choice will probably not be next year’s.