A Compatibility Test Suite for Ollama's OpenAI-Compatible Endpoint

Pointing the OpenAI SDK at http://localhost:11434/v1 works on the first try, which is exactly the problem. The request succeeds, the response has the right shape, and two of the fields you sent were dropped on the floor without a word. A compatibility suite exists to find those before your code depends on one.

Why a 200 proves nothing

An OpenAI-compatible server has two honest ways to handle a field it does not support: reject the request with a 400, or accept it and ignore it. The second is far more common, because rejecting unknown fields breaks clients that send harmless extras, and because a server growing towards compatibility would rather serve a slightly wrong answer than none. The consequence for you is that “the request succeeded” carries no information about whether the field did anything.

So the assertion in every test here is never on the status code. It is on the consequence the field was supposed to have. If you send n: 3, the question is how many entries are in choices. If you send logprobs: true, the question is whether choices[0].logprobs is populated or null. If you send tool_choice with a forced function, the question is whether the message that comes back has tool_calls on it. Each of those is a property of the response body that a silently-ignored field cannot fake.

The fields Ollama does not implement

Ollama’s own documentation is the primary source here and it is unusually explicit — it publishes per-endpoint lists of supported and unsupported fields on its OpenAI compatibility page. At the time of writing it lists /v1/chat/completions as supporting model, messages, frequency_penalty, presence_penalty, response_format, seed, stop, stream, stream_options with include_usage, temperature, top_p, max_tokens, tools and reasoning_effort; and lists tool_choice, logit_bias, user, n and logprobs as unsupported.

The other endpoints have their own gaps. /v1/completions does not implement best_of, echo, logit_bias, user, n or logprobs. /v1/embeddings takes a string or an array of strings but not an array of tokens or an array of token arrays, and ignores user. That last one matters more than it looks: code that tokenises client-side and posts token ids — a normal optimisation against OpenAI — has no working path here at all, and finds out at runtime.

A probe returns three states, not two

Write each probe to return one of supported, ignored or rejected, and you get a test that is useful in both directions. A field that moves from ignored to supported is a feature you can now use; a field that moves from supported to rejected is an upgrade that broke you. A boolean collapses both into “the test failed” and tells you nothing about which.

// probe.ts
import OpenAI from "openai";

export type Support = "supported" | "ignored" | "rejected";

export const client = new OpenAI({
  baseURL: process.env.LLM_BASE_URL ?? "http://localhost:11434/v1",
  // Ollama does not check the key, but the SDK requires a non-empty one.
  apiKey: process.env.LLM_API_KEY ?? "ollama",
});

export const MODEL = process.env.LLM_MODEL ?? "llama3.2";

/** Run a request; if the server rejects it, report that instead of throwing. */
export async function probe<T>(
  run: () => Promise<T>,
  honoured: (result: T) => boolean,
): Promise<Support> {
  let result: T;
  try {
    result = await run();
  } catch (err) {
    if (err instanceof OpenAI.APIError && err.status === 400) return "rejected";
    throw err;
  }
  return honoured(result) ? "supported" : "ignored";
}

Note what probe does not swallow. A 500, a connection refused or a timeout still throws, because those are not answers about field support — they are the server being unavailable, and a suite that reports “everything is ignored” when Ollama is not running is a suite nobody trusts.

The predicate is where the thinking goes, and it has to be chosen so that a false negative is impossible rather than merely unlikely. A predicate of “the answer mentions the word I asked for” fails whenever the model has an off generation and reports a supported field as ignored, which is the worst outcome available: a suite that cries wolf gets a skip annotation within a fortnight. Prefer something structural — an array length, a null check, the presence of a key — and where no structural signal exists, do not write the probe at all. A missing column is honest; a flaky one is not.

The suite

Each case names the field, sends the smallest request that would demonstrate it, and supplies the predicate that decides whether the field did anything. Keep the generations tiny — max_tokens: 16 everywhere — because none of these assertions look at the text.

// compat.test.ts
import { describe, expect, it } from "vitest";
import { client, MODEL, probe } from "./probe";

const base = {
  model: MODEL,
  messages: [{ role: "user" as const, content: "Say hi." }],
  max_tokens: 16,
};

describe("ollama /v1/chat/completions", () => {
  it("ignores n", async () => {
    const state = await probe(
      () => client.chat.completions.create({ ...base, n: 3 }),
      (r) => r.choices.length === 3,
    );
    expect(state).toBe("ignored");
  });

  it("ignores logprobs", async () => {
    const state = await probe(
      () => client.chat.completions.create({ ...base, logprobs: true }),
      (r) => r.choices[0].logprobs != null,
    );
    expect(state).toBe("ignored");
  });

  it("honours stop", async () => {
    const state = await probe(
      () =>
        client.chat.completions.create({
          ...base,
          messages: [{ role: "user", content: "Count: one two three four" }],
          stop: ["three"],
        }),
      (r) => !(r.choices[0].message.content ?? "").includes("three"),
    );
    expect(state).toBe("supported");
  });

  it("reports usage when asked", async () => {
    const stream = await client.chat.completions.create({
      ...base,
      stream: true,
      stream_options: { include_usage: true },
    });
    let usage = null;
    for await (const chunk of stream) usage = chunk.usage ?? usage;
    expect(usage?.completion_tokens).toBeGreaterThan(0);
  });
});

The stop case is the one worth copying as a pattern. It is not asserting that the model said anything in particular; it is asserting that a token the model was heading towards is absent. That is a property of the sampler and the server, not of the prose, which is what makes it a stable assertion against a non-deterministic backend. The same trick covers max_tokens — assert that usage.completion_tokens is at or below the cap and that finish_reason is "length", never that the text ends anywhere in particular.

Running it against a version bump

1. Pull the model the suite uses ahead of time — ollama pull llama3.2 — because a first request that downloads several gigabytes will blow any per-test timeout you have set.
2. Pin the Ollama version in whatever starts it, so the suite is answering a question about a known build rather than about whatever was on the machine that morning. Running it under Docker Compose alongside the app is the cleanest way to hold that pin; see the compose setup for an offline test loop.
3. Run the suite as a separate job from your unit tests, with its own timeout. It is the only part of your test run that needs a model resident in memory.
4. When you bump Ollama, run this suite first and read the diff before you read the changelog. A field moving from ignored to supported is the interesting result, and it is the one a changelog is most likely to bury.

The same pattern generalises to every other compatible endpoint you talk to — vLLM, LM Studio, llama.cpp’s server, a hosted provider advertising an OpenAI-shaped API. Once you have more than one of them, turn the suite into a matrix so the answer for every provider lives in one committed file.