brainbaby

Baby 5.0 benchmarks: measured, not borrowed

Dated measurements of the current Baby 5.0 serving configuration. A small internal test set, with the settings, results and failures available to inspect.

What we ran

  • 30 tasks, two passes, two configurations

    On October 19, 2026, we ran 30 fixed fictional tasks twice in each of two configurations: 120 API requests in total. The categories are reasoning, code reading, structured data, context and instructions, and tool-call formatting. The corpus was fixed before the first request and stayed unchanged between configurations.

  • The current serving configuration

    Both runs used Baby 5.0’s current text-model and pinned FP4 provider configuration, with provider fallback disabled, streaming enabled, low reasoning effort, temperature 0 and a 1,536-token output cap. Requests were serial, with no retries. This is a controlled serving-model test, rather than a call through the full Brainbaby application.

  • Strict, automatic grading

    Text checks require valid JSON with exactly the expected values and no additional fields or prose. Tool checks require exactly one call with the expected function and arguments. A stream error, incomplete response or unverified serving configuration fails the check. No real connector was executed and no generated code was run.

The result, including the failures

  • Structured-output configuration

    Explicit JSON object mode on all text tasks passed 60/60 strict checks. This is success on this small, fixed corpus; it is not a general accuracy score or an independent ranking.

  • Baseline configuration

    The baseline passed 37/60 strict checks. Its 23 failures were invalid JSON, including Markdown wrappers and additional explanations. We kept those outputs and failures in the report. In the baseline, only the six structured-data cases used JSON object mode; other text cases requested JSON through the prompt alone.

  • What this tells us

    For code that needs to parse a machine-readable answer, the output format is part of correctness. Explicit structured-output controls performed better here than a prompt-only formatting instruction. Ordinary conversation should still use natural language; these results do not justify forcing every chat response into JSON.

How to read the timing and cost

  • Measured response times

    With structured-output mode, median first text was 0.75 seconds and p95 was 1.01 seconds across 48 text responses. Median completion was 0.78 seconds and p95 was 1.03 seconds across 60 completed streams.

  • First text and completion are separate

    First-text time starts just before the API request and ends when the test runner receives non-whitespace answer text. It includes all 48 completed, error-free text responses per configuration, even if their strict format check failed. The 12 tool-call samples do not count toward first-text time. Completion covers all 60 completed streams per configuration. p95 uses the nearest-rank method.

  • A point-in-time API measurement

    The two serial passes are not controlled cold-start and warm-start experiments. The tests exclude application authentication, memory, database access, connected services, rendering and the mobile network. Longer answers, larger contexts and real tools can take longer. These results are not an uptime or future-speed guarantee.

  • Reported inference cost

    The structured-output run reported $0.00142 for 60 requests. Cost is the sum of the usage.cost fields received from the serving API, including its reported token accounting. It covers this short test set, not hosting, storage, payment fees, support or the full cost of serving a customer. It is not a profit calculation.

What remains separate

  • Complete app workflows

    Uploaded PDFs, image editing, repository changes, real tool actions, saved conversations, billing and cross-device continuity need their own end-to-end tests. Code-reading checks do not establish repository engineering ability. Tool-call formatting does not prove a Gmail or other connector action works.

  • Comparisons and replication

    This is a small internal diagnostic, not an independent academic benchmark or a head-to-head comparison with another product. The report includes every fictional prompt, expected answer, output, grade, timing and inference cost. Match the corpus, output settings and serving profile when repeating a test.

Sources