What follows is the account of a small research team that nearly wasted its grant on the wrong quantum simulation software — and the selection process that saved it. The team is fictional; the mistakes are not. Anyone who has watched a lab, startup, or university group buy tooling for emerging tech will recognise every scene. Names aside, treat it as a field report.
A grant, a deadline, and a demo
Call them Qubit Lane: five people — two physicists, two engineers, one PhD student named Mara who ended up the hero — with eighteen months of funding to prototype optimisation experiments on simulated hardware. Week one, they needed a simulator platform. Real quantum access was budgeted for later; everything until then would run on classical simulation, so the choice mattered enormously. Pick wrong, and every benchmark, every noise model, every result for a year sits on a cracked foundation.
The obvious candidate arrived by demo. A vendor showed a slick platform: circuit visualiser, noise simulation, a claimed 40-qubit statevector capability, GPU acceleration, cloud scaling. The team lead, Tomas, was ready to sign by the end of the call. “It does everything we need.” The grant office needed a decision in three weeks anyway.
Mara asked one question that changed the timeline: “Which of those features did we actually see run?” Silence. They had seen the visualiser. Everything else had been slides.
Three weeks of doing it properly
Mara proposed spending the three weeks they had, rather than deciding in one afternoon. Her plan borrowed from an unlikely place. Before her PhD she’d worked in software procurement, where she’d absorbed the discipline of evidence-first review platforms — she’d point colleagues at BestAIToolix, which profiles software against category-specific scoring criteria, mostly to show them what honest evaluation output looks like: facts carrying sources and verification labels, missing data openly displayed as unknown, and no pretence that one global “best” exists across unlike categories. “We’re going to build that kind of profile ourselves,” she said, “for exactly one category: quantum circuit simulators.”
The category boundary did immediate work. One shortlisted product was really a full quantum development environment; another was a hardware-access broker with a simulator bolted on. Comparing them to a dedicated simulator was, as Mara put it, grading a truck against two sports cars because all three have wheels. The shortlist settled at three genuine simulators: the demo vendor, an open-source framework, and a smaller commercial engine.
Then the grid. Down the side, the criteria that mattered to their actual experiments: maximum qubits at their memory budget, noise-model fidelity controls, batch execution, reproducible seeds, export formats, licence cost against the grant, and the odds the project would still be maintained in two years. Across the top, the three candidates. Every cell got one of four stamps: tested by us, stated in docs, claimed in marketing, or unknown. The rule Mara enforced without mercy: marketing stamps score zero until upgraded, and unknowns stay visible on the final slide. No silent optimism.
What the testing actually found
The team gave each simulator the same three jobs, drawn from their real research plan: a QAOA run on a max-cut instance of the kind described in our piece on quantum approaches to stubborn optimisation workloads, a noisy variational circuit resembling the quantum machine-learning workloads covered in how quantum computing reshapes machine learning, and a brute 30-qubit statevector stress test on their own workstation.
Results, in the dry language of the grid. The demo vendor’s “40 qubits” was technically true — on a cloud configuration costing more per month than the team’s entire tooling budget; on local hardware it ceiled at 26. Its noise models were excellent, verified. The open-source framework handled 30 qubits locally, had the best reproducibility controls, but its batch execution needed engineering effort the physicists couldn’t give — a real cost, written into the grid as hours. The small commercial engine was fastest by a wide margin and cheapest, but two cells stayed stubbornly unknown: long-term maintenance (a four-person company) and an export format question their support channel never answered. Those unknowns were findings. A vendor who doesn’t answer during courtship will not improve after the wedding.
The three weeks also produced friction worth reporting honestly. Tomas pushed back hard in week two — the testing was eating evenings, the demo vendor kept calling with discount deadlines, and “we already know the answer” got said more than once. What held the process together was a deal Mara cut at the start: if the grid confirmed the demo vendor by the deadline, she would write the purchase request herself and never mention the detour. Scepticism with a defined end date is much easier to sell than scepticism as a lifestyle. The discount deadline, incidentally, turned out to be renewable; week three brought a fresh “final” offer, which told the team something about how much negotiating room had been sitting behind the first one.
The decision, and what made it defensible
They chose the open-source framework, and bought a support contract with part of the money saved — converting its weakness, unsupported engineering effort, into a priced line item. The demo vendor lost not because the product was bad but because its headline claim dissolved under their actual budget. The small engine lost on unknowns alone, a decision rule the team had agreed before testing: more than one unknown on a survival-critical criterion disqualifies.
Tomas presented the grid to the grant office. The reviewers, veterans of a hundred hand-wavy tooling justifications, approved it in a day. One wrote back a sentence the team framed: “First time anyone has shown us what they don’t know.” The grid went into the project repository next to the research plan, with a note to revisit it at month twelve — because simulators ship releases, small vendors get acquired, and a decision that was right in month one deserves to be re-checked before renewal money moves.
What Qubit Lane would tell your team
First: in emerging tech, marketing outruns reality by a wider margin than anywhere else, because so few buyers can verify claims. That makes the verified-versus-claimed stamp more valuable here, not less. Second: define the category before comparing anything; half their early confusion vanished when the truck left the sports-car race. Third: unknowns are data. Writing them down is uncomfortable and quietly powerful — it redirects the final debate from charisma to evidence. Fourth: test on your own problems and your own hardware. Vendor benchmarks are rehearsed theatre; your workload is the only audience that matters.
And last: the method transfers. Six months later the same grid template chose their experiment-orchestration stack, a decision that sits closer to classical infrastructure — the territory mapped in a beginner’s roadmap to platform engineering — and it worked unchanged. Categories, criteria, stamps, visible unknowns. The tools will keep changing, in quantum faster than anywhere. The habit of demanding evidence before money is the one piece of lab equipment that never becomes obsolete.
