Bayesian optimization: can LLMs pigeonhole experimental scientists?

Scientists increasingly ask chatbots what to try next, but it’s unclear what those suggestions are based on. I tested this by asking Claude Opus 5.5 for its next three experiments on four measured chemistry datasets. Rewording the request barely changed the picks beyond ordinary repetition, while changing the names or scrambling the results changed them about three times as much. Names mattered most early on, and results took over as data accumulated. Telling the model to rely on the data changed its picks without improving them. This is a work-in-progress preprint on one model with no human participants. Next steps are a head-to-head against Bayesian optimization and a study with real scientists.

See the code →