Ten summaries for 372 results

On Tuesday evening OpenAI put 722 mathematics manuscripts on GitHub. The openai/math repository, created at 21:47 UTC according to GitHub's records and released at 6 p.m. Eastern, according to Scientific American, groups them into 372 "families" of related papers, each classed by discipline. The README says they were produced by an internal OpenAI model during evaluations on open research problems, which the company expanded after its models saturated its existing maths tests. By Wednesday the release was drawing concern from mathematicians, The Guardian reported.

There is a published standard to measure it against. On 29 September the Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, the body OpenAI said last month it would consult, issued recommendations on releasing AI-generated mathematics, drawn up after more than 600 replies from mathematicians. A footnote says the survey asked about a case in which OpenAI announced many results without details. The document opens with an objection to the whole enterprise: "We want to state clearly from the start: we do not endorse this practice, and we ask them to stop testing advanced mathematical problems on proprietary models." An OpenAI spokesperson told Scientific American the company was taking the guidelines seriously and doing its best to comply, but was not bound by them.

The group's third recommendation is the most specific. "For each result released, the AI lab should make public the name of the model, the prompts used, a (summarized) chain of thought, the time taken, and the estimated cost of computation." The repository does not name the model; the README describes it only as unreleased and internal. It publishes no prompts, which Scientific American also noted. It includes abridged reasoning summaries for ten of the 372 families, among them the irrationality exponent of pi, the Mahler conjectures and the isomorphism of free group factors. On time and cost it gives one average for all of them: "On average, each result used three hours of ChatGPT Pro thinking compute with that model." Scientific American reported that OpenAI said almost every result came from a single prompt to a single agent, and that some may have taken several attempts. Andrew Sutherland of MIT was unconvinced. "Until and unless they release the model and people can replicate their results, I think you should treat any claims about one-shotting problems with a single agent as unverified," he told the magazine. "We should ask for receipts."

The fifth recommendation covers exactly this kind of mass release: "If many results are released at once, then in addition to the results themselves a further document should be written and made public that references all of the released results and explains how many other problems of comparable difficulty the models tried and failed to solve, as well as how the problems were chosen." The README gives one number for the denominator. "Over the course of the evaluation, the model was posed approximately 4,000 problems." It does not say how they were chosen or how many went unsolved, and describes the filter in one sentence: "Aggregating the output into result families and manuscripts and requiring an appropriate level of significance led to the catalog outlined above." The README names two exceptions to that procedure: a zero-free region for the Riemann zeta function, whose write-up was edited by humans for readability, and a proof of the Hodge conjecture for CM abelian varieties.

On formal proof, the group asked that proofs be formalised as far as possible, with the formalisation status stated clearly where that would cause unacceptable delays. The repository's own formalisation catalogue lists 162 papers with a formalised main result, about 22 per cent of the 722 manuscripts by our count, marks its scope as "Partial progress" and gives its review status as "unchecked". The README is candid about the rest. "Some of the unformalized results could have issues. We will endeavor to fix any such issues quickly." Where a Lean proof exists, Scientific American noted, the result is all but certain to be correct; whether it is new, or a recombination of known techniques, will take mathematicians months to establish.

Then there is where the papers live. The advisory group wants results deposited in scholarly repositories, and is specific about which: "These should not be controlled by any AI lab and should guarantee certain standards, including that submissions have a persistent citable identifier and that subsequent modifications are appropriately recorded." The manuscripts sit in OpenAI's own GitHub account. The README promises to keep the release history, record corrections as new versions and supply a BibTeX entry for each paper, and says: "We are also exploring community-hosted repositories for these materials." The obvious community host is already rationing. arXiv limited each submitter to two papers a month from 1 October after a record September.

Not every mathematician wants the results held back. "If we want to know the answers to these math questions, I see no reason why we should ask the company to keep them secret from us," Daniel Litt of the University of Toronto told Scientific American. "To me, it's going to be a good thing for mathematics." Others object to the channel as much as the content. "Math by tweet and math by press release to me is not the way to nurture the ecosystem that created the fertile ground that they have trained on," Bryna Kra of Northwestern told WIRED before the release. Mathematicians have set up new venues this year, WIRED reported, including Hexagon, a repository mainly for AI-generated material, and Palomar, a registry of machine-verified mathematics; OpenAI has been encouraged to use them, and Kra said it had not changed its behaviour.

The deeper complaint is about access. "The use of proprietary internal models by AI labs to do mathematical research risks creating a two-tier system where labs outrun the rest of the field, effectively alienating the mathematical community from its own discipline," the advisory group wrote. OpenAI told Scientific American it is working to release the model as quickly and responsibly as possible, and that many of the new results are not yet understood by its own mathematicians. Until the model is out, the 722 papers are checkable only one by one, by people who cannot ask the system that wrote them how it got there, beyond the ten summaries it chose to publish.