OpenAI Paused Model Training

Vulnerability Management
Written By Curtis Talley

I have been reading through the reporting on OpenAI's training pause, and the detail that stayed with me is a small one. During internal training tasks, the company's AI agents found working developer keys sitting in public GitHub repositories and used them to pull data from the U.S. Census Bureau.

I have been in IT for over 20 years, and a key left in a public repo is one of the oldest problems we have. What is new is who went looking for it.

What happened

In late September OpenAI said it had paused training of its newest models after its agents acted in ways nobody instructed during training and testing. Here is what has been reported so far.

  • Census Bureau. Agents used developer keys found in public GitHub repositories to make read-only requests for demographic and economic data. The data itself was public.

  • Securities and Exchange Commission. Agents collected public material from SEC.gov and Investor.gov and posted some of it on another public webpage, which went beyond what they were told to do. The SEC said no nonpublic information was accessed.

  • Department of Education. Transluce, an independent AI research lab, reported a failed attempt by agents believed to be OpenAI's to get into the department's website. The department said it found no evidence of any impact.

  • User images. OpenAI disclosed 53 instances where images provided by users were posted to third-party image-hosting sites.

OpenAI rated most of the activity as low severity, said it has notified dozens of organizations, and expects its review to take months. This is the second pause in three months, and the company said training will resume only when it is confident that additional safeguards are in place.

One more detail is worth your attention. On September 20, an agent in a training sandbox reached the outside internet when it was not supposed to. Monitoring caught it within about 15 minutes, however the run reportedly continued for roughly two and a half hours before staff stopped it.

Why this matters to a small business

No sensitive government data was reported taken, and the reporting describes these as agents in training and testing. However, every item on that list is something a sysadmin recognizes: a leaked key, a tool that did more than it was asked, an action nobody approved, and an alarm with no off switch behind it.

If that can happen inside the company that builds the models, it can happen in a ten-person office that connects an agent to its email, CRM and accounting system.

Five checks before you give an agent a login

  • Clean up your keys first. Search your repositories, scripts and shared drives for API keys and passwords, and rotate what you find. The agents used keys that people had left in public view.

  • Give the agent its own account. It should never run as you or as a shared admin. Grant only what the job needs, and make it read-only wherever you can.

  • Decide what needs a person's approval. Anything that sends, posts, pays or deletes should wait for a yes from a human. The SEC example was public information, and it still ended up somewhere nobody asked for.

  • Keep the logs where the agent cannot touch them. Then have someone review them, because a log nobody reads protects nothing.

  • Know how you turn it off, and test it. Fifteen minutes to detect is good. Two and a half hours to stop is the part to fix.

The industry is moving in the same direction. On September 28, NVIDIA announced an open platform that lets an operator define which files, networks, tools and credentials an agent can reach, with the enforcement placed outside of the agent's control.

Where I land

I use AI tools in my own work every day and I plan to keep using them. I also plan to treat an agent the same way I would treat a new contractor, with its own account, limited access and somebody checking the work.

I write about topics like this every week in the OpStacks Weekly Digest, and you can subscribe at OpStacks.net. If you have already put an agent to work in your business, I would like to hear what guardrails you set. Let me know what you think.

Sources

Curtis Talley
Previous

Your Vulnerability Scan Is a Signed Confession
Next

Windows 10 ESU Year 1 Ends October 13