OpenAI launches web crawler 'GPTBot' amid plans for next model: GPT-5

Artificial intelligence firm OpenAI has launched “GPTBot” — its new web crawling tool tit says could potentially be used to improve future ChatGPT models.

“Web pages crawled with the GPTBot user agent may potentially be used to improve future models,” OpenAI said in a new blog post, adding it could improve accuracy and expand the capabilities of future iterations.

A web crawler, sometimes called a web spider, is a type of bot that indexes the content of websites across the internet. Search engines like Google and Bing use them in order for the websites to show up in search results. 

OpenAI said the web crawler will collect publicly available data from the world wide web, but will filter out sources that require paywalled content, or is known to gather personally identifiable information, or has text that violates its policies.

It should be noted that website owners can deny the web crawler by adding a “disallow” command to a standard file on the server.

The new crawler comes three weeks after the firm filed a trademark application for “GPT-5,” the anticipated successor to the current GPT-4 model.

The application was filed at the United States Patent and Trademark Office on July 18, and covers the use of the term “GPT-5,” which includes software for AI-based human speech and text, converting audio into text and voice and speech recognition.

However, observers may not want to hold their breath for the next iteration of ChatGPT just yet. In June, OpenAI’s founder and CEO Sam Altman said the firm is “nowhere close” to beginning training GPT-5, explaining that several safety audits need to be conducted prior to starting.

Meanwhile, Concerns have been raised over OpenAI’s data-collecting tactics of late, particularly revolving around copyright and consent.

Japan’s privacy watchdog issued a warning to OpenAI about collecting sensitive data without permission in June, while Italy temporarily banned the use of ChatGPT after alleging it breached various European Union privacy laws in April.

In late June, a class action was filed against OpenAI by 16 plaintiffs alleging the AI firm to have accessed private information from ChatGPT user interactions.

If these allegations are proven to be accurate, OpenAI — and Microsoft, who was named as a defendant — will be in breach of the Computer Fraud and Abuse Act, a law with a precedent for web-scraping cases.