ChatGPT's URL Hallucinations Lead to High Traffic on Error Pages
ChatGPT, the popular AI model, has been found to frequently link to non-existent pages, a phenomenon known as URL hallucination. This can lead to high traffic to error pages, which can negatively impact user experience. A recent study by Ahrefs revealed that ChatGPT 5 links to error pages nearly three times more than Google Search.
To address this issue, a three-step approach is recommended. First, track 404 'page not found' errors in Google Analytics 4. This can be done by adding the 'Page path and screen' dimension as a secondary dimension in your reports. Second, create helpful 404 pages to guide users to the correct content. Third, set up 301 redirects for broken URLs that generate significant traffic.
A study conducted by a team including Adam Kalai and Santosh Vempala from Georgia Tech, along with other OpenAI researchers, delved into the cause and potential solutions for URL hallucinations in AI models. They found that ChatGPT often links to sources, but these links can lead to nonexistent pages. Filtering Google Analytics reports to URLs with traffic from ChatGPT can help identify these error pages. Narrowing down the list by noting the title of pages with 404 errors and searching for them in Google Analytics can further pinpoint the problem.
While traffic from ChatGPT is typically less than 5% for most sites, it's crucial to monitor and address 404 errors to maintain a positive user experience. Bookmarking the URL of the report in Google Analytics allows for periodic checks to ensure the issue is being managed effectively.
