JavaScript Links Render Pages Invisible to AI Search Crawlers, Study Reveals
A recent experiment has revealed that web pages with internal links generated by JavaScript (JS) are largely invisible to most AI search crawlers, including GPTBot, ClaudeBot, Bingbot, PerplexityBot, Meta's, and Amazon's crawlers. The study, conducted on a Brazilian business classification directory, compared two groups of pages: one with hard-coded HTML links and another with JS-injected links. While Google's crawler stack, specifically GoogleOther, demonstrated some ability to render JavaScript and follow these links (reaching 48% of JS-linked pages), Googlebot, the primary crawler for Google Search, only reached 2% of JS-linked pages. Other AI crawlers found exactly zero JS-linked pages. This indicates a significant disparity in how different crawlers process web content, with most AI assistants failing to execute JavaScript to discover navigation paths. The experiment also showed that even for Google, JS-linked pages were discovered 18% to 27% later than HTML-linked pages.