Express middleware that returns the resulting html after executing javascript, allowing crawlers to read on the page

koa detect googlebot, facebookbot, ...

Verify that a request is from Google using Google's recommended DNS verification steps

Track AI crawler traffic (GPTBot, ChatGPT-User, Googlebot, ClaudeBot…) on your server and send it to Spreelytics analytics.

See which AI assistants and crawlers (ChatGPT, Claude, Perplexity, Googlebot...) visit your site. Server-side bot traffic tracking for DidYouSEO.

koa detect googlebot, facebookbot ...

Agent-native multi-bot web crawler simulator. See your site through the eyes of Googlebot, GPTBot, ClaudeBot, and PerplexityBot.

🤖/👨‍🦰 Recognise bots/crawlers/spiders using the user agent string.

Validate urls against robots.txt rules.

Check if a webpage is server-side rendered or client-side rendered. See what Googlebot and AI crawlers actually see.

Handle http request from crawler (Something like Googlebot)

Verifies Googlebot, using Google's recommended method (reverse DNS). This middleware also takes performance into account, and caches the results.

A Slack bot that reminds people to google things before asking their co-workers

Streaming parser for robots.txt files

A lightweight, Open Source robots.txt parser written in JavaScript

Really simple way to add reveal on scroll animation to your React app.

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Tame the robots crawling and indexing your Nuxt site with ease.

Dynamic Rendering allows you to render React components different depending on user agent. If your visitor is a bot, like GoogleBot or Yandex, use Server Side Rendering + hydration. If your visitor is a real user, use only Client Side Rendering.

Instant link indexer and website indexing API. Submit URLs to trigger Googlebot crawls in seconds, check index status, and run AI traffic campaigns. The fastest website indexer for SEO.