202 packages matching “googlebot”
seoserver
v1.1.6 · 13 years ago
<h3>Welcome!</h3> <p>Seo Server is a command line tool that runs a server that allows GoogleBot(and any other crawlers) to crawl your heavily Javascript built websites. The tool works with very little changes to your server or client side code.</p> <p><i>This entire site is driven by Javascript(view the source or see the <a href="https://github.com/apiengine/seoserver-site">code</a>). Click the `What does Google see?` button at the bottom of each page to see Seo Server in action.</i></p>
No known vulnerabilities
express-seo-robots
v1.0.2 · 4 years ago
Simple robots.txt file generation for express
No known vulnerabilities
generate-robotstxt
v8.0.3 · 6 years ago
Awesome generator robots.txt
No known vulnerabilities
crawler-user-agents
v1.56.0 · 1 month ago
This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file.
No known vulnerabilities
robotstxt-util
v4.1.0 · 1 year ago
RFC 9309 spec compliant robots.txt builder and parser. 🦾
No known vulnerabilities
@prerenderbuddy/cli
v0.1.4 · 2 days ago
Open-source crawler-readability and discovery-file diagnostics for public websites.
No known vulnerabilities
@prerenderbuddy/mcp
v0.1.5 · 2 days ago
Local MCP tools for crawler-readability and discovery-file diagnostics.
No known vulnerabilities
isbot-fast
v1.2.0 · 6 years ago
JavaScript module detecting bots/crawlers/spiders via user-agent
No known vulnerabilities
rendertron-middleware-standalone
v0.1.1 · 5 years ago
Simple middleware for rendertron using [rendertron-middleware](https://github.com/GoogleChrome/rendertron/tree/main/middleware). Opinionatedly configured to skip static resources and every bot (including GoogleBot).
No known vulnerabilities
@morioh/is-bot
v1.1.2 · 3 years ago
Detect user-agent is a bot/spider/crawler
No known vulnerabilities
@headwall/trusted-network-providers
v2.2.1 · 17 days ago
Trusted network hosts and address ranges.
No known vulnerabilities
cloudflare-bot-directory
v1.0.39 · 1 day ago
CloudFlare Radar verified bots directory - 500+ web crawlers, search engine bots, and user agents as JSON for bot detection and traffic filtering
No known vulnerabilities
@rearguard/is-bot
v1.9.6 · 4 years ago
Ultra fast bot/crawler detector using user agent strings.
No known vulnerabilities
@types/woothee
v1.11.3 · 2 years ago
TypeScript definitions for woothee
No known vulnerabilities
nuxt-skew-protection
v1.4.0 · 1 day ago
Solve Nuxt version skews with persistent assets and instant updates.
No known vulnerabilities
grunt-escaped-seo
v0.5.1 · 12 years ago
Generate an SEO website and sitemap for google escaped fragments
No known vulnerabilities
to-query
v1.6.22 · 4 months ago
Get query object from a request url
No known vulnerabilities
@trybyte/robotstxt-parser
v1.2.0 · 8 months ago
Google's robots.txt parser ported to TypeScript - RFC 9309 compliant
No known vulnerabilities
@midudev/react-progressive-hydration
v2.0.2 · 6 years ago
Server Side Rendering for a component and skip the hydration step on the client but, as soon as it appears on the viewport, then hydrate it. Useful for list of items.
No known vulnerabilities
hexo-robots
v1.0.4 · 8 hours ago
a hexo plugin that help you create robots.txt file when you build the site.
No known vulnerabilities