@advance-labs/crawler
Apache-2.0Polite, bounded HTTP crawler — sitemap-first discovery, link-following BFS, robots.txt, and site-file detection.
100
Security score
0 known advisories in v0.2.2
Weekly downloads
—
Unpacked size
103.8 kB
Dependencies
4
Last publish
8 days ago
Security advisories
No known vulnerabilities affect v0.2.2.