https://github.com/Folyd/robotstxt
Rust79
3 years ago
A native Rust port of Google's robots.txt parser and matcher C++ library.
Apache License 2.0