Last year we released the robots.txt parser and matcher that we use in our production systems to the open source world. Since then, we’ve seen people build new tools with it, contribute to the open source library (effectively improving our production systems- thanks!), and release new language versions like golang and rust, which make it easier for developers to build new tools.
This post is also available in:
English