In the previous posts about the Robots Exclusion Protocol (REP) we explored what’s already
possible to do with its various components — namely robots.txt and the URI level controls.
In this post we will explore how the REP can play a supporting role in the ever-evolving relation
between automatic clients and the human web.
This post is also available in:
English