Based on Tranco top-1M domains. robots.txt crawled and analyzed. Allow all: fully open · Allow partial: some paths restricted · Denied: crawling blocked · —: no directive found