ccbot blocking analysis

This repo contains a prototype that analyzes crawl output, looking to quantify how much blocking of CCBot is due to bot defenses, vs. disallows in robots.txt files.

Please see OPEN-ATHENA-PILOT.md for an example analysis.