Our crawler

Pioneer Pennant Process reads public college softball pages to keep its school facts current. Its crawler says who it is in every request: PioneerProcessBot/1.0 (+https://pioneerpennant.com/process/about-our-crawler; [email protected]).

What it reads

• Each school's own athletics site: the softball coaches and staff page, the roster, the program's social media links, and its camp and recruiting questionnaire links.
• School logos from NCAA.com.
• As Process adds NAIA and junior college programs, the member and standings pages those associations publish.

Public pages only. It never signs in, never fills in a form, and never works its way past a bot check. It does not read athletes' personal pages or social media accounts.

How often

A check of each school's head coach once a month, and every week from June 1 to September 15, when coaches change jobs. Fuller passes a few times a year.

How it behaves

• It reads your robots.txt first and follows the rules you set for PioneerProcessBot, or for every crawler (*), the way RFC 9309 reads them: the longest matching rule wins, and * and $ work in paths.
• It keeps to your Crawl-delay (or Request-rate) between the start of one request to your site and the next, and waits 5 seconds when you set none. If you ask for more than a minute between requests, it leaves your site alone.
• It keeps to a Visit-time window if you set one.
• Before it follows a redirect to another site, it checks that site's robots.txt too.
• If your robots.txt cannot be read (a server error, a timeout) or turns our crawler away, it leaves your site alone for that run.

Asking it to stop, or fixing a listing

Add a Disallow for PioneerProcessBot to your robots.txt, and the next run follows it. Or email [email protected] with your site's address, and a person answers. The same address fixes a listing: a coach who has moved on, a wrong email, a program that has changed.