ULC Directory Crawler

ULC Directory operates a polite web crawler to build free unclaimed listings for South African businesses from their own public websites. This page documents how the crawler behaves and how site owners can control it.

User-Agent

ULCBot/1.0 (+https://ulc.co.za/bot; [email protected])

What we do

• Fetch publicly accessible HTML pages on a business's own domain only.
• Extract structured data (JSON-LD, Open Graph, microdata) plus factual contact, hours, and service details.
• Never copy long-form prose verbatim. Summaries are paraphrased from facts on the page.
• Never scrape third-party directories or competitors.

What we will not do

• Bypass robots.txt. If your site disallows us, we skip it.
• Follow login-gated or paywalled content.
• Crawl at a rate that stresses your server — we wait at least 2 seconds between requests.
• Process special personal information (race, health, religion, biometrics).

How to block us

Add to your robots.txt:

User-agent: ULCBot
Disallow: /

How to request removal

Visit the listing and click Request removal. We action verified requests within 7 days. This right is reaffirmed by the Protection of Personal Information Act (POPIA) where applicable.

Contact

Email [email protected] for any concerns.