Who we are
UniSupport helps students compare universities abroad. To keep what we show accurate, our crawler reads a small number of public pages on university websites, about twice a year. Every request identifies itself with this User-Agent, so you can recognise it in your logs:
UniSupportBot/1.0 (+https://www.unisupport.co/bot; bot@unisupport.co)What it reads
Public, factual information for applicants: official links (admissions, international students, fees, scholarships), programmes, intakes and deadlines, tuition fees and entry requirements. Every fact we show links back to the university page it came from.
It never collects personal data, never logs in, never submits forms, and never republishes page text.
How it behaves
- It reads robots.txt first and obeys it, including Crawl-delay.
- One request at a time per site, at least 2 seconds apart, and only a few pages per visit.
- It asks whether a page has changed before downloading it again.
- It backs off on errors and on 429 or 503, honouring Retry-After.
- It never tries to get past a CAPTCHA, challenge page or block: it stops and leaves the site alone.
How to stop it, or correct something
To stop it crawling your site, add this to your robots.txt. It takes effect the next time we read robots.txt:
User-agent: UniSupportBot
Disallow: /Or email bot@unisupport.co and we will stop within 5 working days. The same address takes corrections to anything we show about your university.