NetsurferBot

The crawler behind Netsurfer's own index of the small web

What it's for

Netsurfer is a free app for iPhone, iPad and Mac for exploring small, independent websites: you pick a topic and swipe through real sites about it. NetsurferBot builds the app's own index of those sites, starting from public lists of personal websites such as Kagi's Small Web, the 512KB Club, Neocities' site listings and the member lists of webrings such as the IndieWeb webring, Hotline Webring and Fediring, plus sites people suggest to us by email, each looked at by hand, and growing from there through the small sites those sites link to.

The app searches this index first, and asks other search services only when it runs low on results for a topic.

How to recognize it

Every request it makes says so in its User-Agent:

NetsurferBot/0.1 (+https://netsurfer.app/bot; hello@netsurfer.app)

The version number will change over time. The name won't.

How it behaves

• It follows robots.txt, both rules for NetsurferBot and rules for all robots (*).
• It reads your feed, if you have one (RSS, Atom or JSON Feed), instead of visiting every post: usually three requests per site (robots.txt, your home page and your feed). It checks your feed again daily, weekly or monthly, depending on how often you post, and only asks whether anything has changed. Your home page gets a visit about once a month.
• It skips pages marked noindex, in a robots meta tag or an X-Robots-Tag header. On your home page, noindex keeps your whole site out. Posts read from your feed aren't visited one by one, so a noindex on a single post can't be seen: use robots.txt for that (below).
• It finds new sites through links, without extra visits. It notes which other sites your home page and your feed's posts link to, and visits a new site only when at least two of the small sites it reads link to it from their home pages, and it isn't one of the world's most-linked sites (judged by the Majestic Million, CC BY 3.0). Links marked rel="nofollow", "ugc" or "sponsored" are ignored, and so is every link on a page with a nofollow robots meta tag or X-Robots-Tag header. It doesn't wander around your own site: it reads your home page and your feed.
• It's slow on purpose: one request to your site every 5 seconds at most, or longer if your robots.txt sets a Crawl-delay. Sites that share one host's servers also share a pace: all Neocities sites together get one request every 2 seconds at most. If your server says it's busy (HTTP 429 or 503), it leaves and tries again an hour later at the earliest, or later if your server's Retry-After asks.
• It only reads what any visitor can see. No logins, no forms, and it doesn't run JavaScript.
• It keeps what search needs: the address, title, date and text of your home page and your 50 newest posts, which other sites they link to, and for a Neocities site, the tags you gave it on Neocities. It doesn't download your images or other files, and it never republishes your pages. When someone picks your site in Netsurfer, the app opens your real website, full screen, as a browser would.

Keeping your site out

To keep NetsurferBot away from your whole site, add this to your robots.txt:

User-agent: NetsurferBot
Disallow: /

To keep out only part of it, list those paths instead of /. That works for a single post too, for example Disallow: /2024/my-post/. A <meta name="robots" content="noindex"> tag on your home page also keeps your whole site out. NetsurferBot takes a change into account the next time it visits, and removes pages it's no longer allowed to read.

Or email hello@netsurfer.app with your site's address, and we'll remove it from the index and keep it out.

The app also finds sites through other search services, such as Wiby and Neocities' tag pages, without NetsurferBot, so robots.txt can't keep a site out of those. If you'd rather your site never appeared in Netsurfer at all, say so in your email and we'll add it to the app's list of sites it always leaves out.

Contact