# robots.txt for clipart.free # Served as a plain-text TemplateView (no Django i18n `g` context), the same # way /ads.txt is. Crawling is open; only private / non-content surfaces and # thin search-result query strings are disallowed. User-agent: * Allow: / # Admin, APIs and payment webhooks - never content. API docs live at # /developers/ (indexable), never under /api/. Disallow: /admin/ Disallow: /api/ Disallow: /ipns/ # Account / auth / billing flows - private, no SEO value. Disallow: /account/ Disallow: /checkout/ Disallow: /login/ Disallow: /logout/ Disallow: /signup/ Disallow: /verify/ Disallow: /lost-password/ Disallow: /restore-password/ Disallow: /success/ Disallow: /cancel/ Disallow: /delete-account/ # Utility redirects + captcha + well-known. Disallow: /s/ Disallow: /captcha/ Disallow: /.well-known/ # Tag pages. NOT a case of the "a Disallowed URL can never be re-crawled, so it # can never see its noindex" trap: ClipartTagView.INDEXABLE_TAGS is empty and # indexation additionally requires >=20 assets, so no tag page has ever been # indexable; tools/sitemaps.py has never listed one. There is nothing in the # index to freeze here. They are pure crawl cost - 11,099 distinct tags were # walked in 89 minutes on 2026-08-04. Promoting a tag via INDEXABLE_TAGS means # lifting this line for it, or the promoted page will never be crawled. Disallow: /tag/ # Thin/duplicate internal search-result pages (?q=...) and pure re-orders # (?sort=). Language alternates (?lang=xx) are intentionally NOT disallowed - # they are the hreflang targets. Pagination (?page=) is left crawlable so the # deep asset links remain followable. Disallow: /*?q= Disallow: /*?sort= # The /languages/ chooser is kept CRAWLABLE on purpose: the LanguagesPage view # emits `X-Robots-Tag: noindex`, and Googlebot must be able to re-crawl the # already-stuck /languages/?return=... URLs to see that header and drop them # from the index (a robots-Disallowed URL is never re-crawled, so the noindex # would stay inert on it). ?return= is only ever generated by that chooser, so # no other path is exposed, and every page carries a query-less self-canonical. # ?next= (login flow, already blocked at /login/) stays disallowed; ?lang= # stays crawlable (hreflang target). Disallow: /*?next= # --- Per-agent budget (measured from /var/log/nginx/clipartfree.access.log, # 89 minutes on 2026-08-04, 457,842 requests / 30,014 distinct IPs) --------- # # Read the honest limit first: ~85% of that traffic was a distributed scraper # rotating spoofed full-Chrome UAs across 30k IPs at a median of 2 req/min # each. It will not read this file and per-IP rate limiting cannot separate it # from a human. Nothing below touches it - the levers that do are the hreflang # gate in base.html (which stops advertising ~100 language URLs per noindex # page) and the nginx trailing-slash redirect. These directives address only # the ~15% that DOES identify itself. # SleepBot was the single largest identified agent: 35,131 requests in 89 # minutes (7.7% of everything, ~6.6 req/s) against a site it sends no traffic # to. Nothing here is licensed away by refusing it - the assets are CC0 and # remain downloadable by hand. User-agent: SleepBot Disallow: / # AI/answer-engine crawlers are deliberately left ALLOWED - a CC0 library # wants to be in the training and citation set - but they were 33k requests in # 89 minutes between them, so they get a rate ceiling. Crawl-delay is honoured # unevenly (Google ignores it entirely; that is fine, Googlebot was 0 requests # in the whole window and is not the problem). # # !! The Disallow list below is DUPLICATED on purpose. A named User-agent group # !! REPLACES the `*` group outright - it does not inherit from it. Writing a # !! bare `Crawl-delay:` group would have handed these agents /admin/, /api/, # !! /tag/ and every private flow, i.e. loosened exactly what we are tightening. # !! Keep the two lists in sync when you edit either. User-agent: GPTBot User-agent: meta-externalagent User-agent: Amazonbot User-agent: ClaudeBot User-agent: PerplexityBot User-agent: CCBot User-agent: Bytespider User-agent: Applebot-Extended Crawl-delay: 10 Disallow: /admin/ Disallow: /api/ Disallow: /ipns/ Disallow: /account/ Disallow: /checkout/ Disallow: /login/ Disallow: /logout/ Disallow: /signup/ Disallow: /verify/ Disallow: /lost-password/ Disallow: /restore-password/ Disallow: /success/ Disallow: /cancel/ Disallow: /delete-account/ Disallow: /s/ Disallow: /captcha/ Disallow: /.well-known/ Disallow: /tag/ Disallow: /*?q= Disallow: /*?sort= Disallow: /*?next= # SEO backlink crawlers: no referral value, and they walk the deep tree. # Same duplication rule as above. User-agent: AhrefsBot User-agent: SemrushBot User-agent: DotBot User-agent: MJ12bot User-agent: DataForSeoBot Crawl-delay: 30 Disallow: /admin/ Disallow: /api/ Disallow: /ipns/ Disallow: /account/ Disallow: /checkout/ Disallow: /login/ Disallow: /logout/ Disallow: /signup/ Disallow: /verify/ Disallow: /lost-password/ Disallow: /restore-password/ Disallow: /success/ Disallow: /cancel/ Disallow: /delete-account/ Disallow: /s/ Disallow: /captcha/ Disallow: /.well-known/ Disallow: /tag/ Disallow: /*?q= Disallow: /*?sort= Disallow: /*?next= Sitemap: https://clipart.free/sitemap.xml