Two ways your robots.txt blocks images you meant to allow

Most robots.txt files are written once, early, when the only thing anyone wants to keep crawlers out of is the admin area. A line goes in, and nobody opens the file again for two years.

That holds up until the CMS starts serving something you do want crawled from underneath the same prefix. Two rules decide whether that happens quietly: how a crawler resolves overlapping Allow and Disallow rules, and what a named User-agent group inherits from the * group. The short version is that the first rule is more forgiving than people expect and the second is far less forgiving.

Check where your CMS actually serves media from

This site runs on EmDash, which mounts its admin UI and its API under a single /_emdash/ prefix. Uploaded files are served from inside that prefix too, at /_emdash/api/media/file/<key>. So the obvious, cautious rule —

User-agent: *
Disallow: /_emdash/

— covers the editor and the API, which is what it was written for, and also covers every cover image and inline diagram the articles embed, which is not.

This is not an EmDash quirk. Ghost serves uploads from /content/images/, which is usually fine, but plenty of setups put media under /admin/, /api/, /cms/ or /_next/ — all paths people habitually disallow. The question worth asking about your own stack is not "what did I block?" but "what URL does an <img> on a published page actually point at?" View source on a real article and read the src. That takes a few seconds and is the only answer that counts.

Rule one: the longest match wins, and ties go to Allow

The fix is to add a narrower Allow back, which works because of how overlapping rules are resolved. RFC 9309 §2.2.2 says the most specific rule wins, where specificity is measured in octets of the matched path, and that a tie resolves in favour of Allow. Google's robots.txt specification describes the same behaviour.

So given:

Disallow: /_emdash/
Allow: /_emdash/api/media/

a request for /_emdash/api/media/file/abc.png matches both. Disallow matched 10 octets, Allow matched 21. The Allow wins and the image is crawlable. A request for /_emdash/login matches only the Disallow, so the admin area stays closed. Order in the file is irrelevant — only match length matters.

This is the part people get wrong in the safe direction: they assume a broad Disallow is final and work around it by moving files, when a one-line Allow would have done it.

Rule two: a named user-agent group is not an overlay on *

The second rule is where the damage usually happens, because it fails silently.

A crawler selects exactly one group of rules. Per RFC 9309 §2.2.1, it picks the group whose User-agent most specifically matches its own token, and * is a fallback — it applies only when no named group matches. A crawler that finds its own name in the file reads that group and nothing else.

The practical consequence: the moment you add a named group for, say, OpenAI's search crawler, that crawler stops seeing every rule you wrote under *. Not just the Disallow lines — the Allow lines too, including the one keeping your images reachable.

User-agent: *
Disallow: /_emdash/
Allow: /_emdash/api/media/

User-agent: OAI-SearchBot
Allow: /

Read that as OAI-SearchBot does: it matches the second group, so the only rule it has is Allow: /. In this case it happens to be harmless. Invert it — put Disallow: /_emdash/ in the named group without the media Allow — and you have blocked your images for that one crawler while the file still looks correct at a glance, because the rule you need is sitting right there, six lines up, in a group that crawler never reads.

There is no inheritance and no merging. Every group is self-contained, and a named group has to repeat every rule that should apply to it.

What this site's file looks like now

User-agent: *
Allow: /
Disallow: /_emdash/
# Media files are the article images, covers and logo that pages already
# reference. The rest of /_emdash/ stays closed: it is the admin UI and API.
Allow: /_emdash/api/media/

User-agent: OAI-SearchBot
Allow: /
Disallow: /_emdash/
Allow: /_emdash/api/media/

Sitemap: https://jitsumi.com/sitemap.xml
Sitemap: https://jitsumi.com/sitemap-static.xml

The media Allow appears twice on purpose. Duplicating it is the correct shape, not redundancy to tidy up later — and the comment exists so the next person to widen the Disallow understands what the narrow Allow is protecting.

Checking your own site

Three steps, none of which need a tool:

1. Find the real image URL. Open a published article, view source, copy the src of a content image. Do not guess it from the CMS's documentation.
2. Run it against your rules by hand. For each group that applies, find every rule whose path is a prefix of your URL, and keep the longest. If the longest is an Allow, or there is a tie, the URL is crawlable.
3. Check every named group separately. If your file has a User-agent line that is not *, repeat step 2 for that group in isolation. Pretend the * group does not exist, because for that crawler it does not.

Google Search Console's URL Inspection tool will confirm the verdict for Googlebot on a live URL, which is worth doing for one image once you have made a change. Requesting a URL yourself with a crawler's user-agent string only tells you your server responds — it says nothing about whether that crawler will actually request it, because robots.txt is enforced by the crawler, not by you.

What this buys you, honestly

Unblocking your images makes them eligible to be crawled. That is the whole claim.

It is a prerequisite for appearing in Google Images and for your images being fetched for previews and other surfaces, and a blocked image cannot do any of that. It is not a ranking improvement, and fixing it will not move text results. The reason to do it is that the current state is unintentional: nobody decides to hide their own illustrations, they just inherit a Disallow written before the media path existed.

One caveat worth keeping in view: ChatGPT-User, which handles user-initiated fetches, is documented by OpenAI as a case where robots.txt rules may not apply. Rules you write for it are a statement of intent rather than a guarantee.

Frequently asked questions

Does Disallow on my admin path stop Google indexing the articles themselves?

No, as long as the article URLs are outside the disallowed prefix. Blocking /_emdash/ blocks the admin area and anything served beneath it. Pages at /posts/... are unaffected. What you lose is the images embedded in those pages, which are fetched as separate URLs and evaluated against robots.txt separately.

Which rule wins when Allow and Disallow both match the same URL?

The one that matched more characters of the path. RFC 9309 §2.2.2 measures specificity in octets of the matched path and resolves an exact tie in favour of Allow. Position in the file does not matter, so you cannot fix a conflict by reordering lines.

Do I have to repeat my Allow rules in every user-agent group?

Yes. A crawler reads exactly one group — the most specific User-agent match for its own token — and * applies only to crawlers with no named group. There is no inheritance, so a named group must restate every rule that should apply to it.

Will unblocking my images improve my rankings?

There is no basis for expecting that. It makes the images eligible for crawling and for image search, which a blocked image is not. Treat it as repairing something that broke by accident rather than as an optimisation.

Does adding an FAQ section get me rich results in Google?

Not any more. Google stopped displaying FAQ rich results for the general web on 7 May 2026. FAQ structured data is still valid markup and still describes your content accurately for anything else that reads it, but it will not produce an expandable listing in Google. Write the questions because readers ask them, not for the snippet.

Is robots.txt the right tool for keeping a page out of search results?

No. Disallowing a URL stops it being crawled, which also stops a crawler seeing a noindex on it, so a disallowed page can still surface from external links. If you want a page excluded from results, let it be crawled and serve noindex on it. Use robots.txt to manage crawl access, not indexing.