Your sitemap is lying about how fresh your content is

We spent this week rebuilding how we read websites, and found something worth publishing on its own.

Every sitemap entry can carry a lastmod field: the date that page’s content last changed. Search engines use it to decide what to re-crawl. AI systems increasingly use it to decide what is current. It is one of the few honest signals a site gives about itself.

On the business we measured, 139 of 141 pages reported the same lastmod: the day we asked. Not the day the staff page changed. Not the day the finance terms were updated. The day we happened to look.

Ask again tomorrow and every page will claim it changed again.

Why platforms do this

It is rarely deliberate deception. It is a shortcut.

Generating a truthful lastmod means tracking when each page’s content actually changed, which means storing content versions. Generating a convincing-looking one means writing now() into the field when the sitemap is built. Both produce valid XML. Only one of them takes work.

For a long time this cost nothing, because nobody checked. Crawlers that got burned by inflated dates simply stopped trusting the field. Google has been explicit that it treats lastmod as a hint and ignores it when a site proves unreliable.

What it costs you now

Three things, and the third is the one that should bother you.

Your genuine updates get no priority. If every page always claims to be fresh, nothing stands out. The day you rewrite your service pricing, that page looks exactly like the 138 that did not change. You have spent the signal.

Crawlers waste their budget on you, then spend less. A crawler that re-fetches your whole site daily because everything claims to be new eventually adjusts by crawling you less often. The shortcut that was supposed to get you crawled more gets you crawled less.

An assistant cannot tell your 2022 page from your 2026 one. This is the new part, and it is why we care. When a shopper asks an assistant about your financing options, the assistant has no way to know whether the page it is reading was written last week or three years ago. Both claim today. So it answers with equal confidence either way, and if the page is stale, it states stale terms as current fact with your name attached.

A wrong answer that sounds uncertain is a small problem. A wrong answer delivered confidently, attributed to your store, is a different one.

What we do about it

We stopped trusting the field.

Our crawler records what a site claims, but it does not treat it as truth. Instead we compute our own signal: a normalized hash of each page’s meaningful content, with navigation, scripts, styles and rotating elements stripped out. When that hash changes, the content genuinely changed. When it does not, the content did not, no matter what the sitemap says.

That gives us a date we can stand behind, and we keep the two facts strictly separate:

• When we fetched it. A crawl date.
• When its content actually changed. Our own observation.
• When the site claims it changed. Recorded, and treated as a claim.

Collapsing those three into one number is how you end up publishing a confident wrong answer. They are different facts and they deserve different fields.

We are honest about the limitation too. The change date only starts accumulating meaning from the first time we read a page. For a page we saw for the first time today, we cannot tell you whether it was written in 2022 or last week. That resolves over weeks, and until it does we would rather say we do not know than invent a date.

What you can do about it, today

Check your own. Open yoursite.com/sitemap.xml and look at the lastmod values. If they are all the same date, or all today, your platform is generating them rather than tracking them.

Ask your website vendor one question: does lastmod reflect actual content changes, or is it stamped at build time? It is a fair question with a factual answer, and the answer tells you something about how much else on the site is real.

Do not fix it by turning it off. A missing lastmod is more honest than a false one, but a truthful one is better than both. This is a fixable problem on the platform side, and platforms will fix it when enough businesses ask.

The wider point

The web is being read by machines that cannot ask a follow-up question. A human browsing your site can tell a stale page from a current one within seconds: the layout is dated, the photos are old, there is a reference to a 2022 model year. An assistant reading your markup has none of that. It has the fields you gave it.

Which means the fields you give it have to be true. Not decorative, not defensive, not stamped at build time because that was easier. True.

That is most of what being agent-ready actually is, and it is less glamorous than it sounds.