Treat a JSON page at /llms.txt as absent - #82
Merged
Merged
Conversation
Of 1,988 llms.txt files fetched in production, 1,962 open with a Markdown heading as the format specifies. Four are JSON: API hosts answering every path with a status document, served as text/plain. The HTML check caught single-page-app fallbacks but not these. Neither llms.txt nor AGENTS.md is JSON, and a status page carrying a timestamp would record a fake change on every weekly check. - A JSON content type, or a body that parses as JSON, counts as absent - Test reproduces the production case Closes #81 --- Pages affected: - [MCP server changelog](https://ai.mcpharbor.dev/changelog) -- kept free of fake llms.txt changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #81.
A sample of what the fetcher from #75 stored in production:
# …), as llms.txt specifiesThe JSON ones are API hosts answering every path with a status document, served as
text/plain— e.g.{"status":"ok","server":"mock-mcp-server","version":"1.0.0"}. The HTML check catches single-page-app fallbacks but not these. Neither file format is JSON, and a status page with a timestamp would record a fake "llms.txt changed" every week.A JSON content type, or a body that parses as JSON, now counts as absent. The test reproduces the production case.
Tests: 227 passing.
Pages affected:
🤖 Generated with Claude Code