link-followability

Resolve whether a link on a served page is actually followed, and whether the page carrying it can be indexed at all.

A page has three separate places where it can say do not follow this:

1. the rel attribute on the anchor,
2. a <meta name="robots"> tag,
3. an X-Robots-Tag response header.

They are usually read by three different pieces of code that never compare notes, which is how a link gets recorded as live when a header two levels above it has already killed it. This library reads all three, applies the most restrictive directive that binds the crawler you name, and returns one verdict carrying followable, indexable, and the list of reasons that decided it.

No network access, no HTML parsing, no dependencies. You hand it strings you already have.

Install

composer require theluckystrike/link-followability

Requires PHP 8.1 or newer.

Use

use TheLuckyStrike\LinkFollowability\Followability;
use TheLuckyStrike\LinkFollowability\Input;

$verdict = Followability::evaluate(new Input(
    rel: 'noopener noreferrer',
    metaRobots: ['index, follow'],
    xRobotsTag: ['googlebot: noindex'],
    userAgent: 'googlebot',
));

$verdict->followable;   // true
$verdict->indexable;    // false
echo $verdict;          // followable=true indexable=false (x-robots-tag:googlebot:noindex)

The reasons matter more than the booleans. A link recorded as dead should be able to say which directive killed it and which of the three sources it came from:

use TheLuckyStrike\LinkFollowability\Source;

$verdict->reasonsFrom(Source::XRobotsTag)[0]->directive;   // 'noindex'
$verdict->reasonsFrom(Source::XRobotsTag)[0]->userAgent;   // 'googlebot'

For the common case where you only have an anchor:

Followability::fromRel('nofollow noopener')->followable;   // false
Followability::fromRel('noopener external')->followable;   // true

Headers usually arrive as a map from an HTTP client, cased however the server felt like casing them, and repeated when the server sent the tag more than once. Input::fromHeaders() handles both:

$input = Input::fromHeaders('noopener', $response->getHeaders());
$verdict = Followability::evaluate($input);

What it implements

From the robots meta / X-Robots-Tag specification:

• none is equivalent to noindex, nofollow; all is equivalent to index, follow.
• A directive prefixed with a crawler name — googlebot: noindex — binds only that crawler. An unprefixed directive binds every crawler.
• Where directives conflict, the restrictive one wins: index, noindex is noindex.
• Directives that legitimately carry a colon (max-snippet:-1, max-image-preview:large, unavailable_after:…) are not mistaken for a crawler name.

From the rel attribute:

• nofollow, ugc and sponsored each stop the link passing ranking signals; the others do not.
• Tokens may be separated by whitespace or by commas, because publishing forms emit both.
• Matching is case-insensitive and duplicates collapse.

Evaluation starts from followable and indexable and only ever removes permissions, so the order the sources are read in cannot change the answer.

Two distinctions worth knowing

Page nofollow and anchor nofollow are kept apart in the reasons. They fail for different operational causes: one is the host's policy for the whole page, the other is what the publishing form did to your anchor. Only one of those is worth arguing with.

rel="noindex" is not a robots directive. It is not a registered link type and no crawler honours it, so it is reported as a token present on the anchor but it never changes the verdict. Some registries serve it anyway.

Tests

composer install
composer test

Ports

The same logic in other languages, so a pipeline written in any of them gets the same verdict:

• Rust — followability
• Go — link-verdict
• D — linkfollow

Why it exists

It is the link verification step of handsofflinks, an off-page SEO pipeline where a row may not be marked live until the served HTML has been re-fetched and this verdict comes back followable. Marking a link live because it was published, rather than because it was re-read, is the single easiest way to build a report that is wrong.

License

MIT