← Back to blog

Technical SEO · August 16, 2026 · 7 min read

Automating Internal Linking With AI: A Site-Wide Linking Engine Build

Build an embeddings-based automated internal linking engine that surfaces contextual link candidates across your entire site automatically.

By FluxWriter Team

Automated internal linking is one of the highest-leverage, lowest-effort wins available to any site that has accumulated hundreds or thousands of pages — and yet most teams still handle it manually, one post at a time. A well-built embeddings-based recommender can scan every page on a domain, compute semantic similarity, and surface contextually relevant link candidates in seconds. This guide walks through exactly how to build one.

Why Manual Internal Linking Breaks at Scale

A 50-page site can be linked by hand. A 2,000-page site cannot — at least not coherently. The typical result is a hub-and-spoke structure where the homepage and a handful of cornerstone pages absorb almost all internal link equity, while the long tail starves for PageRank.

The practical consequences:

• Crawl budget is wasted on shallow pages that get re-crawled frequently while deeper content is ignored.
• Topical authority signals are diluted because semantically related pages don't reinforce each other.
• Users follow obvious paths and miss relevant content that exists two clicks off the beaten track.

The fix is not "write a linking policy." The fix is a system that reasons about content the same way a search engine does — through meaning, not keyword matching.

The Architecture: Embeddings + Cosine Similarity

The core insight is simple. If you convert every page's content into a vector embedding, semantically similar pages cluster together in vector space. Internal link recommendations then become a nearest-neighbor lookup.

Step 1 — Crawl and Extract Content

Start with a crawl of your site. You don't need Screaming Frog licenses; a minimal Python crawler using httpx and BeautifulSoup is sufficient for most sites under 10,000 pages.

Extract:

• URL
• Title
• Meta description (as a quality signal, not for embedding)
• Main body text (strip nav, footer, sidebars)

Store each page as a row in a SQLite database or a simple JSONL file. The key field is clean body text — garbage in, garbage out.

Step 2 — Generate Embeddings

Use a text embedding model to convert each page's content into a dense vector. Two practical options:

For a 2,000-page site with average page length of 800 words, you're looking at roughly 3.2 million tokens — about $0.06 at OpenAI pricing. Run it once, store vectors, and re-run only on new or updated content.

Chunk long pages at roughly 512 tokens and average the chunk embeddings into a single page-level vector. This prevents a single 5,000-word pillar post from being misrepresented by only its opening paragraphs.

Step 3 — Build the Similarity Index

With embeddings in hand, compute cosine similarity across all page pairs. For 2,000 pages, that's 2,000 × 2,000 = 4 million comparisons — fast enough to run in a few seconds with NumPy.

import numpy as np

def cosine_similarity_matrix(embeddings: np.ndarray) -> np.ndarray:
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    normalized = embeddings / norms
    return normalized @ normalized.T

For sites over 50,000 pages, swap this out for an approximate nearest-neighbor library like FAISS or Annoy. Exact cosine similarity becomes a bottleneck above that threshold.

Step 4 — Filter and Rank Candidates

Raw similarity scores need three filters before they become useful recommendations:

Self-exclusion. A page should never link to itself. Zero out the diagonal of the similarity matrix.

Already-linked exclusion. Parse existing <a href> tags from each page during your crawl and exclude URLs that are already linked. This prevents the engine from noisily re-recommending what already exists.

Minimum similarity threshold. A score below 0.65 (on a 0–1 cosine scale) usually indicates weak topical overlap. Set a floor and tune it against your content.

After filtering, take the top-N candidates per page — 5 to 10 is a practical range. Return both the recommended URL and the best-matching anchor text: pull the target page's most keyword-rich sentence that overlaps with the source page's topic.

Step 5 — Anchor Text Extraction

The recommender should suggest specific anchor text, not just destination URLs. A simple approach: for each (source, target) pair, find the sentence in the target page that has the highest token overlap with the source page's topic cluster, then extract the noun phrase most likely to serve as a natural anchor.

A more robust approach uses the same embeddings: embed each sentence in the target page, find the sentence closest to the source page's centroid vector, and surface it as the suggested anchor. This tends to produce more contextually appropriate anchors than keyword-matching alone.

Serving Recommendations

You have two deployment options:

Batch report. Run the recommender weekly as a cron job. Output a CSV or Google Sheet: Source URL, Target URL, Suggested Anchor, Similarity Score. Your editorial team reviews and applies links during content refreshes.

Real-time API. Expose a /recommend?url=<page_url> endpoint. Your CMS calls it when a writer opens an editor, surfaces the top 5 suggestions inline. This tightens the feedback loop and catches new content immediately after publish.

For most teams, the batch report is the right starting point. It requires no CMS integration, produces an auditable record of changes, and lets editors apply judgment before links go live.

A Concrete Example

Consider a software documentation site with 800 pages. Running the recommender surfaced 4,200 link candidates — an average of 5.25 per page. The editorial team applied 3,100 of them over three weeks (filtering out 1,100 that felt forced or redundant in context).

Six weeks post-implementation, crawl depth for orphaned pages dropped from an average of 6.2 clicks from the homepage to 3.8 clicks. Organic impressions for the long-tail documentation pages rose 31% over the same period (factoring out a concurrent content push).

The key variable: the quality of the embedding model mattered more than the architecture. Switching from a generic model to one fine-tuned on technical content improved recommendation precision — fewer suggestions that were semantically adjacent but topically irrelevant to the source page.

Edge Cases to Handle

Pagination and parameter URLs. Canonicalize before crawling. Embedding /blog/?page=3 as a distinct page wastes compute and pollutes the index.

Thin pages. Pages under roughly 200 words often produce unstable embeddings. Set a minimum word count for inclusion in the index and flag them for content improvement instead.

Seasonal and time-sensitive content. Evergreen pages shouldn't accumulate inbound links from dated news posts. Add a date-decay weight if link equity concentration matters for your use case.

International and multilingual sites. Multilingual embedding models (like multilingual-e5-large) handle mixed-language corpora, but ensure you only recommend cross-links when they make sense for the user — don't link a French page from an English source unless the site is structured for it.

FAQ

How often should I re-run the recommender?

Re-embed pages when content changes, not on a fixed schedule. A practical trigger: if the word count or title of a page changes by more than 15%, queue it for re-embedding. New pages should be embedded and indexed immediately after publish. Running the full similarity matrix weekly is cheap enough that it can just be a cron job regardless.

Does this replace human editorial judgment on internal links?

No. The recommender surfaces candidates; humans (or a rules layer) decide what ships. Some candidates will be technically similar but contextually awkward — a paragraph about "Python loops" doesn't always want a link to a page titled "Python Snake Care." Editorial review catches those. Think of the engine as a junior SEO who has read every page on the site and never forgets anything.

What similarity threshold should I use?

Start at 0.65 cosine similarity and audit the first 50 recommendations manually. If you're seeing many irrelevant suggestions, raise it to 0.72. If you're getting too few candidates on a tightly focused site, lower it to 0.60. The right threshold is site-specific and depends heavily on how topically diverse your content is.

Practical Takeaway

The embeddings pipeline described here — crawl, embed, similarity matrix, filter, recommend — can be built in a weekend with open-source tools and run for under a dollar per month on most sites. The return on that investment compounds: every new page automatically becomes a link candidate for every existing page, and vice versa. Internal linking stops being a task you defer and becomes something the site does for itself.

If you're also producing content at scale, tools like FluxWriter can feed the same pipeline directly — new articles arrive pre-embedded and slot into the recommender index without a separate crawl step.

← All posts