A link checker looks simple: request each link, report the ones that fail. In practice the hard part is deciding what a failure means. A page that returns 404 today may be back tomorrow. A page that returns 403 may be fine for a reader and closed to scripts. A site that returns 429 is asking you to slow down, and every extra request makes things worse. And a checker that follows arbitrary links from your content is a tool that fetches addresses other people wrote.
This guide builds the decision core of a link checker: what to request, how fast, what each answer means, and how many sightings it takes to call a link dead. It does not build the crawler that finds links in your pages, and it makes no claim about search ranking. The code uses only Python's standard library, and the tests ran against a throwaway server on the local machine that returns each kind of response on demand. No outside site was contacted. The time thresholds are example policy values, not standards.
What the standards say each answer means
The HTTP specification is careful about what servers promise. A 404 says the server found no current representation, or is unwilling to say one exists, and it does not say whether the condition is temporary or permanent. A 410 is for the case where the server knows the condition is likely permanent, and the specification says it is meant to help web maintenance by telling others that links to the resource should be removed. Servers are not required to use it, so most dead links will arrive as 404 anyway.
A 301 or 308 says the resource has a new permanent address. The specification adds a caution that matters for automatic repair: the suggestion that a client replace the old reference is usually ignored unless the client is actively editing references, the connection is secured, and the server is a trusted authority for the content. A checker can report a permanent redirect and propose an edit, but a person should approve rewriting links.
A 401 means credentials are missing or refused, and a 403 means the server understood the request and refuses it. The specification also notes that a server wishing to hide a forbidden resource may answer 404 instead. So a 404 can sometimes mean "not for you." A 405 says the method is not supported, which is why a checker that tries HEAD first should fall back to GET before it concludes anything.
A 429 means too many requests in a given time. The response may carry a Retry-After header, and the HTTP specification defines that header as either a number of seconds or a date. RFC 6585 says it does not define how a server counts requests or identifies the client, so you cannot predict the limit and should respect what the server tells you.
What to request at all
OWASP's guidance on server-side request forgery covers the case where an application makes requests to any external address a user can influence. That is a link checker. The guidance says an allowlist is not usable there because the destinations are unknown in advance, so the application has to block what it should not reach. The list includes private ranges, localhost and link-local addresses, and the guidance admits that doing this at the application layer is hard because it takes care to cover every address form. It also advises disabling redirect following or revalidating, since a redirect is a way around input checks.
The checker below applies these checks before every request, including each redirect hop. It accepts only http and https, refuses addresses with embedded credentials, treats a malformed address or an invalid port as unsafe, resolves the host name and refuses any address that is not public at the moment of that lookup, and skips tracking and token links. That last rule is a design choice and not from a standard. Unsubscribe links, click-through links and links with tokens in the query string can perform an action when opened, so a checker should not open them for you. The address check has a gap, described after the code, and it is the reason this filter must not be your only protection.
Robots rules and politeness
RFC 9309 defines the robots.txt format and says its rules are not a form of access authorization. It also says how a crawler should treat the file's availability. If robots.txt is unavailable, with a 4xx response, a crawler may access any resource. If it is unreachable because of a server or network error, the crawler must assume complete disallow. Whether a link checker counts as a crawler for external hosts is your policy decision. A cautious default is to honor robots.txt for hosts you do not own, send one request at a time per host, identify yourself with a contact address in the user agent, and request headers only, never the page body.
The core of the checker
Four parts do the work. A request filter decides what may be fetched. A per-host limiter spaces requests and honors Retry-After. A categorizer turns a response into a category. A decision function turns a history of categories into a plain-words verdict.
def safe_to_fetch(url, resolve=socket.getaddrinfo, allow_hosts=frozenset()):
"""Return 'ok', 'dns', or 'skipped: reason'. Runs before every request, including redirect hops.
allow_hosts is an explicit set of hostnames exempt from the public-address rule (for tests only)."""
try:
p = urlsplit(url)
port = p.port # raises ValueError for a port such as :99999 or :abc
except ValueError:
return "skipped: malformed url or port"
if p.scheme not in ("http", "https"):
return "skipped: scheme"
if not p.hostname:
return "skipped: no host"
if port == 0:
return "skipped: malformed url or port"
if p.username or p.password:
return "skipped: credentials in url"
if TRACKING.search(url):
return "skipped: tracking or tokenized link"
try:
infos = resolve(p.hostname, port or (443 if p.scheme == "https" else 80), type=socket.SOCK_STREAM)
except socket.gaierror:
return "dns"
for info in infos:
ip = ipaddress.ip_address(info[4][0].split("%")[0])
if not ip.is_global and p.hostname not in allow_hosts:
return "skipped: non-public address"
return "ok"
def categorize(status):
if 200 <= status < 300:
return "ok"
return {404: "not_found", 410: "gone", 401: "blocked", 403: "blocked", 429: "rate_limited"}.get(
status, "transient" if status >= 500 or status == 408 else "needs_review")
def decide(history, now):
"""history: list of {'at': iso string, 'category': str}, oldest first. Returns a plain-words verdict."""
if not history:
return "unchecked"
last = history[-1]["category"]
if last == "ok":
return "ok"
if last == "moved_permanently":
return "update the link to the new address"
if last == "blocked":
return "cannot verify: a login or block, check by hand or exclude"
if last in ("rate_limited", "skipped", "needs_review"):
return "not judged: " + last
t = lambda h: datetime.fromisoformat(h["at"])
if last in DEAD_SET and len(history) >= 2 and history[-2]["category"] in DEAD_SET \
and t(history[-1]) - t(history[-2]) >= timedelta(hours=24):
return "dead: report to a person"
if last == "gone":
return "likely gone (410): review once more"
if last in DEAD_SET:
return "watching: recheck after 24 hours"
run = [] # the trailing run of transient failures only
for h in reversed(history):
if h["category"] != "transient":
break
run.append(h)
run.reverse()
if len(run) >= 3 and now - t(run[0]) >= timedelta(days=7):
return "unreachable for a week: review"
return "watching: transient failure"
The full tested version also contains the per-host limiter and the function that performs a request. That function sends HEAD, falls back to GET when a server rejects HEAD, follows redirects by hand so the filter runs on every hop, remembers whether a redirect was permanent, and closes the connection after reading only the status line and headers. The block above is an excerpt. It relies on imports, a TRACKING pattern for tracking and token links, and a DEAD_SET of categories (not_found, gone and dns) that are defined in the full version and not printed here.
The address check is not an SSRF boundary
safe_to_fetch resolves the host name once, checks the answers, and returns ok. The function that later makes the request connects by host name, which resolves it a second time. Between those two lookups the answer can change. A hostile DNS server can return a public address for the check and a private one, such as the cloud metadata address, for the connection. This is called DNS rebinding, and it passes the filter above without any of its rules firing. OWASP's cheat sheet makes the same point: validating the resolved addresses separately does not stop rebinding, and a second unchecked lookup between validation and connection can bypass the checks.
For that reason this simple filter cannot be deployed as the protection against SSRF, and "it refuses private addresses" is not a promise you can make about it. Treat it as a way to avoid wasted and obviously wrong requests. Two things do the real work:
- Network egress isolation. Run the checker where it has no route to internal networks or to the metadata address, enforced by the firewall, container network or cloud network rules, not by the script. If the process cannot reach a private address, a rebinding answer has nothing to hit.
- A client that connects to the address you checked. Resolve once, validate those addresses, and open the connection to that validated address, while keeping the original host name for the HTTP
Hostheader, the TLS server name and certificate verification. OWASP names this approach and points to curl's custom address resolution as an example. Python's standardhttp.clientdoes not do this for you, so it takes a custom connection or a client library that supports it, and the policy must also apply to every redirect hop and every retry.
Neither the egress rules nor a pinned client were built or tested for this guide. The tests below cover the filter's logic only.
The diagram at the top of this page shows what the history function does with three synthetic links. The first returns ok, then 404 on two days a day apart, and is reported as dead. The second returns 403 on every check and is never called dead. The third fails with server errors across a week and is sent for review.
What the tests showed
The address filter refused eight literal forms before any network use: a 10.x address, a 192.168.x address, a 172.16.x address, the link-local metadata address, loopback with a port, the IPv6 loopback, 0.0.0.0, and an address in the shared range 100.64.x. It refused a host name that resolved to a private address, file: and ftp: schemes, an address with a user name and password, and three kinds of tracking or token link. A name that did not resolve was reported as a DNS failure.
Against the local server, a 200 was ok, a 404 was not found, and a 410 was gone. A 301 followed to a good page was reported as moved permanently with the new address, while a 302 to the same page was reported as ok with no permanent address. A pair of pages redirecting to each other was reported as a loop. Both 403 and 401 were categorized as blocked. A 500 and a slow response that outran the timeout were transient. A server that rejected HEAD with 405 got exactly one HEAD and one GET, and the link was ok. A server that wrongly answered HEAD with 404 but served the page on GET was also ok. A redirect that pointed at the metadata address stopped at the first hop with a skipped result and no onward request.
The limiter waited the two seconds a 429 asked for before the next request to that host, and a one-hour Retry-After made the checker defer the host and send nothing. A date-form Retry-After thirty seconds ahead produced a thirty second gap. The robots rules blocked a disallowed path and allowed another.
The history function called a link dead only after two consecutive misses at least 24 hours apart, never after a single miss, and never for a blocked link, even one blocked for two weeks. A run of failures counted as unreachable only when the trailing run of transient failures had at least three checks and began at least a week earlier. Earlier successes do not hide a current outage. A link that returned ok, then failed on days 1, 4 and 9, was sent for review on day 10, as was a link that had been ok for 60 days before a nine-day outage. A link with a three-failure streak only two days old stayed under watch, and so did one whose earlier week-long outage had ended in a success before a single new failure.
A second round of tests ran the safe_to_fetch, categorize and decide code exactly as printed above, with a stand-in for DNS and no network. Ports such as :99999, :65536, :abc, :-1 and :0, and a malformed address http://[::1/x, each returned a skipped result instead of raising an exception, and the stand-in resolver was never called for them. Ports 8080 and 65535 and the default HTTPS port still passed, and the private-address, scheme, credential and tracking refusals were unchanged. The earlier version of the function raised a ValueError on :99999, and the earlier history function stayed on "watching" for the ok-then-outage history. The rebinding gap was also shown directly: a resolver that answers with a public address first and a private address on the next lookup gets an ok from the filter, and the next lookup is private. No real site and no real DNS server were used, and the egress isolation and pinned client from the section above were not built.
These tests cover the logic against a local imitation. They do not cover real sites, bot-blocking services, TLS oddities, very large link lists or pages that report "not found" inside a 200 response, which no status check can see. They also do not cover a hostile DNS server or a network where the checker can reach private addresses, which is why the address filter alone is not a defense.
Where a model helps, and where it should not
A model is a poor choice for the verdict, because the evidence is status codes and timing and rules do it exactly. It can help with the follow-up: suggesting candidate replacement pages for a dead link, or drafting a note to a person. Treat suggestions as unverified. Open the candidate, read it, and approve the edit as a diff. Do not let a model rewrite links on its own, since a plausible wrong replacement is worse than a visible broken one. For the schedule, wrap the job as described in the guide on scheduling agents with cron, and let a failing job halt and report rather than retry forever.
Failure cases
HEAD handled badly. Some servers reject or mishandle HEAD. The fallback to GET covers it, at the cost of a second request.
Soft not-found pages. A page can return 200 and say it is gone. The checker will call it ok.
Bot blocking. Some hosts block scripts and return 403 to anything that is not a browser. The checker reports blocked and does not guess. Keep an exclusion list for hosts you verify by hand.
Rate limits. A fast loop across a few large hosts looks like abuse. Spread requests and respect the pause.
Fragments. The part after a hash is handled in the browser. A status check cannot tell you that a heading anchor is missing.
Private or tokenized links. Links in your own content may include tokens or internal addresses. Skipping them is deliberate, and they need a different kind of check.
DNS rebinding. A host name can resolve to a public address for the check and to a private one for the request. Isolate the checker's network and connect to the address you checked.
Silent success. A job that exits quietly with a stale link list looks healthy. Use the last success time and the monitoring approach in the guide on monitoring without babysitting.
Next step
Export the outbound links from five of your own pages. Run them through the filter and categorizer by hand, with a pause between requests and a user agent that names you. Look at every result that is not ok and decide, for each, which row of the matrix it belongs to. Then store the results with dates so the decision function has history to work with.
Sources
- RFC 9110, HTTP Semantics: the meaning of 301, 308, 401, 403, 404, 405 and 410, the caution about replacing references, and the
Retry-Afterheader. - RFC 6585, Additional HTTP Status Codes: the 429 Too Many Requests status code and its optional
Retry-After. - RFC 9309, Robots Exclusion Protocol: the rules are not access authorization, and how unavailable and unreachable robots.txt files are treated.
- OWASP, Server-Side Request Forgery Prevention Cheat Sheet: requests to arbitrary external addresses, why allowlists do not fit, the private, loopback and link-local ranges to refuse, redirect handling, DNS rebinding between validation and connection, and connecting to a validated address while keeping the host name for
Host, SNI and certificate checks.

