Research a topic with citation checking

Research this question: <QUESTION>. Scope: <SCOPE>. I will use the answer to <DECISION>.

Rules:
- Return a numbered list of claims. For each claim give the claim in one sentence, the source URL, and a verbatim quote of one or two sentences from that page that supports it.
- Prefer primary sources (the original study, official statistics, the organisation's own page) over articles about them.
- If you cannot find a source with a quote, write "no source found" for that claim. Do not guess a URL.
- Mark each source as primary, secondary or opinion.
- End with two short lists: "What I could not find" and "Where sources disagree".

id,url,quote
1,<URL_1>,"<QUOTE_1>"
2,<URL_2>,"<QUOTE_2>"

# check_claims.py: link check and quote check for the research workflow.
# Needs Python 3.9 or later and nothing else (standard library only).
# Usage:  python3 check_claims.py claims.csv
# claims.csv has three columns with a header row: id,url,quote
# It prints one line per claim and writes claims_results.csv next to claims.csv.
# "CHECK" means a person must read that source (workflow step 5).
import csv
import html
import re
import sys
import urllib.error
import urllib.request
from urllib.parse import urlparse

HEADERS = {"User-Agent": "Mozilla/5.0 (claim checker)"}


def words(text):
    """Lower-case letters and digits only, so quote marks, dashes and spacing never matter."""
    text = re.sub(r"(?is)<(script|style).*?</\1>", " ", text)
    text = re.sub(r"<[^>]+>", " ", text)
    text = html.unescape(text).lower()
    return " " + " ".join(re.findall(r"[a-z0-9]+", text)) + " "


def domain(url):
    host = urlparse(url).netloc.lower()
    return host[4:] if host.startswith("www.") else host


def fetch(url):
    request = urllib.request.Request(url, headers=HEADERS)
    try:
        with urllib.request.urlopen(request, timeout=20) as reply:
            return reply.status, reply.geturl(), reply.headers.get("Content-Type", ""), reply.read(3_000_000)
    except urllib.error.HTTPError as error:
        return error.code, url, "", b""
    except Exception as error:
        return "ERROR", url, str(error)[:80], b""


def check(url, quote):
    status, final, kind, body = fetch(url)
    same_site = domain(url) == domain(final)
    if status != 200:  # the page did not load (error, 403, 404 and so on)
        return status, final, same_site, "NOT LOADED: read it yourself", "CHECK"
    if "pdf" in kind.lower() or body[:4] == b"%PDF":
        return status, final, same_site, "PDF: read it yourself", "CHECK"
    page = words(body.decode("utf-8", errors="ignore"))
    wanted = words(quote).strip()
    if wanted and (" " + wanted + " ") in page:
        found = "FOUND"
    elif wanted and (" " + " ".join(wanted.split()[:8]) + " ") in page:
        found = "PARTIAL"
    else:
        found = "NOT FOUND"
    good = status == 200 and same_site and found == "FOUND"
    return status, final, same_site, found, "OK" if good else "CHECK"


def main(path):
    with open(path, newline="", encoding="utf-8") as handle:
        rows = list(csv.DictReader(handle))
    out_path = path.rsplit(".", 1)[0] + "_results.csv"
    flagged = 0
    with open(out_path, "w", newline="", encoding="utf-8") as handle:
        writer = csv.writer(handle)
        writer.writerow(["id", "status", "final_url", "same_site", "quote", "result"])
        for row in rows:
            status, final, same_site, found, result = check(row["url"].strip(), row["quote"])
            writer.writerow([row["id"], status, final, same_site, found, result])
            print(row["id"], status, "same site" if same_site else "OTHER SITE", found, result)
            flagged += result != "OK"
    print(f"{len(rows) - flagged} of {len(rows)} claims passed both checks. Results are in {out_path}")


if __name__ == "__main__":
    if len(sys.argv) != 2:
        sys.exit("Usage: python3 check_claims.py claims.csv")
    main(sys.argv[1])

Here is a brief and its main conclusion: <CONCLUSION>. Find the strongest evidence that this conclusion is wrong, overstated or out of date. Use the same rules: claim, URL and a verbatim quote for each. If you find nothing credible, say so plainly.

Question: <QUESTION>
Date: <DATE>     Decision it feeds: <DECISION>

Answer in three lines:
<ANSWER_LINE_1>
<ANSWER_LINE_2>
<ANSWER_LINE_3>

Claims we checked, each opened, with the quote found:
1. <CLAIM_1> ([<SOURCE_1_NAME>](<SOURCE_1_URL>))
2. <CLAIM_2> ([<SOURCE_2_NAME>](<SOURCE_2_URL>))

Evidence against, and what survived checking: <COUNTER_POINTS>
Read by hand because the script could not check them: <HAND_READ>
What we did not check: <NOT_CHECKED>