Find near-duplicates and group a long list by meaning (embeddings)

Use it when you have hundreds or thousands of short items, such as survey comments, support tickets, product names or notes, and you want to find the near-duplicates and see the main groups without reading every row first.

Not when you already know your labels. To sort items into categories you have defined, use Run one prompt over hundreds of rows. Not either when the list is under about 100 items: reading it is faster.

You need the list as a CSV with one item per row, an API key with a spending cap (Your first API call, with a spending cap), Python 3 with numpy, and about two hours for 2,000 items. Remove names, emails and other personal details first: What not to paste. The steps, threshold and group counts are our suggestion.

What an embedding is

An embeddings model turns each text into a list of numbers, and texts with similar meaning score close to 1 on cosine similarity (the range is -1 to 1).

Try step 4 below before you do it on your own list: move the threshold and see which pairs it would flag.

Steps

Under about 500 items, no code (our suggestion). Paste the list into an assistant in chunks, with IDs, using the prompt below, then hand-check 30 of the pairs it returns as in step 4. You agree on at least 27 of 30, or you treat its list as a reading list. This finds near-duplicates within each chunk only, not across chunks, and it does not group the list; for that, use the steps in the table.

Below is part of a list. Each line starts with an ID, then the item.
List every pair of items that say the same thing in different words.
For each pair, quote both IDs and both texts exactly, closest first.
Do not group, merge or summarise the items. Only use IDs that appear
below. If there are no pairs, write "none".
ITEMS:
<CHUNK>

Replace the following:

• <CHUNK>: one chunk of rows from your cleaned items.csv (step 1), each with its ID.

In step 6 you can ask a chat model to suggest a name, but give it only the 5 members you read, and you choose. A name taken from 5 members of a group of 300 is a guess about the other 295, which is what step 7 checks.

Get the embeddings

The cost is small next to your reading time; prices below are per million input tokens, from each vendor's page on 2026-10-20, and the snippet uses OpenAI's official Python SDK (pip install openai numpy) with the key in the OPENAI_API_KEY environment variable, as Your first API call sets up.

import csv
import numpy as np
from openai import OpenAI

PRICE_PER_MILLION = 0.02  # text-embedding-3-small, USD; check the vendor's model page first

client = OpenAI()  # reads the key from the OPENAI_API_KEY environment variable
with open("items.csv", newline="", encoding="utf-8") as f:
    texts = [row["text"] for row in csv.DictReader(f)]

vectors, tokens = [], 0
for start in range(0, len(texts), 500):  # 500 short items per request
    response = client.embeddings.create(model="text-embedding-3-small", input=texts[start:start + 500])
    vectors += [d.embedding for d in sorted(response.data, key=lambda d: d.index)]
    tokens += response.usage.total_tokens

np.save("vectors.npy", np.array(vectors))
print(f"{len(vectors)} vectors, {tokens} tokens, about ${tokens / 1e6 * PRICE_PER_MILLION:.4f}")

This snippet is our suggestion; we could not run it end to end without an API key.

Find pairs and groups

This script uses numpy only and reads vectors from any vendor (.npy, or .csv with one vector per line). Scaled to length 1, a dot product is the cosine similarity. It groups with k-means from spread-out starting items and a fixed seed, so a re-run gives the same groups.

import csv, sys
import numpy as np

items_path, vec_path, threshold, k = sys.argv[1], sys.argv[2], float(sys.argv[3]), int(sys.argv[4])
with open(items_path, newline="", encoding="utf-8") as f:
    items = list(csv.DictReader(f))
X = np.load(vec_path) if vec_path.endswith(".npy") else np.loadtxt(vec_path, delimiter=",")
assert len(items) == len(X), f"{len(items)} items but {len(X)} vectors"
X = X / np.linalg.norm(X, axis=1, keepdims=True)
S = X @ X.T  # every pair's cosine similarity

def write(path, header, rows):
    with open(path, "w", newline="", encoding="utf-8") as f:
        csv.writer(f).writerows([header, *rows])

# 1. Near-duplicate candidates, closest first
i, j = np.triu_indices(len(X), k=1)
keep = S[i, j] >= threshold
pairs = sorted(zip(S[i, j][keep], i[keep], j[keep]), reverse=True)
write("pairs.csv", ["similarity", "id_a", "id_b", "text_a", "text_b"],
      [[f"{s:.3f}", items[a]["id"], items[b]["id"], items[a]["text"], items[b]["text"]] for s, a, b in pairs])

# 2. Groups: each starting centre is the item least like any centre so far
rng = np.random.default_rng(0)
picks = [int(rng.integers(len(X)))]
for _ in range(k - 1):
    picks.append(int(np.argmin(S[:, picks].max(axis=1))))
centres = X[picks]
for _ in range(100):
    label = np.argmax(X @ centres.T, axis=1)
    new = np.array([X[label == c].mean(0) if np.any(label == c) else centres[c] for c in range(k)])
    new /= np.linalg.norm(new, axis=1, keepdims=True)
    if np.allclose(new, centres):
        break
    centres = new
fit = np.sum(X * centres[label], axis=1)  # similarity of each item to its group centre
write("groups.csv", ["id", "group", "fit", "text"],
      [[items[n]["id"], int(label[n]), f"{fit[n]:.3f}", items[n]["text"]] for n in np.argsort(label, kind="stable")])

# 3. Checks, and 5 random members of each group to read
print(f"{len(items)} items, {len(pairs)} pairs at or above {threshold}")
print(f"every ID in exactly one group: {len(label) == len(items) == len({r['id'] for r in items})}")
for c in range(k):
    members = np.flatnonzero(label == c)
    print(f"\ngroup {c}: {len(members)} items")
    for n in rng.choice(members, min(5, len(members)), replace=False):
        print(f"  {items[n]['id']}  {items[n]['text'][:80]}")

The spread-out start can pick an odd one-off item as a centre; that shows up as a tiny group.

Run it as:

python group_by_meaning.py <ITEMS_CSV> <VECTORS_FILE> <THRESHOLD> <GROUPS>

Replace the following:

• <ITEMS_CSV>: your cleaned list, such as items.csv.
• <VECTORS_FILE>: vectors.npy from the snippet, or a .csv of vectors from another tool.
• <THRESHOLD>: the near-duplicate cut-off, 0.90 to start, then your calibrated value from step 4.
• <GROUPS>: how many groups to make, such as 10.

The 0.90 start is ours, not a vendor figure. Calibrate again whenever you change model; a threshold set on one model does not carry over (our suggestion). For 2,000 items the pairwise table is 4 million numbers, about 32 MB of memory. For lists in the tens of thousands, that table grows a hundredfold, so run the duplicate search in parts.

Once the groups have names you trust, they can become the labels for new items, which is the job of Run one prompt over hundreds of rows. To check whether a different embeddings model groups your items better, treat it as you would any model switch: Test any new model on your own tasks.

How to tell it worked

• The three counts match and every item is in exactly one group.
• You agreed with the threshold on at least 27 of your 30 hand-checked pairs.
• Each group's name fits the 5 members you read, and at most 1 in 10 of the extra members you read does not belong.
• It has failed if you reported group sizes before reading samples, or if one group holds most of the list. Raise the number of groups and read again.

Sources

1. OpenAI, Embeddings guide: definition, models, default dimensions, 8,192-token input limit. Vendor documentation. Link
2. OpenAI, Create embeddings API reference: empty strings rejected, response fields. Vendor documentation. Link
3. OpenAI, model pages for text-embedding-3-small and text-embedding-3-large: prices. Vendor pricing. Link and Link
4. Google, Gemini API embeddings guide: models, dimensions, cosine similarity range. Vendor documentation. Link
5. Google, Gemini API pricing: gemini-embedding-2 prices. Vendor pricing. Link
6. Voyage AI, Pricing: voyage-4 price and free tokens. Vendor pricing. Link

Changes

1. v1.010 Oct 2026first version