# Find near-duplicates and group a long list by meaning (embeddings)

Turn each item in a long list into numbers with an embeddings model, list the closest pairs as near-duplicate candidates, group the rest with a short numpy script, and read samples yourself before you name or count anything.

Workflows · v1.0 · draft · https://hayzlabs.com/workflows/group-a-list-by-meaning/

In short:
- Under about 500 items, paste chunks into an assistant for near-duplicate pairs. Otherwise get one embedding per item and run the numpy script.
- Calibrate the similarity threshold on 30 pairs you check by hand, and name each group from members you read yourself.
- Count nothing until at most 1 in 10 of the extra members you read in each group does not belong.

**Use it when** you have hundreds or thousands of short items, such as survey comments, support tickets, product names or notes, and you want to find the near-duplicates and see the main groups without reading every row first.

**Not when** you already know your labels. To sort items into categories you have defined, use [Run one prompt over hundreds of rows](https://hayzlabs.com/workflows/one-prompt-many-rows/). Not either when the list is under about 100 items: reading it is faster.

**You need** the list as a CSV with one item per row, an API key with a spending cap ([Your first API call, with a spending cap](https://hayzlabs.com/guides/first-api-call-with-a-spending-cap/)), Python 3 with numpy, and about two hours for 2,000 items. Remove names, emails and other personal details first: [What not to paste](https://hayzlabs.com/start/what-not-to-paste/). The steps, threshold and group counts are our suggestion.

## What an embedding is

An embeddings model turns each text into a list of numbers, and texts with similar meaning score close to 1 on cosine similarity (the range is -1 to 1).

Try step 4 below before you do it on your own list: move the threshold and see which pairs it would flag.

<!-- WIDGET thresholdExplorer -->

## Steps

**Under about 500 items, no code (our suggestion).** Paste the list into an assistant in chunks, with IDs, using the prompt below, then hand-check 30 of the pairs it returns as in step 4. You agree on at least 27 of 30, or you treat its list as a reading list. This finds near-duplicates within each chunk only, not across chunks, and it does not group the list; for that, use the steps in the table.

```text title="Near-duplicate pairs, no code · v1.0"
Below is part of a list. Each line starts with an ID, then the item.
List every pair of items that say the same thing in different words.
For each pair, quote both IDs and both texts exactly, closest first.
Do not group, merge or summarise the items. Only use IDs that appear
below. If there are no pairs, write "none".
ITEMS:
<CHUNK>
```

Replace the following:
- `<CHUNK>`: one chunk of rows from your cleaned `items.csv` (step 1), each with its ID.

| # | Step | Who | Time | Check |
|---|---|---|---|---|
| 1 | Clean the list. One item per row, drop blank rows, give each row an ID. Save it as `items.csv` with columns `id,text`. | Person | 20 min | Note the row count. Every ID is unique and no text is blank (the OpenAI API rejects empty strings). |
| 2 | Get one embedding per item from one vendor, with the snippet below. Use the same model for the whole list. | Script | 10 min | `vectors.npy` has exactly as many rows as `items.csv`. The cost printed is what you expected. |
| 3 | Run the script below with a starting threshold of 0.90. It writes every pair at or above it to `pairs.csv`, closest first. | Script | 2 min | The file opens and the top pairs look like the same thing said twice. |
| 4 | Calibrate the threshold on 30 pairs you check by hand: 10 from the top of `pairs.csv`, 10 just above the threshold and 10 just below (lower the threshold by 0.05 to see these). Mark each "same" or "different". Move the threshold to where your "same" pairs stop. | Person | 25 min | You agree with the threshold on at least 27 of 30 pairs. **If it fails:** the model may not separate your items well. Treat `pairs.csv` as a reading list, not a verdict. |
| 5 | Decide what to do with confirmed duplicates: keep one of each, or keep both and note it. Re-run with the new threshold and a number of groups, such as 10 for 2,000 items. | Person and script | 10 min | `groups.csv` has every ID exactly once. The script prints this check. |
| 6 | Name each group. Read the 5 random members the script prints for it and write a short name yourself. | Person | 20 min | The name fits all 5. **If it fails:** the group is mixed. Raise the number of groups and run again. |
| 7 | Read 10 more members of each group, including the 3 with the lowest `fit` score. Count the ones that do not belong. | Person | 30 min | At most 1 in 10 does not belong. **If it fails:** change the number of groups and repeat steps 6 and 7. A group of one or two odd items is fine; leave it as "other". |
| 8 | Record the model, the threshold, the number of groups and the date next to the files. | Person | 2 min | You could re-run it next month and compare. |

In step 6 you can ask a chat model to suggest a name, but give it only the 5 members you read, and you choose. A name taken from 5 members of a group of 300 is a guess about the other 295, which is what step 7 checks.

> [!OUR RULE]
> Count nothing until step 7 passes. Then report each group as "about N items", with the 5 examples you read, and say the groups came from an embeddings model and were named by a person.

## Get the embeddings

The cost is small next to your reading time; prices below are per million input tokens, from each vendor's page on 2026-10-20, and the snippet uses OpenAI's official Python SDK (`pip install openai numpy`) with the key in the `OPENAI_API_KEY` environment variable, as [Your first API call](https://hayzlabs.com/guides/first-api-call-with-a-spending-cap/) sets up.

| Vendor and model | Price per 1M tokens | 2,000 comments of about 50 tokens (100,000 tokens, illustrative) |
|---|---|---|
| OpenAI `text-embedding-3-small` (1,536 numbers per item by default) | $0.02 | about $0.002 |
| OpenAI `text-embedding-3-large` (3,072 by default) | $0.13 | about $0.013 |
| Google `gemini-embedding-2` text, paid tier (3,072 by default) | See Google's pricing page; batch is half the standard price | Check the current price first |
| Voyage AI `voyage-4` | $0.06, with 200 million free tokens per account | about $0.006, or nothing within the free tokens |

```python title="embed_openai.py · v1.0"
import csv
import numpy as np
from openai import OpenAI

PRICE_PER_MILLION = 0.02  # text-embedding-3-small, USD; check the vendor's model page first

client = OpenAI()  # reads the key from the OPENAI_API_KEY environment variable
with open("items.csv", newline="", encoding="utf-8") as f:
    texts = [row["text"] for row in csv.DictReader(f)]

vectors, tokens = [], 0
for start in range(0, len(texts), 500):  # 500 short items per request
    response = client.embeddings.create(model="text-embedding-3-small", input=texts[start:start + 500])
    vectors += [d.embedding for d in sorted(response.data, key=lambda d: d.index)]
    tokens += response.usage.total_tokens

np.save("vectors.npy", np.array(vectors))
print(f"{len(vectors)} vectors, {tokens} tokens, about ${tokens / 1e6 * PRICE_PER_MILLION:.4f}")
```

This snippet is our suggestion; we could not run it end to end without an API key.

## Find pairs and groups

This script uses numpy only and reads vectors from any vendor (`.npy`, or `.csv` with one vector per line). Scaled to length 1, a dot product is the cosine similarity. It groups with k-means from spread-out starting items and a fixed seed, so a re-run gives the same groups.

```python title="group_by_meaning.py · v1.2"
import csv, sys
import numpy as np

items_path, vec_path, threshold, k = sys.argv[1], sys.argv[2], float(sys.argv[3]), int(sys.argv[4])
with open(items_path, newline="", encoding="utf-8") as f:
    items = list(csv.DictReader(f))
X = np.load(vec_path) if vec_path.endswith(".npy") else np.loadtxt(vec_path, delimiter=",")
assert len(items) == len(X), f"{len(items)} items but {len(X)} vectors"
X = X / np.linalg.norm(X, axis=1, keepdims=True)
S = X @ X.T  # every pair's cosine similarity

def write(path, header, rows):
    with open(path, "w", newline="", encoding="utf-8") as f:
        csv.writer(f).writerows([header, *rows])

# 1. Near-duplicate candidates, closest first
i, j = np.triu_indices(len(X), k=1)
keep = S[i, j] >= threshold
pairs = sorted(zip(S[i, j][keep], i[keep], j[keep]), reverse=True)
write("pairs.csv", ["similarity", "id_a", "id_b", "text_a", "text_b"],
      [[f"{s:.3f}", items[a]["id"], items[b]["id"], items[a]["text"], items[b]["text"]] for s, a, b in pairs])

# 2. Groups: each starting centre is the item least like any centre so far
rng = np.random.default_rng(0)
picks = [int(rng.integers(len(X)))]
for _ in range(k - 1):
    picks.append(int(np.argmin(S[:, picks].max(axis=1))))
centres = X[picks]
for _ in range(100):
    label = np.argmax(X @ centres.T, axis=1)
    new = np.array([X[label == c].mean(0) if np.any(label == c) else centres[c] for c in range(k)])
    new /= np.linalg.norm(new, axis=1, keepdims=True)
    if np.allclose(new, centres):
        break
    centres = new
fit = np.sum(X * centres[label], axis=1)  # similarity of each item to its group centre
write("groups.csv", ["id", "group", "fit", "text"],
      [[items[n]["id"], int(label[n]), f"{fit[n]:.3f}", items[n]["text"]] for n in np.argsort(label, kind="stable")])

# 3. Checks, and 5 random members of each group to read
print(f"{len(items)} items, {len(pairs)} pairs at or above {threshold}")
print(f"every ID in exactly one group: {len(label) == len(items) == len({r['id'] for r in items})}")
for c in range(k):
    members = np.flatnonzero(label == c)
    print(f"\ngroup {c}: {len(members)} items")
    for n in rng.choice(members, min(5, len(members)), replace=False):
        print(f"  {items[n]['id']}  {items[n]['text'][:80]}")
```

The spread-out start can pick an odd one-off item as a centre; that shows up as a tiny group.

Run it as:

```text title="Run the script · v1.0"
python group_by_meaning.py <ITEMS_CSV> <VECTORS_FILE> <THRESHOLD> <GROUPS>
```

Replace the following:
- `<ITEMS_CSV>`: your cleaned list, such as `items.csv`.
- `<VECTORS_FILE>`: `vectors.npy` from the snippet, or a `.csv` of vectors from another tool.
- `<THRESHOLD>`: the near-duplicate cut-off, `0.90` to start, then your calibrated value from step 4.
- `<GROUPS>`: how many groups to make, such as `10`.

The 0.90 start is ours, not a vendor figure. Calibrate again whenever you change model; a threshold set on one model does not carry over (our suggestion). For 2,000 items the pairwise table is 4 million numbers, about 32 MB of memory. For lists in the tens of thousands, that table grows a hundredfold, so run the duplicate search in parts.

> [!CHECK]
> Before you trust any group, confirm three counts: rows in `items.csv`, rows in `vectors.npy`, and IDs in `groups.csv`. All three must match. A mismatch means items were dropped or shifted, and every group after that point is wrong.

Once the groups have names you trust, they can become the labels for new items, which is the job of [Run one prompt over hundreds of rows](https://hayzlabs.com/workflows/one-prompt-many-rows/). To check whether a different embeddings model groups your items better, treat it as you would any model switch: [Test any new model on your own tasks](https://hayzlabs.com/workflows/personal-eval-set/).

## How to tell it worked

- The three counts match and every item is in exactly one group.
- You agreed with the threshold on at least 27 of your 30 hand-checked pairs.
- Each group's name fits the 5 members you read, and at most 1 in 10 of the extra members you read does not belong.
- It has failed if you reported group sizes before reading samples, or if one group holds most of the list. Raise the number of groups and read again.

> [!LIMIT]
> Similar here means similar by the model's measure, not by yours. Read for comments on the same topic that say opposite things, since they may land in one group. K-means puts every item in some group, even an item that fits none, so a low `fit` score is a prompt to read, not a label. The number of groups is your choice and changes the result. Group sizes describe this list only, not the people who did not write in. And the embeddings vendor receives every item, so remove personal data first ([What not to paste](https://hayzlabs.com/start/what-not-to-paste/)) and check the vendor's data terms for the tier you use.

## Sources

1. OpenAI, Embeddings guide: definition, models, default dimensions, 8,192-token input limit. Vendor documentation. [Link](https://developers.openai.com/api/docs/guides/embeddings)
2. OpenAI, Create embeddings API reference: empty strings rejected, response fields. Vendor documentation. [Link](https://developers.openai.com/api/reference/resources/embeddings/methods/create)
3. OpenAI, model pages for `text-embedding-3-small` and `text-embedding-3-large`: prices. Vendor pricing. [Link](https://developers.openai.com/api/docs/models/text-embedding-3-small) and [Link](https://developers.openai.com/api/docs/models/text-embedding-3-large)
4. Google, Gemini API embeddings guide: models, dimensions, cosine similarity range. Vendor documentation. [Link](https://ai.google.dev/gemini-api/docs/embeddings)
5. Google, Gemini API pricing: `gemini-embedding-2` prices. Vendor pricing. [Link](https://ai.google.dev/gemini-api/docs/pricing)
6. Voyage AI, Pricing: `voyage-4` price and free tokens. Vendor pricing. [Link](https://docs.voyageai.com/docs/pricing)