Find the Near-Duplicates

Run the deduplication method web-scale datasets use (shingles, MinHash and banded hashing) on a dozen made-up web pages, and tune what counts as a copy.

From Chapter 9: How an LLM Is Actually Built