Skip to content
HALOWERK

Denkwerk

near dup remove

Each document becomes a set of shingles (word n-grams after Unicode word segmentation and lower casing, or character n-grams). MinHash signatures use one 32-bit FNV-1a hash per shingle and num_perm deterministic MurmurHash3-finalizer permutations; locality-sensitive hashing with b bands of r rows (the largest r for which a pair exactly at the threshold is missed with at most 5 % probability) proposes candidate pairs, which are then verified with the exact Jaccard similarity of the shingle.

POST https://denk.halowerk.com/v1/near-dup-remove

Inputs

Fields of the published input schema
FieldTypeRequiredDescription
documentsarrayYes
thresholdnumberNoJaccard similarity at or above which documents count as duplicates.
shinglestringNo
nintegerNo
num_permintegerNo
keepstringNo
Show full input schema
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "documents"
  ],
  "properties": {
    "documents": {
      "type": "array",
      "minItems": 2,
      "maxItems": 3000,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": [
          "text"
        ],
        "properties": {
          "id": {
            "type": "string",
            "maxLength": 128
          },
          "text": {
            "type": "string",
            "minLength": 1,
            "maxLength": 200000
          }
        }
      }
    },
    "threshold": {
      "type": "number",
      "minimum": 0.1,
      "maximum": 1,
      "default": 0.8,
      "description": "Jaccard similarity at or above which documents count as duplicates."
    },
    "shingle": {
      "type": "string",
      "enum": [
        "word",
        "char"
      ],
      "default": "word"
    },
    "n": {
      "type": "integer",
      "minimum": 1,
      "maximum": 20,
      "default": 3
    },
    "num_perm": {
      "type": "integer",
      "minimum": 16,
      "maximum": 512,
      "default": 128
    },
    "keep": {
      "type": "string",
      "enum": [
        "first",
        "longest"
      ],
      "default": "first"
    }
  }
}

Open original schema

Published input example

{
  "documents": [
    {
      "id": "a",
      "text": "Die Lieferung erfolgt innerhalb von drei Werktagen nach Zahlungseingang."
    },
    {
      "id": "b",
      "text": "Die Lieferung erfolgt innerhalb von drei Werktagen nach Zahlungseingang!"
    }
  ],
  "threshold": 0.8
}

Result & documentation

Machine-readable documentation of this hub

Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.

More endpoints in this hub

All endpoints in this hub โ†’