Skip to content
HALOWERK

Datenwerk

news cluster

For up to 500 supplied articles (title and text, at most 1 MB in total) it builds word 5-shingles (configurable 3-8) and finds near-duplicates by exact Jaccard similarity via an inverted shingle index, then clusters articles by single-link union-find over TF-IDF cosine similarity (log term frequency, smoothed inverse document frequency, German and English stop words removed); duplicates always share a cluster. Each cluster gets its size, members, top terms, a representative article (highest.

POST https://daten.halowerk.com/v1/news-cluster

Inputs

Fields of the published input schema
FieldTypeRequiredDescription
articlesarrayYes
duplicate_thresholdnumberNoJaccard similarity of shingle sets from which two articles count as near-duplicates.
cluster_thresholdnumberNoTF-IDF cosine similarity from which two articles are linked.
shingle_sizeintegerNo
Show full input schema
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "articles"
  ],
  "properties": {
    "articles": {
      "type": "array",
      "minItems": 2,
      "maxItems": 500,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": [
          "title"
        ],
        "properties": {
          "id": {
            "type": "string",
            "maxLength": 100
          },
          "title": {
            "type": "string",
            "minLength": 1,
            "maxLength": 500
          },
          "text": {
            "type": "string",
            "maxLength": 200000
          },
          "url": {
            "type": "string",
            "maxLength": 2000
          },
          "published": {
            "type": "string",
            "maxLength": 40
          },
          "source": {
            "type": "string",
            "maxLength": 200
          }
        }
      }
    },
    "duplicate_threshold": {
      "type": "number",
      "minimum": 0.3,
      "maximum": 1,
      "default": 0.6,
      "description": "Jaccard similarity of shingle sets from which two articles count as near-duplicates."
    },
    "cluster_threshold": {
      "type": "number",
      "minimum": 0.05,
      "maximum": 0.95,
      "default": 0.3,
      "description": "TF-IDF cosine similarity from which two articles are linked."
    },
    "shingle_size": {
      "type": "integer",
      "minimum": 3,
      "maximum": 8,
      "default": 5
    }
  }
}

Open original schema

Published input example

{
  "articles": [
    {
      "id": "a",
      "title": "Bundestag beschließt Wehrdienstgesetz",
      "text": "Der Bundestag hat das Gesetz beschlossen."
    },
    {
      "id": "b",
      "title": "Bahn erhöht Preise",
      "text": "Die Bahn erhöht die Preise im Fernverkehr."
    }
  ]
}

Result & documentation

Machine-readable documentation of this hub

Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.

More endpoints in this hub

All endpoints in this hub →