Skip to content
HALOWERK

Denkwerk

retrieval testset

Without a language model: for every passage the most distinctive sentence (highest mean IDF over the corpus, at least min_sentence_words words) becomes an inverse-cloze query and is removed from its target passage before evaluation (Lee, Chang and Toutanova 2019, inverse cloze task); in addition the top TF-IDF terms of the passage form a keyword query. Each pair is checked against a BM25 index over all passages (corpus statistics from the full passages; the target is scored without its query.

POST https://denk.halowerk.com/v1/retrieval-testset

Inputs

Fields of the published input schema
FieldTypeRequiredDescription
passagesarrayYesStrings or objects {id, text}.
methodsarrayNo
keyword_countintegerNo
min_sentence_wordsintegerNo
stemmingstringNo
Show full input schema
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "passages"
  ],
  "properties": {
    "passages": {
      "type": "array",
      "minItems": 1,
      "maxItems": 1000,
      "description": "Strings or objects {id, text}.",
      "items": {
        "anyOf": [
          {
            "type": "string",
            "minLength": 1,
            "maxLength": 20000
          },
          {
            "type": "object",
            "additionalProperties": false,
            "required": [
              "text"
            ],
            "properties": {
              "id": {
                "type": "string",
                "maxLength": 128
              },
              "text": {
                "type": "string",
                "minLength": 1,
                "maxLength": 20000
              },
              "score": {
                "type": "number"
              }
            }
          }
        ]
      }
    },
    "methods": {
      "type": "array",
      "minItems": 1,
      "uniqueItems": true,
      "items": {
        "type": "string",
        "enum": [
          "inverse_cloze",
          "keywords"
        ]
      },
      "default": [
        "inverse_cloze",
        "keywords"
      ]
    },
    "keyword_count": {
      "type": "integer",
      "minimum": 2,
      "maximum": 8,
      "default": 3
    },
    "min_sentence_words": {
      "type": "integer",
      "minimum": 3,
      "maximum": 30,
      "default": 5
    },
    "stemming": {
      "type": "string",
      "enum": [
        "none",
        "german"
      ],
      "default": "none"
    }
  }
}

Open original schema

Published input example

{
  "passages": [
    {
      "id": "a",
      "text": "Die Kündigungsfrist beträgt drei Monate. Sie beginnt am Monatsende."
    },
    {
      "id": "b",
      "text": "Die Kaution beträgt drei Monatsmieten. Sie wird verzinst."
    }
  ]
}

Result & documentation

Machine-readable documentation of this hub

Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.

More endpoints in this hub

All endpoints in this hub →