Skip to content
HALOWERK

Denkwerk

model swap regression

For each item the old and new output are scored with the selected metrics: exact_match (trimmed string equality), squad_em and token_f1 (official SQuAD v1.1 normalisation: lower case, ASCII punctuation and articles a/an/the removed; best over several references), chrf (chrF2 as in sacreBLEU 2.x: character n-grams 1-6 without whitespace, beta 2, best reference), json_valid (parses after stripping code fences), refusal (rule set of refusal-detect, lower is better). Paired comparison per.

POST https://denk.halowerk.com/v1/model-swap-regression

Inputs

Fields of the published input schema
FieldTypeRequiredDescription
itemsarrayYes
metricsarrayNoThe first metric is primary. Default: squad_em, token_f1, chrf with references; json_valid, refusal without.
tolerancenumberNoAbsolute score difference treated as a tie (chrF is on a 0-100 scale, the others 0-1).
alphanumberNo
max_regressionsintegerNo
Show full input schema
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "items"
  ],
  "properties": {
    "items": {
      "type": "array",
      "minItems": 1,
      "maxItems": 2000,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": [
          "old",
          "new"
        ],
        "properties": {
          "id": {
            "type": "string",
            "maxLength": 128
          },
          "old": {
            "type": "string",
            "maxLength": 20000
          },
          "new": {
            "type": "string",
            "maxLength": 20000
          },
          "reference": {
            "anyOf": [
              {
                "type": "string",
                "maxLength": 20000
              },
              {
                "type": "array",
                "minItems": 1,
                "maxItems": 10,
                "items": {
                  "type": "string",
                  "maxLength": 20000
                }
              }
            ]
          }
        }
      }
    },
    "metrics": {
      "type": "array",
      "minItems": 1,
      "maxItems": 6,
      "uniqueItems": true,
      "items": {
        "type": "string",
        "enum": [
          "exact_match",
          "squad_em",
          "token_f1",
          "chrf",
          "json_valid",
          "refusal"
        ]
      },
      "description": "The first metric is primary. Default: squad_em, token_f1, chrf with references; json_valid, refusal without."
    },
    "tolerance": {
      "type": "number",
      "minimum": 0,
      "maximum": 100,
      "default": 0,
      "description": "Absolute score difference treated as a tie (chrF is on a 0-100 scale, the others 0-1)."
    },
    "alpha": {
      "type": "number",
      "minimum": 0.0001,
      "maximum": 0.5,
      "default": 0.05
    },
    "max_regressions": {
      "type": "integer",
      "minimum": 0,
      "maximum": 200,
      "default": 20
    }
  }
}

Open original schema

Published input example

{
  "items": [
    {
      "id": "q1",
      "reference": "Paris",
      "old": "Paris",
      "new": "The capital is Paris."
    },
    {
      "id": "q2",
      "reference": "1889",
      "old": "1889",
      "new": "1887"
    }
  ]
}

Result & documentation

Machine-readable documentation of this hub

Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.

More endpoints in this hub

All endpoints in this hub โ†’