Skip to content
HALOWERK

Denkwerk

tokenizer diff

The text is NFC-normalized, tokenized by each selected tokenizer and every token is mapped back to its exact UTF-8 bytes (tiktoken byte table; GPT-2 byte mapping for byte-level BPE; Metaspace and byte-fallback for SentencePiece-style vocabularies). Tokens that end inside a multi-byte character are marked partial and shown as hex. Agreement between two tokenizers is the Jaccard index of their internal token boundaries (byte offsets). Words (Unicode word segmentation) are listed with their.

POST https://denk.halowerk.com/v1/tokenizer-diff

Inputs

Fields of the published input schema
FieldTypeRequiredDescription
textstringYesUTF-8 text.
tokenizersarrayNoTokenizers to compare.
max_tokens_listedintegerNo
Show full input schema
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "text"
  ],
  "properties": {
    "text": {
      "type": "string",
      "minLength": 1,
      "maxLength": 20000,
      "description": "UTF-8 text."
    },
    "tokenizers": {
      "type": "array",
      "minItems": 2,
      "maxItems": 6,
      "uniqueItems": true,
      "items": {
        "type": "string",
        "enum": [
          "o200k_base",
          "cl100k_base",
          "p50k_base",
          "r50k_base",
          "qwen2.5",
          "deepseek-v3",
          "mistral-tekken",
          "falcon",
          "gpt-neox",
          "yi-1.5",
          "granite-3.3",
          "glm-4.5"
        ],
        "description": "Tokenizer id; see supported_tokenizers in the token-count-multi result."
      },
      "description": "Tokenizers to compare.",
      "default": [
        "o200k_base",
        "cl100k_base",
        "qwen2.5",
        "mistral-tekken"
      ]
    },
    "max_tokens_listed": {
      "type": "integer",
      "minimum": 0,
      "maximum": 2000,
      "default": 300
    }
  }
}

Open original schema

Published input example

{
  "text": "Lebensversicherungsgesellschaft antidisestablishmentarianism",
  "tokenizers": [
    "o200k_base",
    "qwen2.5"
  ]
}

Result & documentation

Machine-readable documentation of this hub

Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.

More endpoints in this hub

All endpoints in this hub โ†’