Denkwerk
tokenizer diff
The text is NFC-normalized, tokenized by each selected tokenizer and every token is mapped back to its exact UTF-8 bytes (tiktoken byte table; GPT-2 byte mapping for byte-level BPE; Metaspace and byte-fallback for SentencePiece-style vocabularies). Tokens that end inside a multi-byte character are marked partial and shown as hex. Agreement between two tokenizers is the Jaccard index of their internal token boundaries (byte offsets). Words (Unicode word segmentation) are listed with their.
POST https://denk.halowerk.com/v1/tokenizer-diffInputs
| Field | Type | Required | Description |
|---|---|---|---|
| text | string | Yes | UTF-8 text. |
| tokenizers | array | No | Tokenizers to compare. |
| max_tokens_listed | integer | No |
Show full input schema
{
"type": "object",
"additionalProperties": false,
"required": [
"text"
],
"properties": {
"text": {
"type": "string",
"minLength": 1,
"maxLength": 20000,
"description": "UTF-8 text."
},
"tokenizers": {
"type": "array",
"minItems": 2,
"maxItems": 6,
"uniqueItems": true,
"items": {
"type": "string",
"enum": [
"o200k_base",
"cl100k_base",
"p50k_base",
"r50k_base",
"qwen2.5",
"deepseek-v3",
"mistral-tekken",
"falcon",
"gpt-neox",
"yi-1.5",
"granite-3.3",
"glm-4.5"
],
"description": "Tokenizer id; see supported_tokenizers in the token-count-multi result."
},
"description": "Tokenizers to compare.",
"default": [
"o200k_base",
"cl100k_base",
"qwen2.5",
"mistral-tekken"
]
},
"max_tokens_listed": {
"type": "integer",
"minimum": 0,
"maximum": 2000,
"default": 300
}
}
}Published input example
{
"text": "Lebensversicherungsgesellschaft antidisestablishmentarianism",
"tokenizers": [
"o200k_base",
"qwen2.5"
]
}Result & documentation
Machine-readable documentation of this hub
Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.
More endpoints in this hub
- Denkwerk0.003 USDC
agent card generate
POSThttps://denk.halowerk.com/v1/agent-card-generateBuilds an Agent2Agent protocol Agent Card with protocolVersion, name, description, url, version, capabilities, default input/output modes and skills. Skills come from the supplied list or are derived from OpenAPI operations (operationId, su
- Denkwerk0.003 USDC
agent permission diff
POSThttps://denk.halowerk.com/v1/agent-permission-diffTakes the tool names (with optional descriptions) and permission scopes of an agent before and after a change and reports added and removed tools and scopes, classifies new capabilities by risk (payments, sending messages, deleting or writi
- Denkwerk0.003 USDC
annotation guide draft
POSThttps://denk.halowerk.com/v1/annotation-guide-draftTakes example texts per label and derives, without a language model, the most distinguishing terms per label (TF-IDF contrast against the other labels), representative examples, cross-label example pairs with high similarity that need a wri