Skip to content
HALOWERK

Denkwerk

quantization advice

For every llama.cpp file type, weights = parameters × effective bits per weight (measured on Llama-3.1-8B, tools/quantize/README.md, commit ad1b88c) plus KV cache and overhead (same formulas as vram-estimate) are compared with the memory budget. Quality comes from the perplexity increase that llama.cpp publishes per type in tools/quantize/quantize.cpp (Llama-3-8B, commit 732707d): tiers near_lossless ≤ 0.03, high ≤ 0.2, balanced ≤ 0.7, low above (tier limits are this service's convention)..

POST https://denk.halowerk.com/v1/quantization-advice

Inputs

Fields of the published input schema
FieldTypeRequiredDescription
hf_modelstringNoHugging Face model id; config.json and the safetensors parameter count are read from the Hub.
params_bnumberNoTotal parameters in billions (overrides the Hub value).
n_layersintegerNoNumber of transformer layers.
n_headsintegerNoAttention (query) heads.
n_kv_headsintegerNoKey/value heads (GQA); defaults to n_heads.
head_dimintegerNoDimension per head; defaults to hidden_size / n_heads.
hidden_sizeintegerNoModel width.
vocab_sizeintegerNoVocabulary size (for the compute buffer).
gpustringNo
vram_gibnumberNoUsable memory per GPU in GiB.
n_gpusintegerNo
ram_gibnumberNoSystem RAM available for offloaded layers.
context_tokensintegerNo
batchintegerNo
kv_cache_typestringNo
overhead_gibnumberNo
min_qualitystringNo
allow_imatrixbooleanNoAllow types that require an importance matrix.
Show full input schema
{
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "hf_model": {
      "type": "string",
      "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]{0,95}/[A-Za-z0-9][A-Za-z0-9._-]{0,95}$",
      "description": "Hugging Face model id; config.json and the safetensors parameter count are read from the Hub."
    },
    "params_b": {
      "type": "number",
      "exclusiveMinimum": 0,
      "maximum": 5000,
      "description": "Total parameters in billions (overrides the Hub value)."
    },
    "n_layers": {
      "type": "integer",
      "minimum": 1,
      "maximum": 1000,
      "description": "Number of transformer layers."
    },
    "n_heads": {
      "type": "integer",
      "minimum": 1,
      "maximum": 1024,
      "description": "Attention (query) heads."
    },
    "n_kv_heads": {
      "type": "integer",
      "minimum": 1,
      "maximum": 1024,
      "description": "Key/value heads (GQA); defaults to n_heads."
    },
    "head_dim": {
      "type": "integer",
      "minimum": 1,
      "maximum": 4096,
      "description": "Dimension per head; defaults to hidden_size / n_heads."
    },
    "hidden_size": {
      "type": "integer",
      "minimum": 1,
      "maximum": 131072,
      "description": "Model width."
    },
    "vocab_size": {
      "type": "integer",
      "minimum": 1,
      "maximum": 2000000,
      "description": "Vocabulary size (for the compute buffer)."
    },
    "gpu": {
      "type": "string",
      "enum": [
        "h100-sxm",
        "h100-nvl",
        "h200-sxm",
        "h200-nvl",
        "a100-80gb-sxm",
        "a100-80gb-pcie",
        "a100-40gb-sxm",
        "a100-40gb-pcie",
        "l40s",
        "l40",
        "l4",
        "a10",
        "t4",
        "rtx-a6000",
        "rtx-4090",
        "rtx-3090",
        "apple-m2-ultra",
        "apple-m4-max",
        "apple-m4-pro"
      ]
    },
    "vram_gib": {
      "type": "number",
      "exclusiveMinimum": 0,
      "maximum": 100000,
      "description": "Usable memory per GPU in GiB."
    },
    "n_gpus": {
      "type": "integer",
      "minimum": 1,
      "maximum": 64,
      "default": 1
    },
    "ram_gib": {
      "type": "number",
      "minimum": 0,
      "maximum": 100000,
      "description": "System RAM available for offloaded layers."
    },
    "context_tokens": {
      "type": "integer",
      "minimum": 1,
      "maximum": 4194304,
      "default": 4096
    },
    "batch": {
      "type": "integer",
      "minimum": 1,
      "maximum": 4096,
      "default": 1
    },
    "kv_cache_type": {
      "type": "string",
      "enum": [
        "f32",
        "f16",
        "bf16",
        "fp8",
        "q8_0",
        "q5_1",
        "q5_0",
        "q4_1",
        "q4_0",
        "iq4_nl"
      ],
      "default": "f16"
    },
    "overhead_gib": {
      "type": "number",
      "minimum": 0,
      "maximum": 64,
      "default": 0.5
    },
    "min_quality": {
      "type": "string",
      "enum": [
        "near_lossless",
        "high",
        "balanced",
        "any"
      ],
      "default": "balanced"
    },
    "allow_imatrix": {
      "type": "boolean",
      "default": true,
      "description": "Allow types that require an importance matrix."
    }
  }
}

Open original schema

Published input example

{
  "params_b": 8.03,
  "n_layers": 32,
  "n_kv_heads": 8,
  "head_dim": 128,
  "vram_gib": 8,
  "min_quality": "high"
}

Result & documentation

Machine-readable documentation of this hub

Source of this page: the published HALOWERK catalog. Updated: 2026-11-17 UTC.

More endpoints in this hub

All endpoints in this hub →