Llm Integrator

Learn about configurations >

• accelerator | string

  Default: cpu

  Hardware the model runs on: "cpu" or "nvidia-gpu". Selects the default runtime image, resource defaults and vLLM settings. With "nvidia-gpu" the cluster must advertise nvidia.com/gpu (e.g. via the NVIDIA GPU operator) and GPU nodes must not be tainted.

• autoscaling-metric | string

  Default: num-requests

  vLLM metric the workers scale on while autoscaling: "num-requests" (requests being processed plus queued), "num-requests-waiting" (queued requests only) or "kv-cache-usage" (KV-cache utilisation, 0-1). Replicas are set so the metric per replica stays at autoscaling-target.

• autoscaling-polling-interval | int

  Default: 15

  Seconds between KEDA's queries of the autoscaling metric.

• autoscaling-scale-down-delay | int

  Default: 300

  Seconds the metric must stay low before a replica is removed, between 0 and 3600. Keeps loaded models from being dropped on short dips in load.

• autoscaling-target | float

  Target value of autoscaling-metric per replica: above 0 for num-requests and num-requests-waiting, between 0 and 1 for kv-cache-usage. 0 uses the metric's default: for num-requests 2 on "cpu" and 16 on "nvidia-gpu", for num-requests-waiting 2, for kv-cache-usage 0.8.

• cpu-limit | string

  CPU limit per worker, as a Kubernetes quantity. When empty, the default for the selected accelerator is used.

• cpu-request | string

  CPU request per worker, as a Kubernetes quantity (e.g. "4" or "500m"). When empty, the default for the selected accelerator is used.

• enable-prefill-decode | boolean

  Deploy the model in prefill/decode (disaggregated) mode, adding a dedicated prefill worker alongside the decode worker. The decode worker recomputes the prompt (no KV-cache transfer). With "nvidia-gpu" the cluster must fit two workers of gpu-count GPUs each. When false, the model is deployed as a single colocated worker.

• gpu-count | int

  Default: 1

  Number of GPUs per worker pod (nvidia.com/gpu); must be at least 1 when accelerator is "nvidia-gpu". The model is split across all of them with vLLM tensor parallelism (--tensor-parallel-size). Ignored when accelerator is "cpu".

• gpu-memory-utilization | float

  Fraction of GPU memory vLLM may use, in (0, 1], passed as --gpu-memory-utilization. 0 keeps vLLM's default (0.9). Required when several models share one GPU (time-slicing or MPS). Ignored when accelerator is "cpu".

• hf-token-secret | secret

  URI of a Juju user secret holding the Hugging Face access token used to download gated or private hf:// models. The secret must expose the token under a "token" key. Create and grant it before configuring, e.g. juju add-secret my-hf-token token=hf_xxx then juju grant-secret my-hf-token llm-integrator, and set this option to the resulting secret:... URI. Only used with hf:// model URIs; leave unset for public models.

• max-model-len | int

  Maximum model context length passed to vLLM as --max-model-len. 0 keeps the model's own value. Useful to fit large-context models on smaller GPUs.

• max-replicas | int

  Default: 1

  Maximum number of replicas of each worker. Equal to min-replicas runs a fixed number of replicas; above min-replicas autoscales the workers with KEDA on autoscaling-metric. Autoscaling requires relations to keda-controller (keda) and to the Prometheus that scrapes kserve-llmisvc's metrics-endpoint (prometheus-api); the model keeps serving while they are missing. In prefill/decode mode the decode and prefill workers are scaled independently with the same settings. The cluster must fit max-replicas workers for scale-up to complete. Limitations: Prometheus served over TLS is not supported yet, and the application name must be at most 39 characters (31 with enable-prefill-decode), as Kubernetes truncates longer pod names that the autoscaling query matches.

• memory-limit | string

  Memory limit per worker, as a Kubernetes quantity. When empty, the default for the selected accelerator is used.

• memory-request | string

  Memory request per worker, as a Kubernetes quantity (e.g. "16Gi"). When empty, the default for the selected accelerator is used.

• min-replicas | int

  Default: 1

  Number of replicas of each worker, or the minimum while autoscaling. Must be at least 1: vLLM reports no metrics without a running pod.

• model-name | string

  Optional name the served model is exposed as (spec.model.name of the LLMInferenceService), e.g. llama-3.1-8b. When left empty it defaults to the model reference derived from model-uri: the part after hf:// for Hugging Face URIs, or the last path segment of the bucket key for s3:// URIs.

• model-uri | string

  URI of the model to deploy. Two schemes are supported: Hugging Face URIs (hf://), e.g. hf://meta-llama/Llama-3.1-8B-Instruct, and S3 URIs (s3://), e.g. s3://my-bucket/models/llama-3.1-8b. When an s3:// URI is used the charm must be related to an s3-integrator over the s3-credentials endpoint to obtain the bucket credentials.

• runtime-image | string

  Optional OCI image override for the vLLM serving runtime container. When empty, the charm uses the vLLM image it ships for the selected accelerator. A custom image must match the accelerator (e.g. a CUDA build for nvidia-gpu); a mismatch is not detected and only fails at pod start.

• storage-initializer-image | string

  Optional OCI image override for the storage-initializer init container that downloads the model into the workload, for both hf:// and s3:// model URIs. When empty, the charm uses the storage-initializer image it ships.

• vllm-extra-args | string

  Extra arguments appended to vllm serve, e.g. "--dtype bfloat16 --enable-prefix-caching". Parsed with shell-like quoting but never run through a shell. Flags managed by the charm (--host, --port, --model, --served-model-name, --tensor-parallel-size, --pipeline-parallel-size, --data-parallel-size, --max-model-len, --gpu-memory-utilization) and --kv-transfer-config are rejected.