--- title: Charmhub | Deploy Llm Integrator using Charmhub - The Open Operator Collection description: Deploy the latest version of Llm Integrator on any cloud. url: https://charmhub.io/llm-integrator/configurations --- # Llm Integrator [Michal Hucko](https://charmhub.io/publisher/michalhucko "View all packages from Michal Hucko") * [Michal Hucko](https://charmhub.io/publisher/michalhucko "View all packages from Michal Hucko") Platform: 24.04 edge 166 ``` juju deploy llm-integrator --channel edge ``` [Learn to deploy on juju >](https://juju.is/docs/juju/manage-applications) * [accelerator](https://charmhub.io/llm-integrator/configurations#accelerator) * [autoscaling-metric](https://charmhub.io/llm-integrator/configurations#autoscaling-metric) * [autoscaling-polling-interval](https://charmhub.io/llm-integrator/configurations#autoscaling-polling-interval) * [autoscaling-scale-down-delay](https://charmhub.io/llm-integrator/configurations#autoscaling-scale-down-delay) * [autoscaling-target](https://charmhub.io/llm-integrator/configurations#autoscaling-target) * [cpu-limit](https://charmhub.io/llm-integrator/configurations#cpu-limit) * [cpu-request](https://charmhub.io/llm-integrator/configurations#cpu-request) * [enable-prefill-decode](https://charmhub.io/llm-integrator/configurations#enable-prefill-decode) * [gpu-count](https://charmhub.io/llm-integrator/configurations#gpu-count) * [gpu-memory-utilization](https://charmhub.io/llm-integrator/configurations#gpu-memory-utilization) * [hf-token-secret](https://charmhub.io/llm-integrator/configurations#hf-token-secret) * [max-model-len](https://charmhub.io/llm-integrator/configurations#max-model-len) * [max-replicas](https://charmhub.io/llm-integrator/configurations#max-replicas) * [memory-limit](https://charmhub.io/llm-integrator/configurations#memory-limit) * [memory-request](https://charmhub.io/llm-integrator/configurations#memory-request) * [min-replicas](https://charmhub.io/llm-integrator/configurations#min-replicas) * [model-name](https://charmhub.io/llm-integrator/configurations#model-name) * [model-uri](https://charmhub.io/llm-integrator/configurations#model-uri) * [runtime-image](https://charmhub.io/llm-integrator/configurations#runtime-image) * [storage-initializer-image](https://charmhub.io/llm-integrator/configurations#storage-initializer-image) * [vllm-extra-args](https://charmhub.io/llm-integrator/configurations#vllm-extra-args) [Learn about configurations >](https://juju.is/docs/juju/configuration#heading--application-configuration) * accelerator | string Default: cpu Hardware the model runs on: "cpu" or "nvidia-gpu". Selects the default runtime image, resource defaults and vLLM settings. With "nvidia-gpu" the cluster must advertise nvidia.com/gpu (e.g. via the NVIDIA GPU operator) and GPU nodes must not be tainted. * autoscaling-metric | string Default: num-requests vLLM metric the workers scale on while autoscaling: "num-requests" (requests being processed plus queued), "num-requests-waiting" (queued requests only) or "kv-cache-usage" (KV-cache utilisation, 0-1). Replicas are set so the metric per replica stays at autoscaling-target. * autoscaling-polling-interval | int Default: 15 Seconds between KEDA's queries of the autoscaling metric. * autoscaling-scale-down-delay | int Default: 300 Seconds the metric must stay low before a replica is removed, between 0 and 3600. Keeps loaded models from being dropped on short dips in load. * autoscaling-target | float Target value of autoscaling-metric per replica: above 0 for num-requests and num-requests-waiting, between 0 and 1 for kv-cache-usage. 0 uses the metric's default: for num-requests 2 on "cpu" and 16 on "nvidia-gpu", for num-requests-waiting 2, for kv-cache-usage 0.8. * cpu-limit | string CPU limit per worker, as a Kubernetes quantity. When empty, the default for the selected accelerator is used. * cpu-request | string CPU request per worker, as a Kubernetes quantity (e.g. "4" or "500m"). When empty, the default for the selected accelerator is used. * enable-prefill-decode | boolean Deploy the model in prefill/decode (disaggregated) mode, adding a dedicated prefill worker alongside the decode worker. The decode worker recomputes the prompt (no KV-cache transfer). With "nvidia-gpu" the cluster must fit two workers of gpu-count GPUs each. When false, the model is deployed as a single colocated worker. * gpu-count | int Default: 1 Number of GPUs per worker pod (nvidia.com/gpu); must be at least 1 when accelerator is "nvidia-gpu". The model is split across all of them with vLLM tensor parallelism (--tensor-parallel-size). Ignored when accelerator is "cpu". * gpu-memory-utilization | float Fraction of GPU memory vLLM may use, in (0, 1], passed as --gpu-memory-utilization. 0 keeps vLLM's default (0.9). Required when several models share one GPU (time-slicing or MPS). Ignored when accelerator is "cpu". * hf-token-secret | secret URI of a Juju user secret holding the Hugging Face access token used to download gated or private hf:// models. The secret must expose the token under a "token" key. Create and grant it before configuring, e.g. `juju add-secret my-hf-token token=hf_xxx` then `juju grant-secret my-hf-token llm-integrator`, and set this option to the resulting `secret:...` URI. Only used with hf:// model URIs; leave unset for public models. * max-model-len | int Maximum model context length passed to vLLM as --max-model-len. 0 keeps the model's own value. Useful to fit large-context models on smaller GPUs. * max-replicas | int Default: 1 Maximum number of replicas of each worker. Equal to min-replicas runs a fixed number of replicas; above min-replicas autoscales the workers with KEDA on autoscaling-metric. Autoscaling requires relations to keda-controller (keda) and to the Prometheus that scrapes kserve-llmisvc's metrics-endpoint (prometheus-api); the model keeps serving while they are missing. In prefill/decode mode the decode and prefill workers are scaled independently with the same settings. The cluster must fit max-replicas workers for scale-up to complete. Limitations: Prometheus served over TLS is not supported yet, and the application name must be at most 39 characters (31 with enable-prefill-decode), as Kubernetes truncates longer pod names that the autoscaling query matches. * memory-limit | string Memory limit per worker, as a Kubernetes quantity. When empty, the default for the selected accelerator is used. * memory-request | string Memory request per worker, as a Kubernetes quantity (e.g. "16Gi"). When empty, the default for the selected accelerator is used. * min-replicas | int Default: 1 Number of replicas of each worker, or the minimum while autoscaling. Must be at least 1: vLLM reports no metrics without a running pod. * model-name | string Optional name the served model is exposed as (spec.model.name of the LLMInferenceService), e.g. llama-3.1-8b. When left empty it defaults to the model reference derived from model-uri: the part after hf:// for Hugging Face URIs, or the last path segment of the bucket key for s3:// URIs. * model-uri | string URI of the model to deploy. Two schemes are supported: Hugging Face URIs (hf://), e.g. hf://meta-llama/Llama-3.1-8B-Instruct, and S3 URIs (s3://), e.g. s3://my-bucket/models/llama-3.1-8b. When an s3:// URI is used the charm must be related to an s3-integrator over the s3-credentials endpoint to obtain the bucket credentials. * runtime-image | string Optional OCI image override for the vLLM serving runtime container. When empty, the charm uses the vLLM image it ships for the selected accelerator. A custom image must match the accelerator (e.g. a CUDA build for nvidia-gpu); a mismatch is not detected and only fails at pod start. * storage-initializer-image | string Optional OCI image override for the storage-initializer init container that downloads the model into the workload, for both hf:// and s3:// model URIs. When empty, the charm uses the storage-initializer image it ships. * vllm-extra-args | string Extra arguments appended to `vllm serve`, e.g. "--dtype bfloat16 --enable-prefix-caching". Parsed with shell-like quoting but never run through a shell. Flags managed by the charm (--host, --port, --model, --served-model-name, --tensor-parallel-size, --pipeline-parallel-size, --data-parallel-size, --max-model-len, --gpu-memory-utilization) and --kv-transfer-config are rejected.