Odin API Reference

Odin serves models on Kubernetes through two custom resources in the odin.moreh.io/v1alpha1 API group: InferenceService (a running inference deployment) and InferenceServiceTemplate (a reusable spec fragment that an InferenceService can reference).

InferenceService

An InferenceService describes a model deployment — the inference engine, the model, parallelism, and the pod templates that run it. The pod templates (template, workerTemplate) are standard Kubernetes PodTemplateSpec objects.

Spec

Field	Type	Required	Description
framework	string	no	Framework specifies the inference engine (e.g. vllm, sglang).
inferencePoolRefs	array	no	InferencePoolRefs is a list of references to InferencePools.
model	object	no	Model identifies the model to serve.
parallelism	object	no	Parallelism defines the parallelism parameters for distributed inference.
replicas	integer	no	Number of replicas for Deployments or LeaderWorkerSets.
revisionHistoryLimit	integer	no	RevisionHistoryLimit is the number of old ReplicaSets to retain for this InferenceService's Deployment. Has no effect on LeaderWorkerSet-backed InferenceServices (WorkerTemplate set) — LWS has no equivalent field upstream.
rolloutStrategy	object	no	RolloutStrategy defines rollout settings for Deployment or LeaderWorkerSet.
runtimeBaseRef	object	no	RuntimeBaseRef references the cluster-scoped RuntimeBase applied as the lowest-priority layer of the merged InferenceService spec. Across an InferenceService and its templateRefs, runtimeBaseRef may be declared in at most one place.
service	object	no	Service defines configuration for the Kubernetes Service associated with the InferenceService.
template	object	no	Template describes the pod template for Deployment or LeaderWorkerSet leader.
templateRefs	array	no	TemplateRefs is a list of references to InferenceServiceTemplates.
workerTemplate	object	no	WorkerTemplate describes the pod template for LeaderWorkerSet workers.

inferencePoolRefs

Field	Type	Required	Description
name	string	no	Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

model

Field	Type	Required	Description
name	string	yes	Name is the model identifier (e.g. HuggingFace model ID).

parallelism

Field	Type	Required	Description
data	integer	no	Data parallelism size.
dataLocal	integer	no	DataLocal data local parallelism size.
dataRPCPort	integer	no	DataRPCPort is the data parallelism RPC port.
expert	boolean	no	Expert enables expert parallelism.
pipeline	integer	no	Pipeline parallelism size.
tensor	integer	no	Tensor parallelism size.

rolloutStrategy

Field	Type	Required	Description
rollingUpdate	object	no	RollingUpdate defines the parameters for RollingUpdate strategy.
type	string	no	Type defines the rollout strategy.

runtimeBaseRef

Field	Type	Required	Description
name	string	yes	Name is the name of the referent.

service

Field	Type	Required	Description
target	string	no	Target specifies which pods receive the InferenceService label for Service endpoint selection. For LeaderWorkerSet: leader (rank-0 only), workers (worker pods only), all, or auto. For Deployment: target is ignored; label is always applied.

templateRefs

Field	Type	Required	Description
group	string	no	Group is the group of the referent. When omitted, odin.moreh.io is assumed.
kind	string	no	Kind is the kind of the referent. When omitted, the reference resolves to the ClusterInferenceServiceTemplate with the given name if one exists, otherwise to an InferenceServiceTemplate.
name	string	yes	Name is the name of the referent.

Status

Field	Type	Required	Description
conditions	array	no	Conditions represent the latest available observations of an object's state.
hpaPodSelector	string	no	HPAPodSelector for pods that belong to the InferenceService object, this is needed for HPA to know what pods belong to the InferenceService object.
observedGeneration	integer	no	ObservedGeneration reflects the InferenceService's metadata.generation that this status was reconciled against. Set at end of a successful reconcile only.
readyReplicas	integer	no	ReadyReplicas track the number of groups that are in ready state (updated or not).
renderedSpec	object	no	RenderedSpec contains the fully resolved InferenceServiceSpec after merging templateRefs and applying template rendering. NOTE: Schemaless + PreserveUnknownFields prevent controller-gen from expanding the full InferenceServiceSpec schema inline, which would exceed CRD size limits. Go type safety is preserved at compile time; API-server-level validation is omitted.
replicas	integer	no	Replicas track the total number of groups that have been created (updated or not, ready or not)
updatedReplicas	integer	no	UpdatedReplicas track the number of groups that have been updated (ready or not).
workload	object	no	Workload describes the active child workload (Deployment or LeaderWorkerSet) and surfaces a single diagnostic signal explaining the current state when not Ready. Rebuilt from scratch every reconcile; not preserved across workload-kind swaps.

conditions

Field	Type	Required	Description
lastTransitionTime	string	yes	lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
message	string	yes	message is a human readable message indicating details about the transition. This may be an empty string.
observedGeneration	integer	no	observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
reason	string	yes	reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
status	string	yes	status of the condition, one of True, False, Unknown.
type	string	yes	type of condition in CamelCase or in foo.example.com/CamelCase.

workload

Field	Type	Required	Description
conditions	array	no	Conditions is a verbatim snapshot of the child's status.conditions at reconcile time. Consumers needing raw child state read this; consumers wanting a summary read InferenceService.status.conditions[Ready] + .workload.diagnostic.
diagnostic	object	no	Diagnostic is a single picked failure signal explaining why the workload is not Ready. Absent when Ready=True or when no qualifying signal exists.
kind	string	yes	Kind is the kind of the active child workload.
name	string	yes	Name is the name of the active child workload.
observedGeneration	integer	no	ObservedGeneration is the child workload's status.observedGeneration. Absent when the child has not yet been created or has not reported a status.

InferenceServiceTemplate

An InferenceServiceTemplate is a reusable fragment of InferenceService spec. An InferenceService references one or more templates through spec.templateRefs; the referenced templates are merged into its rendered spec.

Spec

Field	Type	Required	Description
framework	string	no	Framework specifies the inference engine (e.g. vllm, sglang).
model	object	no	Model identifies the model to serve.
parallelism	object	no	Parallelism defines the parallelism parameters for distributed inference.
runtimeBaseRef	object	no	RuntimeBaseRef references the cluster-scoped RuntimeBase applied as the lowest-priority layer of the merged InferenceService spec. Across an InferenceService and its templateRefs, runtimeBaseRef may be declared in at most one place.
service	object	no	Service defines configuration for the Kubernetes Service associated with the InferenceService.
template	object	no	Template describes the pod template for Deployment or LeaderWorkerSet leader.
workerTemplate	object	no	WorkerTemplate describes the pod template for LeaderWorkerSet workers.

model

Field	Type	Required	Description
name	string	yes	Name is the model identifier (e.g. HuggingFace model ID).

parallelism

Field	Type	Required	Description
data	integer	no	Data parallelism size.
dataLocal	integer	no	DataLocal data local parallelism size.
dataRPCPort	integer	no	DataRPCPort is the data parallelism RPC port.
expert	boolean	no	Expert enables expert parallelism.
pipeline	integer	no	Pipeline parallelism size.
tensor	integer	no	Tensor parallelism size.

runtimeBaseRef

Field	Type	Required	Description
name	string	yes	Name is the name of the referent.

service

Field	Type	Required	Description
target	string	no	Target specifies which pods receive the InferenceService label for Service endpoint selection. For LeaderWorkerSet: leader (rank-0 only), workers (worker pods only), all, or auto. For Deployment: target is ignored; label is always applied.