Learn how to deploy and configure model endpoints when your environment does not already provide a model-serving solution.

Before you can connect IBM Bob to a large language model (LLM), you must have access to a deployed model endpoint. Bob does not provision, host, or manage model-serving infrastructure. If your organization already provides model endpoints through OpenShift AI, GPU-based inference servers, cloud AI services, or other inference platforms, proceed directly to Configuring the Model Gateway.

Deployment options

Choose the option that best fits your environment and operational requirements.

Option	When to use
OpenShift AI (on-cluster, OCI/ModelCar)	Air-gapped or self-hosted models; RHOAI already available on the cluster
Public cloud	Frontier models via AWS Bedrock, Azure OpenAI, or Google Vertex AI
Private infrastructure	Model served on separate GPU servers or a dedicated inference cluster

OpenShift AI (on-cluster, OCI/ModelCar)

Red Hat OpenShift AI (RHOAI) is the recommended platform for serving models on-cluster, including air-gapped deployments. The preferred approach for loading model weights is the OCI/ModelCar pattern: model files are baked into an OCI image under /models/ and pushed to a private registry. When the InferenceService is deployed, KServe injects a modelcar-init init container that pulls the image and copies the weights into a shared volume at /mnt/models/, which the serving runtime (for example, vLLM) then loads from. The model image is cached on the node after the first pull — subsequent restarts on the same node skip the download entirely.

Prerequisites:

Red Hat OpenShift AI operator installed on the cluster

KServe enabled and configured

GPU-enabled worker nodes with the NVIDIA GPU Operator (or equivalent) configured

oc CLI authenticated to the target cluster with permission to create resources in the target namespace

A private container registry accessible from the cluster, with credentials to push images to it

Configure Red Hat OpenShift AI

Configure the DataScienceClusterInitialization and DataScienceCluster resources as follows:

Disable serviceMesh.

Enable KServe by setting managementState: Managed.

Configure KServe to use RawDeployment mode.

Remove all unused Red Hat OpenShift AI components.

Example KServe configuration:

kserve:
  defaultDeploymentMode: RawDeployment
  nim:
    managementState: Managed
  rawDeploymentServiceConfig: Headed
  serving:
    ingressGateway:
      certificate:
        type: OpenshiftDefaultIngress
    managementState: Removed
    name: knative-serving
  managementState: Managed

Enable ModelCar support

ModelCar support is controlled by the inferenceservice-config ConfigMap in the redhat-ods-applications namespace. The storageInitializer key must contain "enableModelcar": true.

Verify the current configuration:

oc get configmap inferenceservice-config \
  -n redhat-ods-applications \
  -o jsonpath='{.data.storageInitializer}'

The output must contain:

{
  "enableModelcar": true,
  "cpuModelcar": "10m",
  "memoryModelcar": "15Mi"
}

If enableModelcar is missing or false, update the ConfigMap:

# Retrieve current value, merge the flag, and patch
CURRENT=$(oc get configmap inferenceservice-config \
  -n redhat-ods-applications \
  -o jsonpath='{.data.storageInitializer}')
PATCHED=$(echo "$CURRENT" | python3 -c "
import json, sys
d = json.load(sys.stdin)
d['enableModelcar'] = True
print(json.dumps(d))
")
oc patch configmap inferenceservice-config \
  -n redhat-ods-applications \
  --type merge \
  -p "{\"data\":{\"storageInitializer\":$(echo $PATCHED | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')}}"

Restart the KServe controller to apply the change:

oc rollout restart deployment kserve-controller-manager -n redhat-ods-applications
oc rollout status deployment kserve-controller-manager -n redhat-ods-applications

Package model files into an OCI image

Download the model weights from Hugging Face and package them into an OCI image. The image must have all model files under the /models directory. For more information, see KServe documentation. For vLLM deployments, include the safetensors model shards and exclude legacy checkpoint files such as .bin, .pt, and original/*.

Example Dockerfile:

FROM busybox:latest

# Model weights — safetensors shards and index
COPY <model-name>/model-*.safetensors         /models/
COPY <model-name>/model.safetensors.index.json /models/

# Model configuration
COPY <model-name>/config.json             /models/
COPY <model-name>/generation_config.json  /models/

# Tokenizer
COPY <model-name>/tokenizer.json           /models/
COPY <model-name>/tokenizer_config.json    /models/
COPY <model-name>/special_tokens_map.json  /models/

If the model includes a chat template (for example, chat_template.jinja), add it. Alternatively, copy the entire directory in one instruction — simpler, but includes files not needed by vLLM:

FROM busybox:latest
COPY <model-name>/ /models/

Push the image to a private registry

Push the OCI image to a private container registry that is reachable from the cluster:

<registry>/<project>/<model-name>:latest

You can use any supported image management tool, including:

Podman
Skopeo
CI/CD pipelines
Registry-specific tooling

Create the target namespace and image pull secret

Create the project namespace and configure credentials for pulling the model image:

oc new-project <namespace>

Create the registry pull secret:

# Pull secret so KServe can pull the model image from your private registry
oc create secret docker-registry model-registry-secret \
  --docker-server=<registry> \
  --docker-username=<username> \
  --docker-password=<password-or-token> \
  -n <namespace>

Create a service account:

# Service account used by the KServe predictor
oc create sa model-puller-sa -n <namespace>

Associate the pull secret with the service account:

# Attach the pull secret — both commands are required:
# oc secrets link covers general secret use;
# imagePullSecrets is needed by KServe's ModelCar init container specifically
oc secrets link model-puller-sa model-registry-secret --for=pull -n <namespace>
oc patch serviceaccount model-puller-sa -n <namespace> \
  -p '{"imagePullSecrets": [{"name": "model-registry-secret"}]}'

Note:

Both commands are required. The imagePullSecrets entry is used by the ModelCar initialization container during model image retrieval.

Create a ServingRuntime

Deploy a vLLM-based ServingRuntime in the target namespace:

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: vllm-runtime
  namespace: <namespace>
spec:
  multiModel: false
  supportedModelFormats:
    - name: pytorch
      autoSelect: true
  containers:
    - name: kserve-container
      image: vllm/vllm-openai:<version>
      ports:
        - containerPort: 3000
          protocol: TCP
      livenessProbe:
        httpGet:
          path: /health
          port: 3000
        periodSeconds: 30
        timeoutSeconds: 5
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /health
          port: 3000
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 3
      startupProbe:
        httpGet:
          path: /health
          port: 3000
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 60

Apply the runtime configuration:

oc apply -n <namespace> -f serving-runtime-vllm.yaml

Deploy the InferenceService

Create an InferenceService that references the OCI model image using an oci:// storage URI.

Key configuration requirements:

Reference the previously created ServingRuntime.
Specify the OCI image location in storageUri.
Configure CPU, memory, and GPU resource limits that match the model requirements.
Mount shared memory (/dev/shm) for vLLM.
Configure vLLM runtime arguments and environment variables.

Example:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: <model-name>
  annotations:
    serving.kserve.io/autoscalerClass: external
    serving.kserve.io/deploymentMode: RawDeployment
spec:
  predictor:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: kubernetes.io/arch
                  operator: In
                  values:
                    - amd64
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
    volumes:
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 64Gi
    model:
      modelFormat:
        name: pytorch
      runtime: vllm-runtime
      storageUri: "oci://<registry>/<namespace-or-project>/<model-name>:latest"
      resources:
        requests:
          cpu: "<cpu-request>"          # e.g. "8"
        limits:
          cpu: "<cpu-limit>"            # e.g. "16"
          memory: <memory-limit>        # e.g. 96Gi — size to model VRAM requirements
          nvidia.com/gpu: "<gpu-count>" # e.g. "1"
      volumeMounts:
        - name: shm
          mountPath: /dev/shm
      args:
        - /mnt/models/
        - --served-model-name=<model-name>
        - --port=3000
        - --enable-auto-tool-choice
        - --tool-call-parser=openai
      env:
        - name: HOME
          value: /tmp
        - name: MAX_LOG_LEN
          value: "100"        # truncate vLLM log lines to avoid log flooding
        - name: HF_HUB_CACHE
          value: /tmp
        - name: TRITON_CACHE_DIR
          value: /tmp
        - name: XDG_CACHE_HOME
          value: /tmp
        - name: HF_HOME
          value: /tmp/hf_home
        - name: NUM_GPUS
          value: "<gpu-count>" # must match nvidia.com/gpu limit above
        - name: CUDA_VISIBLE_DEVICES
          value: "<gpu-indices>" # e.g. "0" for a single GPU; "0,1" for two
        - name: VLLM_WORKER_MULTIPROC_METHOD
          value: spawn
        - name: LOGNAME
          value: vllm
        - name: USER
          value: vllm

Apply the InferenceService manifest:

oc apply -n <namespace> -f isvc-<model-name>.yaml

Verify the deployment

Monitor deployment status, pod startup, and runtime logs:

# Watch the InferenceService reach Ready state
oc get inferenceservice <model-name> -n <namespace> -w

# Watch the predictor pod start up
oc get pods -n <namespace> -w

# Tail predictor logs (model load can take several minutes once Running)
oc logs -f deployment/<model-name>-predictor -n <namespace>

When the InferenceService reports READY: True, validate the endpoint.

Verify model registration and send a test inference request:

POD=$(oc get pods -n <namespace> \
  -l app=isvc.<model-name>-predictor \
  -o jsonpath='{.items[0].metadata.name}')

oc exec -n <namespace> "$POD" -c kserve-container -- \
  curl -fsS http://127.0.0.1:3000/v1/models

oc exec -n <namespace> "$POD" -c kserve-container -- \
  curl -fsS http://127.0.0.1:3000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-name>",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 50
  }'

Troubleshooting

Issue	Possible cause	Resolution
`ErrImagePull`	Missing or invalid registry credentials	Confirm that `model-registry-secret` exists, has valid credentials, and is linked to `model-puller-sa` using both `oc secrets link` and `imagePullSecrets` patch.
`ImagePullBackOff`	Image pull secret not configured for ModelCar	Ensure the service account includes the `imagePullSecrets` configuration. Run `oc patch serviceaccount model-puller-sa -n <namespace> -p '{"imagePullSecrets": [{"name": "model-registry-secret"}]}'`
Predictor remains in `Init:0/1`	Model image download in progress	Check `oc describe pod` events for `Pulling`/`Pulled` progress; pull time scales with image size and registry bandwidth.
Predictor remains in `Init:0/1` indefinitely	Model files not found	Verify that model files are stored under `/models` in the OCI image.
`Engine core initialization failed`	Initial CUDA compilation timeout	The first startup can take longer while caches are generated. Restart and retry.
Model loading fails	Unsupported model file format	Use Hugging Face safetensors files and exclude legacy checkpoints.
OpenSSL FIPS self-test errors	Container image is not FIPS compatible	Use a FIPS-compatible vLLM image.
`OutOfMemory` / `OOMKilled`	Insufficient GPU memory	Increase GPU resources, reduce context length, or use a quantized model.
InferenceService remains `Pending`	Cluster resources unavailable	Verify GPU availability and release resources from unused workloads.

For more information, see:

Red Hat OpenShift AI — Serving models

KServe documentation

KServe ModelCar — OCI model packaging

Public cloud model endpoints

Use this option when models are hosted by a cloud provider, such as IBM watsonx, AWS Bedrock, Azure OpenAI, or Google Vertex AI.

Prerequisites:

Outbound HTTPS (port 443) connectivity from the OpenShift cluster to the cloud service endpoint.

Valid API keys, IAM credentials, or equivalent authentication credentials.

Before you configure IBM Bob

Ensure that:

The model deployment is provisioned and active.

Authentication credentials are generated and stored securely.

Any provider-specific networking or access requirements are completed.

After the endpoint is available, proceed to Configuring the Model Gateway.

Private infrastructure model endpoints

Use this option when models are hosted on customer-managed infrastructure outside the IBM Bob cluster, such as:

Dedicated GPU servers

Separate OpenShift clusters

Bare-metal inference servers

Enterprise AI platforms

Prerequisites:

The model endpoint exposes an OpenAI-compatible API.

HTTPS connectivity exists between the IBM Bob cluster and the endpoint.

Authentication credentials are available as Kubernetes secrets.

Before you configure IBM Bob

Validate the following:

Network connectivity from the cluster to the endpoint.

TLS or mutual TLS configuration, if required.

Network policies and firewalls allow access.

Endpoint authentication is functioning correctly.

After connectivity and authentication are verified, proceed to Configuring the Model Gateway.

Model serving infrastructure

Deployment options

OpenShift AI (on-cluster, OCI/ModelCar)

Configure Red Hat OpenShift AI

Enable ModelCar support

Package model files into an OCI image

Push the image to a private registry

Create the target namespace and image pull secret

Create a ServingRuntime

Deploy the InferenceService

Verify the deployment

Troubleshooting

Public cloud model endpoints

Private infrastructure model endpoints

On this page