Learn how to create and manage a Model Gateway configuration file, including provider settings, model definitions, credential management, TLS configuration, and deployment examples for supported model providers.

The Model Gateway enables Bob self-hosted to connect to and route requests to supported AI models across different providers, including OpenAI-compatible endpoints, AWS Bedrock, and Google Vertex AI. A Model Gateway configuration defines model endpoints, authentication settings, routing behavior, and model capabilities, while sensitive credentials and certificates are securely managed through config.yaml.

The model gateway configuration is a YAML file that defines which models the gateway connects to and how to authenticate with each provider.

Warning:

Configure only one core inference model in the model gateway at a time. Configuring multiple core models simultaneously is not supported and results in undefined behavior.

Model specification

Each entry in the models list must have a unique model_name. The full model spec is:

- model_name: example_model
  # Provider types (pick one per model):
  openai_compatible:  # — any OpenAI-compatible REST endpoint (Azure, vLLM, …)
  bedrock:            # — AWS Bedrock invoke endpoint
  vertex:             # — Google Vertex AI (Gemini)

  # Model metadata
  model_info:
    exposed: true                         # true: visible in /v1/model/info list; false: internal-only
    provider: openai                      # model provider (e.g. openai, anthropic, google, nvidia, mistral)
    family: gpt                           # model family within the provider (e.g. gpt, sonnet, gemini)
    version: "5.6-sol"                    # model version string (see table below)
    max_input_tokens: 270000              # context window size
    max_output_tokens: 20000              # maximum tokens the model may generate
    supports_prompt_caching: true         # true if the model supports prompt caching
    supports_vision: true                 # true if the model supports image inputs
    mode: chat                            # chat | embedding | image_generation

  # Optional free-form request parameters appended to every request sent to
  # the provider, including standard parameters (e.g. temperature, top_p,
  # max_tokens) and provider-specific extensions (e.g. top_k,
  # repetition_penalty, anthropic_beta).
  chat_inference_params:
    temperature: 0.7
    top_p: 0.9

Note:

For users to send image inputs in the chat interface, the core inference model must have supports_vision: true set in its model_info block. If this flag is absent or false, image input will not be available.

Warning:

If supports_vision is set to true for a model that does not actually support image inputs, use of images will fail with an error from the provider. Ensure this flag is only enabled for models that support image inputs. Note that the following models do not support image inputs and should have supports_vision: false: Laguna S2.1 and Nvidia Nemotron 3.

Supported `provider` / `family` / `version` values

Bob uses the provider, family, and version fields together to select model-specific prompt tuning and optimizations. Use the closest matching values for unlisted models.

`provider`	`family`	`version`	Example model ID
`openai`	`gpt`	`5.6-sol`	`gpt-5.6-sol`
`openai`	`gpt-oss`	`20b`	`openai/gpt-oss-20b`
`anthropic`	`sonnet`	`5`	`us.anthropic.claude-sonnet-5`
`google`	`gemini`	`3.6-flash`	`gemini-3.6-flash`
`google`	`gemini`	`3.1-pro`	`gemini-3.1-pro`
`nvidia`	`nemotron`	`3-ultra`	`nvidia/nemotron-3-ultra-550b`
`mistral`	`medium`	`3.5`	`mistralai/mistral-medium-3-5-0`
`poolside`	`laguna`	`s2.1`	`poolside/laguna-s-2.1`

Reserved model name slots

The following model_name values must be present in your models list:

`model_name`	Used for
`premium-ide`	Default model for interactive chat
`security`	Guardrail / security checks
`background`	Background and autonomous agent tasks

The template defines security and background as YAML merge aliases of the active model, so they all follow the same provider config automatically. For security and background, a smaller and more efficient model is recommended where available (for example, openai/gpt-oss-20b).

Providers

`openai_compatible`

Connects to any externally hosted model with an OpenAI-compatible REST API (for example, Azure OpenAI, vLLM, custom endpoints).

- model_name: premium-ide
  openai_compatible:
    model: gpt-5.6-sol                     # Model ID as expected by the provider
    base_url: https://<resource>.cognitiveservices.azure.com/openai
    api_key: env.AZURE_API_KEY             # env.<VAR> reads from the model gateway secrets
    extra_headers:                         # Extra headers to include when inferencing
      example_header: env.HEADER_VALUE
    ca_cert_pem: env.CA_CERT               # CA certificate for TLS verification
  model_info:
    exposed: true
    provider: openai
    family: gpt
    version: "5.6-sol"
    max_input_tokens: 270000
    max_output_tokens: 20000
    supports_prompt_caching: true
    supports_vision: true
    mode: chat

`bedrock`

Connects to a model served via AWS Bedrock.

- model_name: premium-ide
  bedrock:
    model: us.anthropic.claude-sonnet-5    # Model ID as expected by the provider
    region: us-east-1                      # AWS region where your Bedrock endpoint lives
    access_key_id: env.AWS_ACCESS_KEY      # Bedrock Access Key ID
    secret_access_key: env.AWS_SECRET_ACCESS_KEY
  model_info:
    exposed: true
    provider: anthropic
    family: sonnet
    version: "5"
    max_input_tokens: 270000
    max_output_tokens: 20000
    supports_prompt_caching: true
    supports_vision: true
    mode: chat

`vertex`

Connects to a model served via Google Vertex AI (Gemini). Credentials must be provided as a base64-encoded service-account JSON string.

- model_name: premium-ide
  vertex:
    model: gemini-3.6-flash                # Model ID as expected by the provider
    project: <gcp-project-id>              # GCP project ID
    location: global                       # Vertex AI region / "global" for Global API
    credentials: env.GEMINI_CREDENTIALS    # Service-account JSON, base64-encoded
  model_info:
    exposed: true
    provider: google
    family: gemini
    version: "3.6-flash"
    max_input_tokens: 270000
    max_output_tokens: 20000
    supports_prompt_caching: true
    supports_vision: true
    mode: chat

Routers

The routers section configures model resolution between the frontend and the backend. The configured default_model will be the main LLM used during chat interactions.

routers:
  - name: router
    strategy: static
    static:
      default_model: premium-ide   # must match a model_name in the models list

Field	Description
`name`	Router name (always use `router`)
`strategy`	Routing strategy. Currently only `static` is supported
`static.default_model`	The `model_name` to use as the main LLM. Must exactly match a `model_name` entry in the `models` list

Note:

premium-ide is a required model name slot (see Reserved model name slots), so the recommended practice is to keep default_model: premium-ide and change which provider/model the premium-ide entry points to in your models list.

YAML anchors

The model gateway config supports standard YAML anchors (&name) and merge keys (<<: *name). The template uses this to support reuse and avoid repetition.

x-providers:
  gemini: &gemini
    vertex:
      model: gemini-3.6-flash
      project: <gcp-project-id>
      location: global
      credentials: env.GEMINI_CREDENTIALS
    model_info:
      exposed: true
      provider: google
      family: gemini
      version: "3.6-flash"
      max_input_tokens: 270000
      max_output_tokens: 20000
      supports_prompt_caching: true
      supports_vision: true
      mode: chat

models:
  - &chat_llm
    <<: *gemini
    model_name: premium-ide

  - <<: *chat_llm
    model_name: security               # overrides premium-ide from the merged anchor

  - <<: *chat_llm
    model_name: background

Credential and secret management

Secrets

For any secrets referenced in the model config (API keys, passwords, certificates), configure them in the install config.yaml under bob.modelGateway.secrets. Secrets are mounted into the Inference Service container at runtime.

bob:
  modelGateway:
    secrets:
      VAR_BAR_1: FOO_1
      VAR_BAR_2: FOO_2
      VAR_BAR_N: FOO_N

Reference a secret in the model config using the env.<VAR_NAME> syntax:

api_key: env.VAR_BAR_1

TLS and certificate requirements

When using private endpoints or internal model servers (for example, vLLM or OpenShift AI with custom or self-signed enterprise certificates), you must supply your internal Root/Intermediate CA certificate so inferencing containers can establish TLS trust.

Note:

This model-endpoint TLS configuration is separate from the certificate required for Bob IDE and Bob Shell to connect to the Bob backend. For the backend endpoint certificate and client trust steps, see TLS certificates.

There are two ways to add a CA for inferencing:

Option 1 — Inferencing trust bundle (recommended)

Add the CA to bob-inferencing-bundle-tls by labelling a Kubernetes Secret. The operator detects the label and rebuilds the bundle automatically — no model gateway config changes are required and the CA is available to all inferencing containers immediately.

oc create secret generic my-model-endpoint-ca \
  --from-file=tls.crt=model-ca.crt \
  -n <instance-namespace>

oc label secret my-model-endpoint-ca \
  bob.ibm.com/inferencing-bundle-ca=true \
  -n <instance-namespace>

The bob-inferencing-bundle-tls Secret is mounted into inferencing containers as their trust store. It always contains everything in bob-bundle-tls plus any additionally labelled Secrets. No pod restart is needed — Secrets are projected into the running container.

For full details and verification steps, see TLS certificates.

Option 2 — Per-model `ca_cert_pem` (model gateway config)

Use this when you need to supply a CA for a single specific model endpoint without affecting other inferencing containers. Custom TLS certificates follow the same two-file distributed configuration pattern as API credentials:

In model-gateway.yaml: Set ca_cert_pem to an environment variable name (for example, env.CA_CERT).

In config.yaml: Add the matching variable name under bob.modelGateway.secrets and paste the full PEM-encoded certificate string.

1. Model gateway config (model-gateway.yaml):

models:
  - model_name: example-model
    openai_compatible:
      model: mistral-3.5
      base_url: https://vllm.internal.corp:8000/v1
      api_key: env.MODEL_API_KEY
      ca_cert_pem: env.CA_CERT           # Points to the variable name defined in config.yaml
    model_info:
      exposed: true
      mode: chat

2. Install config (config.yaml):

bob:
  modelGateway:
    secrets:
      MODEL_API_KEY: "<your-api-key>"
      # The actual PEM certificate content matching env.CA_CERT above:
      CA_CERT: |
        -----BEGIN CERTIFICATE-----
        MIIFazCCA1OgAwIBAgIRAIIQjJaDSmJT3g4qg05...
        ... [full PEM-encoded CA certificate data] ...
        -----END CERTIFICATE-----

During deployment, bobctl injects CA_CERT into the bob-inference-model-secrets Kubernetes Secret. This is then mounted into the inference gateway service and used for TLS handshake against your private model server.

Complete example

The following is a complete example mirroring the structure of config-model-gateway-template.yaml from the Bob Bundle. Save it to a file (for example, /tmp/example/model-gateway.yaml) and reference it at install time.

Model Gateway config

# ── Provider definitions ───────────────────────────────────────────────────────
# The x-providers key is ignored by the model gateway — it is a YAML scratchpad
# for provider examples. Only the active provider's anchor is referenced below.
x-providers:

  # AWS Bedrock
  bedrock: &bedrock
    bedrock:
      model: us.anthropic.claude-sonnet-5
      region: us-east-1
      access_key_id: env.AWS_ACCESS_KEY
      secret_access_key: env.AWS_SECRET_ACCESS_KEY
    model_info:
      exposed: true
      provider: anthropic
      family: sonnet
      version: "5"
      max_input_tokens: 270000
      max_output_tokens: 20000
      supports_prompt_caching: true
      supports_vision: true
      mode: chat

  # Google Vertex AI (Gemini)
  gemini: &gemini
    vertex:
      model: gemini-3.6-flash
      project: <gcp-project-id>
      location: global
      credentials: env.GEMINI_CREDENTIALS
    model_info:
      exposed: true
      provider: google
      family: gemini
      version: "3.6-flash"
      max_input_tokens: 270000
      max_output_tokens: 20000
      supports_prompt_caching: true
      supports_vision: true
      mode: chat

  # Azure OpenAI
  azure: &azure
    openai_compatible:
      model: gpt-5.6-sol
      base_url: https://<resource>.cognitiveservices.azure.com/openai
      api_key: env.AZURE_API_KEY
    model_info:
      exposed: true
      provider: openai
      family: gpt
      version: "5.6-sol"
      max_input_tokens: 270000
      max_output_tokens: 20000
      supports_prompt_caching: true
      supports_vision: true
      mode: chat

  # Custom inference service (RHOAI, vLLM, etc.)
  inference_service: &isvc
    openai_compatible:
      model: nvidia/nemotron-3-ultra-550b
      base_url: https://<llm-inference-server>
      api_key: env.INFERENCE_API_KEY
    model_info:
      exposed: true
      provider: nvidia
      family: nemotron
      version: "3-ultra"
      max_input_tokens: 262144
      supports_vision: false
      mode: chat

# ── Models ────────────────────────────────────────────────────────────────────
models:

  # ── ACTIVE MODEL ─────────────────────────────────────────────────────────────
  # Change <<: *bedrock to e.g. *azure, *gemini, or *isvc to switch providers.
  - &chat_llm
    <<: *bedrock                       # ← change provider here
    model_name: premium-ide
    model_info:
      exposed: true

  # ── Aliases ───────────────────────────────────────────────────────────────────
  - <<: *chat_llm
    model_name: security               # overrides premium-ide from the merged anchor

  - <<: *chat_llm
    model_name: background

# ── Router ────────────────────────────────────────────────────────────────────
routers:
  - name: router
    strategy: static
    static:
      default_model: premium-ide

Secrets (`config.yaml`)

bob:
  modelGateway:
    secrets:
      AWS_ACCESS_KEY: "<your-aws-access-key-id>"
      AWS_SECRET_ACCESS_KEY: "<your-aws-secret-access-key>"
      GEMINI_CREDENTIALS: "<base64-encoded service-account JSON>"
      AZURE_API_KEY: "<your-azure-api-key>"
      INFERENCE_API_KEY: "<your-inference-api-key>"
      CA_CERT: <PEM encoded CA cert>

Install command

bobctl install --model-config /tmp/example/model-gateway.yaml --accept-license --registry-creds <username:password>

For a connected environment, specify your entitlement key for icr.io. --registry-creds may only be omitted if you have a private registry that does not require authentication to pull from.

Deploying the configuration

During initial installation

Pass the path to your model gateway config file using the --model-config flag at install time:

bobctl install --model-config path/to/model-gateway-config.yaml --accept-license --registry-creds <username:password>

Warning:

If bobctl install is run without --model-config, Bob is installed with an empty model gateway config. The Inference Service runs but has no connection to any model for inferencing. Use bobctl update-model-config post-install to push a model config to the cluster.

How credentials are deployed to the cluster

The model gateway configuration file references credentials as environment variables (for example, env.AWS_ACCESS_KEY, env.AZURE_API_KEY). Different model providers require different secrets — AWS IAM keys for Bedrock, API keys for Azure OpenAI, or service account JSON for Google Gemini.

During installation, these credentials are provided in your config.yaml under bob.modelGateway.secrets. The bobctl CLI automatically processes this section and creates a Kubernetes Secret named bob-inference-model-secrets in the cluster, mounting the keys as environment variables directly inside the inference gateway service container.

bob:
  modelGateway:
    secrets:
      # AWS Bedrock authentication
      AWS_ACCESS_KEY: "<your-aws-access-key-id>"
      AWS_SECRET_ACCESS_KEY: "<your-aws-secret-access-key>"

      # Azure OpenAI authentication
      AZURE_API_KEY: "<your-azure-api-key>"

      # Google Cloud Vertex AI / Gemini authentication
      GEMINI_CREDENTIALS: "<your-gemini-credentials-json>"

      # Custom endpoint API keys / tokens or internal proxy auth
      # RITS_APIKEY: "<your-api-key>"

      # Custom CA certificate in PEM format for self-signed internal endpoints
      # CA_CERT: |
      #   -----BEGIN CERTIFICATE-----
      #   ...
      #   -----END CERTIFICATE-----

Updating the configuration post-install (`bobctl update-model-config`)

Use bobctl update-model-config to push a model gateway config and/or secrets to a live cluster without reinstalling. This is the required path when bobctl install was run without --model-config, and the same command used when switching the core inference model.

Under the hood it runs helm template against the model-config.yaml and model-secrets.yaml templates in the local ibm-bob chart, then applies the rendered manifests to the cluster. Once applied, the ibm-bob-operator reconciles the bob-inference deployment with the new model config.

The command manages two separate cluster resources:

Flag	Cluster resource	Source
`--model-config <file>`	`ConfigMap` `bob-inference-model-config`	The file you pass in
`--update-secrets`	`Secret` `bob-inference-model-secrets`	`bob.modelGateway.secrets` in `config.yaml`

At least one of the two must be provided — passing neither is an error.

Prerequisites:

You must be logged into the cluster (oc login)

config.yaml must exist next to bobctl (copy from config-template.yaml)

helm ≥ 3.14.0 must be on your PATH

Common usage:

# Update only the model config file
bobctl update-model-config --model-config ./my-model-config.yaml

# Update only the secrets (keys come from config.yaml)
bobctl update-model-config --update-secrets

# Update both at once
bobctl update-model-config --model-config ./my-model-config.yaml --update-secrets

# Preview what would be applied without touching the cluster
bobctl update-model-config --model-config ./my-model-config.yaml --update-secrets --dry-run

All flags:

Flag	Default	Description
`--model-config <file>`		Path to the model config file to push into the ConfigMap
`--update-secrets`		Push `bob.modelGateway.secrets` from `config.yaml` into the Secret
`--output-config <file>`	`model-gateway-config.yaml`	Where to write the rendered ConfigMap manifest
`--output-secret <file>`	`model-gateway-secret.yaml`	Where to write the rendered Secret manifest
`--cleanup`	false	Delete the rendered manifest files after applying them
`--dry-run`	false	Print what would run without executing any `oc` commands

Configuring the Model Gateway

Model specification

Supported `provider` / `family` / `version` values

Reserved model name slots

Providers

`openai_compatible`

`bedrock`

`vertex`

Routers

YAML anchors

Credential and secret management

Secrets

TLS and certificate requirements

Option 1 — Inferencing trust bundle (recommended)

Option 2 — Per-model `ca_cert_pem` (model gateway config)

Complete example

Model Gateway config

Secrets (`config.yaml`)

Install command

Deploying the configuration

During initial installation

How credentials are deployed to the cluster

Updating the configuration post-install (`bobctl update-model-config`)

On this page