Learn how to create and manage a Model Gateway configuration file, including provider settings, model definitions, credential management, TLS configuration, and deployment examples for supported model providers.

The Model Gateway enables Bob on-premises to connect to and route requests to supported AI models across different providers, including OpenAI-compatible endpoints, AWS Bedrock, and Google Vertex AI. A Model Gateway configuration defines model endpoints, authentication settings, routing behavior, fallback options, and model capabilities, while sensitive credentials and certificates are securely managed through config.yaml.

The model gateway configuration is a YAML file that defines which models the gateway connects to and how to authenticate with each provider.

Warning:

Configure only one core inference model in the model gateway at a time. Configuring multiple core models simultaneously is not supported and results in undefined behavior.

Model specification

Each entry in the models list must have a unique model_name. The full model spec is:

- model_name: example_model
  # Provider types (pick one per model):
  openai_compatible:  # — any OpenAI-compatible REST endpoint (Azure, …)
  bedrock:            # — AWS Bedrock invoke endpoint
  vertex:             # — Google Vertex AI (Gemini)

  # Model info parameters (optional)
  model_info:
    exposed: true                          # true: visible in /models list; false: internal-only
    max_input_tokens: 128000               # context window size
    max_output_tokens: 4096                # maximum tokens the model may generate
    input_cost_per_token: 0.0000025        # cost in USD per input token (used for usage metering)
    output_cost_per_token: 0.000010        # cost in USD per output token
    cache_read_input_token_cost: 0.00      # cost for cache-hit input tokens (prompt caching)
    cache_creation_input_token_cost: 0.00  # cost to write a new cache entry
    supports_prompt_caching: true          # true if the model supports prompt caching
    supports_function_calling: true        # true if the model supports tool/function calls
    supports_tool_choice: true             # true if tool_choice param is honoured
    supports_reasoning: false              # true if the model supports a reasoning_effort param
    supports_vision: false                 # true if the model accepts image inputs in chat messages
    mode: chat                             # chat | embedding | image_generation

  # Optional ordered list of model_name values to try when this model is unavailable
  fallbacks:
    - fallback_model

  # Optional free-form request parameters appended to every request sent to
  # the provider, including standard parameters (e.g. temperature, top_p,
  # max_tokens) and provider-specific extensions (e.g. top_k,
  # repetition_penalty, anthropic_beta).
  chat_inference_params:
    temperature: 0.7
    top_p: 0.9

Warning:

Set supports_vision: true only on models that accept image inputs. If supports_vision is set to true on a model that does not support image inputs, sending an image in chat results in a provider error. The following models do not support image inputs and must use supports_vision: false or omit the flag: Laguna S2.1 and Nvidia Nemotron 3.

Providers

`openai_compatible`

Connects to any externally hosted model with an OpenAI-compatible REST API (for example, Azure OpenAI, custom endpoints).

openai_compatible:
  model: gpt-4o                          # Model ID as expected by the provider
  base_url: https://base_url             # Base URL of model deployment
  api_key: env.API_KEY                   # API key to authenticate. env.<VAR> reads from the container environment
  extra_headers:                         # Extra headers to include when inferencing
    example_header: env.HEADER_VALUE
  insecure_skip_verify: true             # Disable TLS verification (not recommended for production)
  ca_cert_pem: env.CA_CERT               # CA certificate for TLS verification

`bedrock`

Connects to a model served via AWS Bedrock.

bedrock:
  model: claude                                  # Model ID as expected by the provider
  region: us-east-1                              # AWS region where your Bedrock endpoint lives
  access_key_id: env.AWS_ACCESS_KEY              # Bedrock Access Key ID
  secret_access_key: env.AWS_SECRET_ACCESS_KEY   # Bedrock Secret Access Key

`vertex`

Connects to a model served via Google Vertex AI (Gemini). Credentials must be provided as a base64-encoded service-account JSON string.

vertex:
  model: gemini                              # Model ID as expected by the provider
  project: my-gcp-project                    # GCP project ID
  location: global                           # Vertex AI region / "global" for Global API
  credentials: env.GEMINI_CREDENTIALS        # Service-account JSON, base64-encoded

YAML anchors

Credential anchors

Define credentials once and reference them with <<: *anchor-name across multiple model entries to avoid repetition.

# AWS Bedrock credentials — referenced by bedrock models.
x-aws-bedrock-auth: &aws-bedrock-auth
  region: us-east-1
  access_key_id: env.AWS_ACCESS_KEY
  secret_access_key: env.AWS_SECRET_ACCESS_KEY

models:
  - model_name: my-bedrock-model
    bedrock:
      model: claude
      <<: *aws-bedrock-auth            # merge credentials anchor defined above
    model_info:
      exposed: true
      mode: chat

Model anchors

Merge a full model block across several entries to avoid repeating provider config and model_info.

x-my-base-model: &my-base-model
  vertex:
    model: gemini-2.5-pro
    project: my-gcp-project
    location: global
    credentials: env.GEMINI_CREDENTIALS
  model_info:
    exposed: false
    supports_reasoning: true
    mode: chat

models:
  - model_name: my-gemini-model
    <<: *my-base-model

  - model_name: my-second-gemini-model
    <<: *my-base-model

Credential and secret management

Secrets

For any secrets referenced in the model config (API keys, passwords, certificates), configure them in the install config.yaml under bob.modelGateway.secrets. Secrets are mounted into the Inference Service container at runtime.

bob:
  modelGateway:
    secrets:
      VAR_BAR_1: FOO_1
      VAR_BAR_2: FOO_2
      VAR_BAR_N: FOO_N

Reference a secret in the model config using the env.<VAR_NAME> syntax:

api_key: env.VAR_BAR_1

TLS and certificate requirements

The root CA bundle includes standard public CA certificates for cloud providers. However, when using private endpoints or internal model servers (for example, vLLM or OpenShift AI with custom or self-signed enterprise certificates), you must supply your internal Root/Intermediate CA certificate to establish TLS trust.

Note:

This model-endpoint TLS configuration is separate from the certificate required for Bob IDE and Bob Shell to connect to the Bob backend. For the backend endpoint certificate and client trust steps, see TLS certificates.

Custom TLS certificates follow the same two-file distributed configuration pattern as API credentials:

In model-gateway.yaml: Set ca_cert_pem to an environment variable name (for example, env.CA_CERT).

In config.yaml: Add the matching variable name under bob.modelGateway.secrets and paste the full PEM-encoded certificate string.

Model gateway config:

models:
  - model_name: example-model
    openai_compatible:
      model: mistral-3.5
      base_url: https://vllm.internal.corp:8000/v1
      api_key: env.MODEL_API_KEY
      ca_cert_pem: env.CA_CERT           # Points to the variable name defined in config.yaml
    model_info:
      exposed: true
      mode: chat

Install config (config.yaml):

bob:
  modelGateway:
    secrets:
      MODEL_API_KEY: "<your-api-key>"
      # The actual PEM certificate content matching env.CA_CERT above:
      CA_CERT: |
        -----BEGIN CERTIFICATE-----
        MIIFazCCA1OgAwIBAgIRAIIQjJaDSmJT3g4qg05...
        ... [full PEM-encoded CA certificate data] ...
        -----END CERTIFICATE-----

During deployment, bobctl injects CA_CERT into the bob-inference-model-secrets Kubernetes Secret. This is then mounted into the inference gateway service and used for TLS handshake against your private model server.

Complete example

The following is a complete example covering all provider types. Save the model gateway config to a file (for example, /tmp/example/model-gateway.yaml) and reference it at install time.

Model gateway config (/tmp/example/model-gateway.yaml)

# ── Credential anchors (shared across model entries via YAML merge keys) ──────
# AWS Bedrock credentials
x-aws-bedrock-auth: &aws-bedrock-auth
  region: us-east-1
  access_key_id: env.AWS_ACCESS_KEY
  secret_access_key: env.AWS_SECRET_ACCESS_KEY

# Google Vertex AI credentials
x-vertex-auth: &vertex-auth
  project: my-gcp-project
  location: global
  credentials: env.GEMINI_CREDENTIALS

# ── Shared model anchors (optional) ───────────────────────────────────────────
x-my-base-model: &my-base-model
  vertex:
    model: gemini-2.5-pro
    <<: *vertex-auth
  model_info:
    exposed: true
    max_input_tokens: 200000
    max_output_tokens: 12000
    input_cost_per_token: 0.00000125
    output_cost_per_token: 0.00001
    cache_read_input_token_cost: 0.000000125
    supports_reasoning: true
    mode: chat

# ── Models ────────────────────────────────────────────────────────────────────
models:
  # OpenAI-compatible model (e.g. Azure OpenAI) with API key
  - model_name: my-gpt-model
    openai_compatible:
      model: gpt-4o
      base_url: https://<resource>.cognitiveservices.azure.com/openai
      api_key: env.BOB_AZURE_API_KEY
    model_info:
      exposed: true
      max_input_tokens: 128000
      max_output_tokens: 4096
      input_cost_per_token: 0.0000025
      output_cost_per_token: 0.000010
      supports_function_calling: true
      supports_tool_choice: true
      mode: chat
    chat_inference_params:
      temperature: 0.7
      top_p: 0.9

  # AWS Bedrock model
  - model_name: my-bedrock-model
    bedrock:
      model: us.anthropic.claude-3-5-sonnet-20241022-v2:0
      <<: *aws-bedrock-auth
    fallbacks:                          # optional: ordered list of fallback model names
      - my-fallback-model
    model_info:
      exposed: false
      max_input_tokens: 200000
      max_output_tokens: 8192
      input_cost_per_token: 0.000003
      output_cost_per_token: 0.000015
      cache_creation_input_token_cost: 0.00000375
      cache_read_input_token_cost: 0.0000003
      supports_prompt_caching: true
      supports_function_calling: true
      supports_tool_choice: true
      mode: chat
    chat_inference_params:
      temperature: 0.7
      top_k: 50

  # Google Vertex AI (Gemini) model using shared model anchor
  - model_name: my-gemini-model
    <<: *my-base-model

  # OpenAI-compatible model with custom CA certificate
  - model_name: example-model-mini
    openai_compatible:
      model: gpt-4o-mini
      base_url: https://llm-mock-server.ca-tor.containers.appdomain.cloud
      ca_cert_pem: env.CA_CERT
    model_info:
      exposed: true
      max_input_tokens: 131072
      input_cost_per_token: 0.00000015
      output_cost_per_token: 0.0000006
      supports_function_calling: true
      supports_tool_choice: true
      mode: chat

  # OpenAI-compatible model with extra headers
  - model_name: another-example-model-mini
    openai_compatible:
      model: gpt-4o-mini
      base_url: https://llm-mock-server.ca-tor.containers.appdomain.cloud
      extra_headers:
        model_key: env.MODEL_KEY
    model_info:
      exposed: true
      max_input_tokens: 131072
      input_cost_per_token: 0.00000015
      output_cost_per_token: 0.0000006
      supports_function_calling: true
      supports_tool_choice: true
      mode: chat

Secrets (config.yaml)

bob:
  modelGateway:
    secrets:
      BOB_AZURE_API_KEY: someapikeyvalue
      AWS_ACCESS_KEY: someapikeyvalue
      AWS_SECRET_ACCESS_KEY: someapikeyvalue
      GEMINI_CREDENTIALS: <base64_vertex_credentials>
      CA_CERT: <PEM encoded CA cert>
      MODEL_KEY: apikeyvalue

Install command

bobctl install --model-config /tmp/example/model-gateway.yaml

Deploying the configuration

During initial installation

Pass the path to your model gateway config file using the --model-config flag at install time:

bobctl install --model-config path/to/model-gateway-config.yaml --accept-license

Warning:

If bobctl install is run without --model-config, Bob is installed with an empty model gateway config. The Inference Service runs but has no connection to any model for inferencing. Use bobctl update-model-config post-install to push a model config to the cluster.

How credentials are deployed to the cluster

The model gateway configuration file references credentials as environment variables (for example, env.AWS_ACCESS_KEY, env.BOB_AZURE_API_KEY). Different model providers require different secrets — AWS IAM keys for Bedrock, API keys for Azure OpenAI, or service account JSON for Google Gemini.

During installation, these credentials are provided in your config.yaml under bob.modelGateway.secrets. The bobctl CLI automatically processes this section and creates a Kubernetes Secret named bob-inference-model-secrets in the cluster, mounting the keys as environment variables directly inside the inference gateway service container.

bob:
  modelGateway:
    secrets:
      # AWS Bedrock authentication
      AWS_ACCESS_KEY: "<your-aws-access-key-id>"
      AWS_SECRET_ACCESS_KEY: "<your-aws-secret-access-key>"

      # Azure OpenAI authentication
      BOB_AZURE_API_KEY: "<your-azure-api-key>"

      # Google Cloud Vertex AI / Gemini authentication
      BOB_GEMINI_CREDENTIALS: "<your-gemini-credentials-json>"

      # Custom endpoint API keys / tokens or internal proxy auth
      # RITS_APIKEY: "<your-api-key>"

      # Custom CA certificate in PEM format for self-signed internal endpoints
      # CA_CERT: |
      #   -----BEGIN CERTIFICATE-----
      #   ...
      #   -----END CERTIFICATE-----

Updating the configuration post-install (`bobctl update-model-config`)

Use bobctl update-model-config to push a model gateway config and/or secrets to a live cluster without reinstalling. This is the required path when bobctl install was run without --model-config, and the same command used when switching the core inference model.

The command manages two separate cluster resources:

Flag	Cluster resource	Source
`--model-config <file>`	`ConfigMap` `bob-inference-model-config`	The file you pass in
`--update-secrets`	`Secret` `bob-inference-model-secrets`	`bob.modelGateway.secrets` in `config.yaml`

At least one of the two must be provided — passing neither is an error.

Prerequisites:

You must be logged into the cluster (oc login)

config.yaml must exist next to bobctl (copy from config-template.yaml)

helm ≥ 3.14.0 must be on your PATH

Common usage:

# Update only the model config file
bobctl update-model-config --model-config ./my-model-config.yaml

# Update only the secrets (keys come from config.yaml)
bobctl update-model-config --update-secrets

# Update both at once
bobctl update-model-config --model-config ./my-model-config.yaml --update-secrets

# Preview what would be applied without touching the cluster
bobctl update-model-config --model-config ./my-model-config.yaml --update-secrets --dry-run

All flags:

Flag	Default	Description
`--model-config <file>`		Path to the model config file to push into the ConfigMap
`--update-secrets`		Push `bob.modelGateway.secrets` from `config.yaml` into the Secret
`--output-config <file>`	`model-gateway-config.yaml`	Where to write the rendered ConfigMap manifest
`--output-secret <file>`	`model-gateway-secret.yaml`	Where to write the rendered Secret manifest
`--cleanup`	false	Delete the rendered manifest files after applying them
`--dry-run`	false	Print what would run without executing any `oc` commands

Configuring the Model Gateway

Model specification

Providers

`openai_compatible`

`bedrock`

`vertex`

YAML anchors

Credential anchors

Model anchors

Credential and secret management

Secrets

TLS and certificate requirements

Complete example

Deploying the configuration

During initial installation

How credentials are deployed to the cluster

Updating the configuration post-install (`bobctl update-model-config`)

On this page