Learn about the models required by Bob on-premises, including supported core inference models, guardrail models, and provider-specific safety features used to enable AI-powered capabilities.

Bob on-premises requires access to a supported core inference model to provide code generation, explanations, chat, and assistant capabilities. Depending on your deployment requirements, you can connect Bob to air-gapped models hosted within your environment or frontier models accessed through cloud providers. For enhanced safety and policy compliance, IBM recommends configuring a guardrail model or using provider-native guardrail capabilities.

Required models

Before installing Bob, ensure that the following models are deployed and accessible.

Model role	Purpose
Core inference model	Processes code generation, explanations, chat requests, and assistant interactions.
Guardrail model	Screens inputs and outputs for safety and policy compliance. Using a guardrail model is highly recommended.

Important:

Configure exactly one core inference model at a time.

Note:

A guardrail model is strongly recommended for all deployments. For air-gapped deployments, use openai/gpt-oss-20b. When configured, Bob automatically routes safety and policy evaluation requests through the guardrail service.

Supported models

The following core inference models are available for use with IBM Bob on-premises. Configure a single core inference model in the Model Gateway. Running multiple core inference models at the same time is not supported.

Air-gapped models

Resource requirements (CPU, RAM, GPU/VRAM, and concurrency sizing) depend on model quantization, context length, serving runtime (such as vLLM or TGI), and target throughput. Refer to the product documentation for hardware specifications and deployment guides.

Model	Reference
Mistral 3.5	Mistral AI documentation
Nvidia Nemotron 3	NVIDIA NeMo / Nemotron documentation
Poolside Laguna S2.1	Poolside AI documentation

Frontier models

Frontier cloud models are accessed via provider-managed APIs (such as AWS Bedrock, Google Cloud Vertex AI, or Azure OpenAI). Refer to the product documentation for service availability, quotas, rate limits, and endpoint configuration.

Model	Reference
Claude Sonnet 5	Anthropic Claude documentation / AWS Bedrock
Claude Opus 4.8	Anthropic Claude documentation / AWS Bedrock
Google Gemini 3.7 Flash	Google Cloud Vertex AI documentation
OpenAI GPT5.6 Sol	Azure OpenAI Service documentation

Provider guardrails

Using a guardrail model is highly recommended for safety, toxicity filtering, and policy screening across all inputs and outputs. Refer to the model repository and serving runtime documentation for deployment details and hardware guidelines.

Model	Reference	Description
`openai/gpt-oss-20b`	Hugging Face Model Hub / OpenAI documentation	Required for safety and policy screening

For air-gapped models, use openai/gpt-oss-20b to implement guardrails. After this guardrail model is configured, the IDE and Bob Shell clients are automatically configured to use it for content filtering.

Frontier model guardrails

When using frontier models, you can use the provider's guardrail capability instead of the openai/gpt-oss-20b guardrail model.

AWS Bedrock guardrails

Create a Bedrock guardrail and obtain its guardrailId and version. For guidance, see the Bedrock guardrails documentation and the CreateGuardrail API reference. Add the guardrail configuration under chat_inference_params for the Bedrock model:

- model_name: bedrock-model
  bedrock:
    model: <bedrock-model-id>
    access_key_id: env.AWS_ACCESS_KEY
    region: us-east-1
    secret_access_key: env.AWS_SECRET_ACCESS_KEY
  model_info:
    exposed: true
    max_input_tokens: 270000
  chat_inference_params:
    guardrailConfig:
      guardrailIdentifier: <guardrail-id>
      guardrailVersion: <version-number-or-DRAFT>

Google Vertex AI

Configure the Vertex AI safety filters in chat_inference_params. See the Vertex AI safety filter documentation.

Supported categories	Supported thresholds
`HARM_CATEGORY_SEXUALLY_EXPLICIT`, `HARM_CATEGORY_HATE_SPEECH`, `HARM_CATEGORY_HARASSMENT`, `HARM_CATEGORY_DANGEROUS_CONTENT`	`BLOCK_NONE`, `BLOCK_ONLY_HIGH`, `BLOCK_MEDIUM_AND_ABOVE` (default), `BLOCK_LOW_AND_ABOVE` (strictest)

- model_name: gemini-model
  vertex:
    model: gemini-model
    project: <project-id>
    location: global
    credentials: env.BOB_GEMINI_CREDENTIALS
  model_info:
    max_tokens: 20000
    max_input_tokens: 270000
    exposed: false
  chat_inference_params:
    safetySettings:
      - category: HARM_CATEGORY_HATE_SPEECH
        threshold: BLOCK_MEDIUM_AND_ABOVE
      - category: HARM_CATEGORY_HARASSMENT
        threshold: BLOCK_ONLY_HIGH
      - category: HARM_CATEGORY_SEXUALLY_EXPLICIT
        threshold: BLOCK_LOW_AND_ABOVE
      - category: HARM_CATEGORY_DANGEROUS_CONTENT
        threshold: BLOCK_LOW_AND_ABOVE

Azure OpenAI

In Azure, create a content filter and assign it to the model deployment; no model gateway configuration changes are required. See the Azure OpenAI content filter documentation. To apply a policy at request time instead, configure the x-policy-id header:

- model_name: gpt-model
  openai_compatible:
    model: openai/gpt-model
    base_url: https://<endpoint>.azure.com/openai
    api_key: env.AZURE_API_KEY
    extra_headers:
      x-policy-id: <custom-content-filter-name>
  model_info:
    max_tokens: 12000
    max_input_tokens: 200000
    exposed: true

Model serving infrastructure

Bob requires access to one or more model inference endpoints to perform AI-powered tasks. Bob connects to deployed models through the Model Inference Gateway but does not provision, host, or manage model-serving infrastructure.

Most on-premises environments already have model serving infrastructure available, whether that is a shared GPU cluster running run.ai, Red Hat OpenShift AI, a dedicated vLLM serving farm, or access to a public cloud provider's model API (AWS Bedrock, Azure OpenAI, Google Vertex AI). Bob requires a network-reachable endpoint from the OpenShift cluster that exposes an OpenAI-compatible API.

For more information on deploying a supported model, see Model serving infrastructure.

Required and supported models