Learn about the models required by Bob self-hosted, including supported core inference models, guardrail models, and provider-specific safety features used to enable AI-powered capabilities.

Bob self-hosted requires access to a supported core inference model to provide code generation, explanations, chat, and assistant capabilities. Depending on your deployment requirements, you can connect Bob to air-gapped models hosted within your environment or frontier models accessed through cloud providers. For enhanced safety and policy compliance, IBM recommends configuring a guardrail model or using provider-native guardrail capabilities.

Required models

Before installing Bob, ensure that the following models are deployed and accessible.

Model role	Purpose
Core inference model	Processes code generation, explanations, chat requests, and assistant interactions.
Guardrail model	Screens inputs and outputs for safety and policy compliance. Using a guardrail model is highly recommended.

Important:

Configure exactly one core inference model at a time.

Note:

A guardrail model is strongly recommended for all deployments. For air-gapped deployments, use openai/gpt-oss-20b. When configured, Bob automatically routes safety and policy evaluation requests through the guardrail service.

Supported models

The following core inference models are available for use with IBM Bob self-hosted. Configure a single core inference model in the Model Gateway. Running multiple core inference models at the same time is not supported.

Tip:

For optimal results on complex IBM Z tasks, such as deep COBOL refactoring, PL/I analysis, and multi-file code transformations, frontier models such as Claude Sonnet deliver the highest level of reasoning, accuracy, and generation quality.

Frontier models (recommended): Best suited for enterprise IBM Z workloads requiring advanced reasoning, large-context understanding, and multi-step code generation.
Self-hosted open-weight models: Ideal for strict air-gap compliance and standard coding assistance. For complex mainframe language patterns, results might benefit from targeted prompt tuning and domain-specific context.

Air-gapped models

Resource requirements (CPU, RAM, GPU/VRAM, and concurrency sizing) depend on model quantization, context length, serving runtime (such as vLLM or TGI), and target throughput. Refer to the product documentation for hardware specifications and deployment guides.

Model	Reference
Mistral Medium 3.5	Mistral AI documentation / Mistral Medium 3.5
Nvidia Nemotron 3 Ultra	NVIDIA Nemotron documentation / Nemotron 3 Ultra FP4 / Nemotron 3 Ultra FP8
Poolside Laguna S 2.1	Poolside Laguna-S-2.1-FP8 documentation / Poolside Laguna-S-2.1-NVFP4 documentation / Poolside Laguna models

Note:

You can only use Mistral Medium 3.5 if you use IBM Bob Premium Package for Z. It is not available for general Bob self-hosted deployments.

Frontier models

Frontier cloud models are accessed via provider-managed APIs (such as AWS Bedrock, Google Cloud Vertex AI, or Azure OpenAI). Refer to the product documentation for service availability, quotas, rate limits, and endpoint configuration.

Model	Reference
Claude Sonnet 5	Anthropic Claude documentation / AWS Bedrock
Claude Opus 4.8	Anthropic Claude documentation / AWS Bedrock
Google Gemini 3.7 Flash	Google Cloud Vertex AI documentation
OpenAI GPT5.6 Sol	Azure OpenAI Service documentation

Provider guardrails

Using a guardrail model is highly recommended for safety, toxicity filtering, and policy screening across all inputs and outputs. Refer to the model repository and serving runtime documentation for deployment details and hardware guidelines.

Model	Reference	Description
`openai/gpt-oss-20b`	Hugging Face Model Hub / OpenAI documentation	Required for safety and policy screening

For air-gapped models, use openai/gpt-oss-20b to implement guardrails. After this guardrail model is configured, the IDE and Bob Shell clients are automatically configured to use it for content filtering.

Frontier model guardrails

When using frontier models, you can use the provider's guardrail capability instead of the openai/gpt-oss-20b guardrail model.

AWS Bedrock guardrails

Create a Bedrock guardrail and obtain its guardrailId and version. For guidance, see the Bedrock guardrails documentation and the CreateGuardrail API reference. Add the guardrail configuration under chat_inference_params for the Bedrock model:

- model_name: bedrock-model
  bedrock:
    model: <bedrock-model-id>
    access_key_id: env.AWS_ACCESS_KEY
    region: us-east-1
    secret_access_key: env.AWS_SECRET_ACCESS_KEY
  model_info:
    exposed: true
    max_input_tokens: 270000
    max_output_tokens: 20000
  chat_inference_params:
    guardrailConfig:
      guardrailIdentifier: <guardrail-id>
      guardrailVersion: <version-number-or-DRAFT>

Google Vertex AI

Configure the Vertex AI safety filters in chat_inference_params. See the Vertex AI safety filter documentation.

Supported categories	Supported thresholds
`HARM_CATEGORY_SEXUALLY_EXPLICIT`, `HARM_CATEGORY_HATE_SPEECH`, `HARM_CATEGORY_HARASSMENT`, `HARM_CATEGORY_DANGEROUS_CONTENT`	`BLOCK_NONE`, `BLOCK_ONLY_HIGH`, `BLOCK_MEDIUM_AND_ABOVE` (default), `BLOCK_LOW_AND_ABOVE` (strictest)

- model_name: gemini-model
  vertex:
    model: gemini-model
    project: <project-id>
    location: global
    credentials: env.GEMINI_CREDENTIALS
  model_info:
    max_input_tokens: 270000
    max_output_tokens: 20000
    exposed: false
  chat_inference_params:
    safetySettings:
      - category: HARM_CATEGORY_HATE_SPEECH
        threshold: BLOCK_MEDIUM_AND_ABOVE
      - category: HARM_CATEGORY_HARASSMENT
        threshold: BLOCK_ONLY_HIGH
      - category: HARM_CATEGORY_SEXUALLY_EXPLICIT
        threshold: BLOCK_LOW_AND_ABOVE
      - category: HARM_CATEGORY_DANGEROUS_CONTENT
        threshold: BLOCK_LOW_AND_ABOVE

Azure OpenAI

In Azure, create a content filter and assign it to the model deployment; no model gateway configuration changes are required. See the Azure OpenAI content filter documentation. To apply a policy at request time instead, configure the x-policy-id header:

- model_name: gpt-model
  openai_compatible:
    model: openai/gpt-model
    base_url: https://<endpoint>.azure.com/openai
    api_key: env.AZURE_API_KEY
    extra_headers:
      x-policy-id: <custom-content-filter-name>
  model_info:
    max_input_tokens: 270000
    max_output_tokens: 20000
    exposed: true

Model token limits

The following token limits are recommended for the supported models.

Model	Maximum input tokens	Maximum output tokens
Laguna S 2.1	270,000	20,000
Nemotron 3 Ultra	262,144	20,000
Mistral Medium 3.5	262,144	20,000
Sonnet 4.6	270,000	20,000

Configure model token limits

Specify the token limits in the model_info section of the model definition.

For example:

model_info:
  provider: poolside
  family: laguna
  version: s2.1
  max_input_tokens: 270000
  max_output_tokens: 20000

max_input_tokens defines the maximum number of tokens that can be included in a single request, including prompts, conversation history, and any additional context.

max_output_tokens defines the maximum number of tokens that the model can generate in a response.

Configure these values according to the capabilities of the underlying model.

Setting values that exceed the model's supported limits can result in request failures or truncated responses.

If a model provider publishes updated limits, update the configuration to match the provider's documented capabilities.

Model serving infrastructure

Bob requires access to one or more model inference endpoints to perform AI-powered tasks. Bob connects to deployed models through the Model Inference Gateway but does not provision, host, or manage model-serving infrastructure.

Most on-premises environments already have model serving infrastructure available, whether that is a shared GPU cluster running run.ai, Red Hat OpenShift AI, a dedicated vLLM serving farm, or access to a public cloud provider's model API (AWS Bedrock, Azure OpenAI, Google Vertex AI). Bob requires a network-reachable endpoint from the OpenShift cluster that exposes an OpenAI-compatible API.

Required and supported models