Post-installation verification
Verify that the Model Gateway is operating correctly after installation by validating service health, model connectivity, inference requests, routing behavior, and error handling.
After you install and configure the Model Inference Gateway, verify that the deployment is functioning correctly. Complete the following validation tasks in order. Confirm that each step succeeds before proceeding to the next.
Verify gateway health
Check that the Inference Service pod is running and all containers are ready:
oc get pods -n <bob-namespace> -l app=bob-inferenceCheck the startup logs for gateway initialization. A successful startup logs each configured model being registered. Credential or config parse errors surface here:
oc logs -n <bob-namespace> deploy/bob-inference --since=5mConfirm the gateway's /v1/models endpoint is responding from within the cluster — this is the clearest signal that it is up and has loaded the config:
oc exec -n <bob-namespace> deploy/bob-gateway -- \
curl -sk https://bob-inference.<bob-namespace>.svc.cluster.local:7330/v1/model/info | jq .Validate model connectivity
Check the /v1/models response for each expected model_name. A model that failed to connect (bad URL, auth error, TLS failure) is absent from the list or present with an error status.
Only models with exposed: true in model_info appear in the public /v1/models list. Models with exposed: false (for example, the guardrail model) do not appear but should still be reachable internally. Absence from the list is not always a connectivity failure.
Test model inference
Send a minimal test completion request directly to the Inference Service from within the cluster for both the core model and the guardrail model:
oc exec -n <bob-namespace> deploy/bob-gateway -- \
curl -sk https://bob-inference.<bob-namespace>.svc.cluster.local:7330/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<model_name>",
"messages": [{"role": "user", "content": "Say hello."}],
"max_tokens": 10
}' | jq .A successful response contains a choices array with a message.content field.
Common errors:
| HTTP status | Cause |
|---|---|
401 Unauthorized | Invalid credentials or insufficient permissions |
404 Not Found | Specified model ID does not exist on the provider |
502 Bad Gateway | Upstream model endpoint cannot be reached |
The guardrail model must be reachable and able to process requests. If the guardrail model is unavailable, IBM Bob rejects all inference requests. Ensure that the guardrail validation succeeds before considering the installation complete.
Validate model routing
Confirm the primary model routes correctly by checking that the model field in the response matches the requested model_name.
If fallbacks are configured on any model entry, validate the fallback path by temporarily pointing the primary model's base_url to an unreachable host and confirming the gateway falls back to the next model in the list. Restore the correct base_url after testing.
For models with exposed: false, confirm internal routing still resolves them correctly even though they do not appear in the public model list.
Validate error handling
Run the following deliberate negative tests to confirm the gateway handles failures gracefully:
| Test | How to trigger | Expected behavior |
|---|---|---|
| Invalid API key | Temporarily set a wrong key value in the secret | Gateway returns 401, does not crash |
| Unreachable endpoint | Set base_url to an invalid host | Gateway returns 502 or 503, logs the upstream error |
| Malformed model ID | Set model to a non-existent ID | Provider returns 404, error is surfaced to the caller |
| Missing secret env var | Remove a key from bob.modelGateway.secrets | Request-time failure with a logged error referencing the missing var |
| Expired TLS cert | Supply an expired cert as ca_cert_pem | TLS handshake failure logged at request time |
Collect logs and troubleshoot issues
Stream live logs during a test request to observe gateway routing decisions in real time:
oc logs -n <bob-namespace> deploy/bob-inference -fCollect a full log dump for sharing with support:
oc logs -n <bob-namespace> deploy/bob-inference --since=1h > bob-inference.logKey log patterns:
| Log pattern | What it means |
|---|---|
| Model registered successfully | The gateway loaded the model entry without errors |
connection refused / no route to host | Connectivity failure to the model endpoint |
401 / 403 from upstream | Credential is wrong or lacks required permissions |
certificate signed by unknown authority | CA cert is missing or incorrect in ca_cert_pem |
environment variable not found | A secret referenced with env.* is not in the mounted secrets |
For event-level errors (for example, secret mount failures that prevent the container from starting):
oc describe pod -n <bob-namespace> -l app=inference-serviceSwitching the core inference model
To switch from one supported core inference model to another post-install:
Ensure the new model is deployed
Confirm the new model is deployed and serving. See Model serving infrastructure.
Update the model gateway config file
Update your model-gateway.yaml to reference the new model. See Configuring the Model Gateway.
Apply the change
Apply the updated configuration using bobctl update-model-config:
bobctl update-model-config --model-config ./my-model-config.yaml --update-secretsSee Deploying the configuration for the full flag reference.
Validation before installation
Validate model connectivity, credentials, certificates, and configuration settings before installation to identify and resolve Model Gateway issues before deploying Bob on-premises.
Operations and troubleshooting
Monitor, maintain, and troubleshoot Model Gateway deployments, including model configuration updates, credential rotation, health monitoring, log collection, and resolution of common connectivity and authentication issues.