Monitor, maintain, and troubleshoot Model Gateway deployments, including model configuration updates, credential rotation, health monitoring, log collection, and resolution of common connectivity and authentication issues.

After deployment, ongoing operational management of the Model Gateway helps ensure reliable access to configured AI models and minimizes service disruptions. Administrators can review active model configurations, update provider credentials, rotate secrets, monitor gateway health, and collect diagnostic information when issues occur.

Viewing configured models

There are two levels to viewing models — what is in the config and what bob-inference has actually loaded at runtime.

Config level — retrieve the current model gateway config from the inference ConfigMap:

oc get cm bob-inference-config -n <bob-namespace> -o yaml | yq '.data."config.yaml"'

Runtime level — what bob-inference has registered and is actively routing to:

oc exec -n <bob-namespace> deploy/bob-gateway -- \
  curl -sk https://bob-inference.<bob-namespace>.svc.cluster.local:7330/v1/model/info | jq .

Note:

The runtime /v1/model/info endpoint only lists models where exposed: true. The config file is the only way to see the full set of configured models.

Updating provider credentials

Credentials are stored under bob.modelGateway.secrets in config.yaml. Use bobctl update-model-config --update-secrets to push updated credentials to the cluster.

Update the credential

Update the credential in config.yaml under bob.modelGateway.secrets.

Push the updated secret to the cluster

bobctl update-model-config --update-secrets

The ibm-bob-operator reconciles the bob-inference deployment and mounts the new secret automatically.

Tip:

If you are updating both the model config file and credentials at the same time, you can push both in a single command:

bobctl update-model-config --model-config ./my-model-config.yaml --update-secrets

Rotating secrets

Secret rotation follows the same pattern as a credential update, with additional care around timing to avoid downtime.

Recommended zero-downtime rotation sequence:

Update bob.modelGateway.secrets in config.yaml with the new credential value.

Push the new secret to the cluster: bobctl update-model-config --update-secrets.

Restart the Inference Service to apply the new secret:

oc rollout restart deploy/inference-service -n <bob-namespace>

Confirm the new credential is working using the inference test from Post-installation verification.

Revoke the old credential at the provider side only after the deployment is confirmed healthy.

Provider-specific notes:

openai_compatible / API keys — new key is effective immediately after the secret is updated. Safe to revoke the old key once the deployment is healthy.

bedrock — ensure the new IAM access key is active in AWS before running update-model-config. IAM propagation can take a few seconds.

vertex — generate and base64-encode the new service account key, update config.yaml, run bobctl update-model-config --update-secrets, verify, then delete the old key in GCP.

ca_cert_pem — for certificate renewal, verify the new cert is not expired before applying. See TLS certificate validation.

Monitoring gateway health

Pod restart count — a rising restart count is an early signal of a recurring startup failure (bad secret mount, config parse error):

oc get pods -n <bob-namespace> -l app=bob-inference \
  -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'

Liveness and readiness probes — check the current probe configuration and status:

oc describe deploy/bob-inference -n <bob-namespace> | grep -A 10 "Liveness\|Readiness"

Periodic health check — the /v1/model/info endpoint serves as a simple liveness check. If it returns a valid model list, the gateway is up. This can be polled from a monitoring tool or a cron job inside the cluster.

Collecting logs and diagnostics

Logs for a specific time window (most useful when investigating a reported incident):

oc logs -n <bob-namespace> deploy/bob-inference \
  --since-time="2025-01-01T12:00:00Z" > bob-inference.log

Logs from a previous pod instance (if the pod has restarted and the failure logs are gone):

oc logs -n <bob-namespace> deploy/bob-inference --previous

Full diagnostic bundle for support:

oc describe pod -n <bob-namespace> -l app=bob-inference >> diagnostics.txt
oc get events -n <bob-namespace> --sort-by='.lastTimestamp' >> diagnostics.txt
oc logs -n <bob-namespace> deploy/bob-inference --since=1h >> diagnostics.txt

Note:

The diagnostic bundle contains pod environment variable names but not secret values (secrets are mounted, not printed to logs). Review logs for any accidental credential output before sending to support.

Troubleshooting connectivity and authentication issues

Model not appearing in /v1/model/info

Check if exposed: false is set — if so, this is expected behavior.

Check startup logs for a registration error for that model_name.

Verify the provider block — correct base_url, model ID, and credentials.

401 Unauthorized on inference requests

Confirm the secret value in bob.modelGateway.secrets is correct and current.

Confirm the env.<VAR> reference in the model config exactly matches the secret key name (case-sensitive).

Restart the Inference Service and retry — the secret may have been updated without a restart.

Test the credential out-of-band. See Authentication validation.

502 Bad Gateway or connection refused

Confirm the model endpoint is up and reachable from the cluster. See Connectivity testing.

Check for base_url issues — trailing slashes, wrong scheme (http vs https), wrong port.

Check for network policy changes that may have blocked outbound traffic since install.

certificate signed by unknown authority

Confirm ca_cert_pem is set and references a valid env var.

Confirm the env var is present in bob.modelGateway.secrets.

Verify the cert is not expired: openssl x509 -noout -dates.

Confirm the cert covers the endpoint's hostname by checking Subject Alternative Names.

Inference Service pod in CrashLoopBackOff

Check logs from the previous instance: oc logs --previous.

Look for config parse errors — malformed YAML in the model gateway config.

Look for missing secret mounts in the Events section: oc describe pod.

Confirm the model gateway config YAML is valid before reapplying.

Operations and troubleshooting