Monitor, maintain, and troubleshoot Model Gateway deployments, including model configuration updates, credential rotation, health monitoring, log collection, and resolution of common connectivity and authentication issues.

After deployment, ongoing operational management of the Model Gateway helps ensure reliable access to configured AI models and minimizes service disruptions. Administrators can review active model configurations, update provider credentials, rotate secrets, monitor gateway health, and collect diagnostic information when issues occur.

Viewing configured models

There are two levels to viewing models — what is in the config and what bob-inference has actually loaded at runtime.

Config level — retrieve the current model gateway config from the inference ConfigMap:

oc get cm bob-inference-config -n <bob-namespace> -o yaml | yq '.data."config.yaml"'

Runtime level — what bob-inference has registered and is actively routing to: query the /v1/models endpoint directly (see Validate model connectivity).

Note:

The runtime /v1/models endpoint only lists models where exposed: true. The config file is the only way to see the full set of configured models.

Updating provider credentials

Credentials are mounted as environment variables from bob.modelGateway.secrets. Updating them is a two-step operation — update the secret value, then restart the Inference Service to pick up the new mount.

Update the secret

Edit the Bob CR and apply the new value:

oc edit bob <instance-name> -n <bob-namespace>
# Update the value under bob.modelGateway.secrets

Restart the Inference Service

Restart the Inference Service so the new secret is mounted:

oc rollout restart deploy/inference-service -n <bob-namespace>
oc rollout status deploy/inference-service -n <bob-namespace>

Warning:

The Inference Service pod must be restarted after any secret update. Model config changes (adding or removing models, changing base_url) and secret changes can be batched into a single CR update followed by one restart.

Rotating secrets

Secret rotation follows the same pattern as a credential update, with additional care around timing to avoid downtime.

Recommended zero-downtime rotation sequence:

Update bob.modelGateway.secrets with the new credential value.

Restart the Inference Service: oc rollout restart deploy/inference-service -n <bob-namespace>.

Confirm the new credential is working using the inference test from Post-installation verification.

Revoke the old credential at the provider side only after the pod is confirmed healthy.

Provider-specific notes:

openai_compatible / API keys — new key is effective immediately on restart. Safe to revoke the old key once the pod is healthy.

bedrock — ensure the new IAM access key is active in AWS before restarting. IAM propagation can take a few seconds.

vertex — generate and base64-encode the new service account key, update the secret, restart and verify, then delete the old key in GCP.

ca_cert_pem — for certificate renewal, verify the new cert is not expired before applying. See TLS certificate validation.

Monitoring gateway health

Pod restart count — a rising restart count is an early signal of a recurring startup failure (bad secret mount, config parse error):

oc get pods -n <bob-namespace> -l app=bob-inference \
  -o custom-columns='NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount'

Liveness and readiness probes — check the current probe configuration and status:

oc describe deploy/bob-inference -n <bob-namespace> | grep -A 10 "Liveness\|Readiness"

Periodic health check — the /v1/models endpoint serves as a simple liveness check. If it returns a valid model list, the gateway is up. This can be polled from a monitoring tool or a cron job inside the cluster.

Collecting logs and diagnostics

Logs for a specific time window (most useful when investigating a reported incident):

oc logs -n <bob-namespace> deploy/bob-inference \
  --since-time="2025-01-01T12:00:00Z" > bob-inference.log

Logs from a previous pod instance (if the pod has restarted and the failure logs are gone):

oc logs -n <bob-namespace> deploy/bob-inference --previous

Full diagnostic bundle for support:

oc describe pod -n <bob-namespace> -l app=bob-inference >> diagnostics.txt
oc get events -n <bob-namespace> --sort-by='.lastTimestamp' >> diagnostics.txt
oc logs -n <bob-namespace> deploy/bob-inference --since=1h >> diagnostics.txt

Note:

The diagnostic bundle contains pod environment variable names but not secret values (secrets are mounted, not printed to logs). Review logs for any accidental credential output before sending to support.

Troubleshooting connectivity and authentication issues

Model not appearing in /v1/models

Check if exposed: false is set — if so, this is expected behavior.

Check startup logs for a registration error for that model_name.

Verify the provider block — correct base_url, model ID, and credentials.

401 Unauthorized on inference requests

Confirm the secret value in bob.modelGateway.secrets is correct and current.

Confirm the env.<VAR> reference in the model config exactly matches the secret key name (case-sensitive).

Restart the Inference Service and retry — the secret may have been updated without a restart.

Test the credential out-of-band. See Authentication validation.

502 Bad Gateway or connection refused

Confirm the model endpoint is up and reachable from the cluster. See Connectivity testing.

Check for base_url issues — trailing slashes, wrong scheme (http vs https), wrong port.

Check for network policy changes that may have blocked outbound traffic since install.

certificate signed by unknown authority

Confirm ca_cert_pem is set and references a valid env var.

Confirm the env var is present in bob.modelGateway.secrets.

Verify the cert is not expired: openssl x509 -noout -dates.

Confirm the cert covers the endpoint's hostname by checking Subject Alternative Names.

Inference Service pod in CrashLoopBackOff

Check logs from the previous instance: oc logs --previous.

Look for config parse errors — malformed YAML in the model gateway config.

Look for missing secret mounts in the Events section: oc describe pod.

Confirm the model gateway config YAML is valid before reapplying.

Operations and troubleshooting