> For the complete documentation index, see [llms.txt](https://uclix.gitbook.io/run-ai-rca-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://uclix.gitbook.io/run-ai-rca-docs/readme.md). # English **๐Ÿ‡ฌ๐Ÿ‡ง English** ยท [๐Ÿ‡ฐ๐Ÿ‡ท ํ•œ๊ตญ์–ด](/run-ai-rca-docs/ko.md) ## Run:AI RCA Run:AI RCA is a KubeRCA-inspired incident analysis cockpit for NVIDIA Run:ai environments: Alertmanager intake, incident/alert dashboards, structured RCA reports, realtime updates, chat, and reusable incident memory. Instead of a single agent, it uses a component-oriented multi-agent design with the NVIDIA NeMo Agent Toolkit as the orchestration backbone. RCA is read-only by default and degrades gracefully when Run:ai, Prometheus, Loki, or Kubernetes access is missing. ### Repository Layout ``` agent/ FastAPI analysis service and NeMo Agent Toolkit workflow config backend/ Go API server for Alertmanager intake, incidents, alerts, SSE frontend/ React dashboard charts/ Helm chart for Kubernetes deployment docs/ Architecture and operation notes ``` ### Architecture ![Run:AI RCA architecture](/files/hz1cxDUEHGx2mQHUJhoT) The diagram shows the components and the external systems the Agent reads. Inside the Agent, an **orchestrator** runs the analysis pipeline โ€” planner โ†’ seven parallel collectors โ†’ central investigation loop and per-collector drill-down โ†’ signature matching, ranking, a skeptical self-check โ†’ synthesis โ€” detailed in [RCA Pipeline](/run-ai-rca-docs/readme/rca-pipeline.md). The orchestrator itself consults the optional **TypeDB ontology** (`typedb.enabled`, default on in Helm) for relational facts pgvector can't express โ€” node blast radius, prior same-alert incidents, and graph-derived family/XID remediation โ€” populated from eligible resolved incidents in the Postgres store. Set `typedb.ingest.requireApproval=true` when only operator-approved incidents should enter the graph. **pgvector** similarity is owned by the backend, which passes similar incidents and feedback hints into each analysis request. Full walkthrough: [RCA Pipeline](/run-ai-rca-docs/readme/rca-pipeline.md) ยท [Knowledge Base](/run-ai-rca-docs/readme/knowledge-base.md). ### Local Development ```bash # Agent cd agent && python -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" && uvicorn app.main:app --reload --port 8000 # Backend cd backend && go test ./... && go run . # Frontend cd frontend && npm install && npm run dev ``` The frontend expects the backend at `http://localhost:8080` by default. ### Deployment Container images and the Helm chart are published to GHCR on `main` pushes and version tags (`v*.*.*`). Pull requests build/lint only. Images are tagged with the chart `appVersion` plus `sha-...`; the chart is published as an OCI artifact. * `ghcr.io//runai-rca-backend`, `-agent`, `-frontend` * `ghcr.io//charts/runai-rca` #### 1. Secret The backend auto-creates the target database if it is missing (needs `CREATEDB`, or pre-create it). Existing databases are never modified. ```bash kubectl create namespace runai-rca kubectl create secret generic runai-rca-secrets -n runai-rca \ --from-literal=DATABASE_URL='postgres://user:pw@pg-host:5432/runai_rca?sslmode=require' \ --from-literal=POSTGRES_DSN='postgres://user:pw@pg-host:5432/runai_rca?sslmode=require' \ --from-literal=RUNAI_CLIENT_ID='' \ --from-literal=RUNAI_CLIENT_SECRET='' \ --from-literal=GRAFANA_SERVICE_ACCOUNT_TOKEN='' ``` #### 2. Install ```bash helm upgrade --install runai-rca oci://ghcr.io//charts/runai-rca \ --version -n runai-rca \ --set global.imageRegistry=ghcr.io/ \ --set secrets.existingSecret=runai-rca-secrets \ --set agent.env.runaiBaseUrl=https://runai.example.com \ --set agent.env.prometheusUrl=http://prometheus.monitoring.svc:9090 \ --set agent.env.lokiUrl=http://loki-read.monitoring.svc.cluster.local:3100 ``` Bundled single-pod Postgres instead of an external DB: `--set postgresql.enabled=true`. #### LLM synthesis (optional) RCA synthesis runs through the in-process NeMo Agent Toolkit engine by default. To let NAT own the default LLM transport through an OpenAI-compatible endpoint (e.g. LiteLLM): ```bash --set agent.env.enableNatRuntime=true \ --set agent.env.natConfigFile=/app/configs/runai_rca_engine.yml \ --set agent.env.llmBaseUrl=https://llm.example.com/v1 \ --set agent.env.llmModel= \ --set secrets.llmApiKey='' ``` Workflow config: `runai_rca_engine.yml`. #### Runtime checks Automatic RCA starts only after Alertmanager posts to Backend `/webhook/alertmanager`; a Slack notification alone does not prove that the RCA webhook receiver was routed. Check live intake and analysis state with: ```bash curl -s http:///api/v1/alerts curl -s http:///api/v1/analysis-runs ``` Agent `/healthz` means the Agent API process is alive. Collector cards in the UI turn `ok` only after an RCA run stores collector `artifacts`; pod `Running` or health `200` is not enough by itself. Chat is context-grounded from the active incident/alert RCA content. In the current implementation it does not call the LLM path directly; `ENABLE_NAT_RUNTIME=true` affects `/analyze` synthesis, while `/chat` returns a deterministic context answer. When no detail RCA is attached, Backend supplies dashboard and analysis-run state so Chat can report current alerts, latest run status, agent timeout/failure warnings, and configured runtime mode. ### Configuration Key values (full secret keys: `DATABASE_URL`, `POSTGRES_DSN`, `RUNAI_CLIENT_ID`, `RUNAI_CLIENT_SECRET`, `RUNAI_BEARER_TOKEN`, `NVIDIA_API_KEY`, `LLM_API_KEY`): | Helm value | Purpose | | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `global.imageRegistry` / `imagePullSecrets` | Registry prefix and pull secrets for all images | | `secrets.existingSecret` | Existing Secret with DB/Run:ai/NVIDIA/LLM credentials | | `agent.env.runaiBaseUrl` / `runaiTokenUrl` | Run:ai API URL; the token endpoint is auto-derived from it (documented `/api/v1/token` first) โ€” set `runaiTokenUrl` only as an override for unusual deployments | | `runaiMcp.oidcIssuerUrl` | Token `iss` URL for the official Run:ai MCP when the control-plane root OIDC discovery endpoint returns HTML | | `agent.env.prometheusUrl` / `lokiUrl` | In-cluster Prometheus / Loki URLs. Loki defaults to the direct read service, not the authenticated gateway. | | `grafanaMcp.grafanaUrl` / `grafanaOrgId` | Grafana endpoint and organization used by the shared MCP service; its service-account token is `GRAFANA_SERVICE_ACCOUNT_TOKEN`. | | `typedb.ingest.requireApproval` | Ingest only Dashboard-approved (`user_approved_at`) resolved incidents. Defaults to `true`; `requireReview` is deprecated. | | `agent.env.enableNatRuntime` / `natConfigFile` | Enable NeMo synthesis and select workflow config | | `agent.env.llmBaseUrl` / `llmModel` | OpenAI-compatible endpoint and model | | `agent.rbac.clusterWide` / `namespaces` | Read-only RBAC scope for evidence collection | | `postgresql.enabled` / `auth.*` | Use bundled Postgres and its user/password/database | | `ingress.*` | Frontend host, TLS, class, annotations | | `{backend,agent,frontend}.image.tag` | Override image tags (default: chart appVersion) | RCA tables are created automatically with idempotent `CREATE TABLE IF NOT EXISTS`; no migration step is needed. pgvector is used when available, otherwise the backend falls back to JSONB cosine search. Sensitive values are redacted before evidence leaves a collector; add patterns via `MASKING_REGEX_LIST_JSON`. ### Documentation Full table of contents (GitBook-ready): [`SUMMARY.md`](https://github.com/uclix-nvidia-sw/runaiRCA/tree/main/SUMMARY.md). * [Getting Started](/run-ai-rca-docs/readme/getting-started.md) โ€” run locally and get your first RCA * [Architecture](/run-ai-rca-docs/readme/architecture.md) โ€” implementation contract * [RCA Pipeline](/run-ai-rca-docs/readme/rca-pipeline.md) โ€” every analysis stage, planner โ†’ synthesis * [Learning and Ontology Guide](/run-ai-rca-docs/readme/learning-and-ontology.md) โ€” probes, diagnoses, trace-v3, and safe knowledge publication in plain language * [Knowledge Base](/run-ai-rca-docs/readme/knowledge-base.md) โ€” curated catalogs + TypeDB ontology * [Ontology & Ingestion Guide](/run-ai-rca-docs/readme/ontology-guide.md) โ€” approved RCA graph model, data-shaping rules, and Studio checks * [Evaluation & Runtime Harness](/run-ai-rca-docs/readme/evaluation.md) โ€” output gates, repair, and operator scoring * [Operating Model](/run-ai-rca-docs/readme/operating-model.md) โ€” operating model * [Data Stores](/run-ai-rca-docs/readme/database.md) โ€” PostgreSQL + TypeDB ontology * [UI Direction](/run-ai-rca-docs/readme/ui-direction.md) โ€” UI/UX direction * [Deployment](/run-ai-rca-docs/readme/deployment.md) โ€” detailed deployment, RBAC, and DB notes * [API Reference](/run-ai-rca-docs/readme/api.md) โ€” backend and agent endpoints * [Configuration Reference](/run-ai-rca-docs/readme/configuration.md) โ€” full env var and Helm value reference --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://uclix.gitbook.io/run-ai-rca-docs/readme.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.