How to diagnose Kubernetes alerts with an n8n agent and MCP servers

Engineering & securityn8n

The problem

When an alert fires at night, the on-call engineer spends the first minutes gathering context from several dashboards before they can start fixing anything.

Tools needed

• n8n
• Prometheus Alertmanager
• Kubernetes, Grafana, DigitalOcean and GitHub MCP servers
• Qdrant (internal docs)
• Mattermost

Step-by-step recipe

1. Point an Alertmanager webhook at an n8n workflow.
2. Deduplicate alerts over a window (48 hours in the source) and fetch the triggering PromQL expression from the Prometheus Rules API.
3. Let an AI agent investigate read-only through Kubernetes, Grafana, cloud and GitHub MCP servers plus a vector store of internal docs.
4. Have it write a report: summary, event timeline, up to two root-cause hypotheses and troubleshooting steps.
5. Find the original alert post in Mattermost and reply in its thread.

Copyable prompt

An alert fired: {{alert_name}} with expression {{promql}} and labels {{labels}}.
Investigate using the read-only tools available (Kubernetes, Grafana, docs). Do not change anything.
Reply with: 1) a one-paragraph summary, 2) a timeline of relevant events, 3) up to two root-cause hypotheses with the evidence for each, 4) next troubleshooting steps for the on-call engineer. If a tool call fails, retry once, then continue without it and say what's missing.

What the builder reported

The source gives no measured results; it describes the report format. The author was still weighing extra data sources such as database stats and pipeline status.

These are the source author's own claims. AgentRecipes has not tested or verified them.

Watch out for

• Limit tool retries to one per failed call to avoid loops, and strip noisy labels such as job and instance from the prompt.
• Keep the agent read-only; you configure your own MCP endpoints and credentials.

Source

More engineering & security recipes