How I'd monitor a Proxmox cluster with Checkmk (and why I'd run Pulse next to it)

A Proxmox cluster with Ceph has plenty of parts that can fail quietly: an OSD going down, a disk filling up, a backup failing every other night, a node losing quorum. The Proxmox UI shows you all of it if you go and look, but nobody goes and looks at 3 a.m. If I had to set it up from scratch, I’d use two tools with different jobs: Checkmk for alerts and history, and Pulse to see the state of everything at a glance. Here’s how, step by step, plus the traps worth knowing about before you fall into them.

Everything here is generic: no server or organisation names. Node names, the Checkmk site and the domain are examples.

1. Why two tools instead of one

It sounds redundant, but each one does one thing well:

• Checkmk is the “real” monitoring: it watches Proxmox, but also the switches, the Windows boxes, the printers and anything else you need. It has thresholds, history, scheduled downtimes and rule-based notifications. Its weak spot is that the UI isn’t built for a quick glance at a virtualisation cluster.
• Pulse is a dashboard made for Proxmox (it also covers Docker, PBS and a few other things). One screen shows you the nodes, which VM is eating CPU, how storage is doing and which backups failed. It installs in five minutes and reads the API with a read-only token.

The rule I’d follow: Checkmk alerts, Pulse is for looking. More on that at the end.

2. A read-only user for Checkmk

Checkmk talks to Proxmox over the API through its special agent. Don’t give it root: create a dedicated user with the PVEAuditor role, which can only read. On any cluster node:

pveum user add checkmk@pve --password 'A_LONG_PASSWORD' --comment "Monitoring (read-only)"
pveum acl modify / --users checkmk@pve --roles PVEAuditor

The @pve realm is Proxmox’s own user database, so it doesn’t depend on the node’s OS or your directory. Since the cluster shares its configuration, doing it on one node covers them all.

If you followed part 2 of the Proxmox series, this will look familiar: it’s the same user.

3. Adding the nodes to Checkmk

In Checkmk, create one host per node (pve-node1, pve-node2, pve-node3). Under Monitoring agents, pick Configured API integrations and Checkmk agent, because we’re going to use both.

Then the special agent rule:

1. Setup > Agents > VM, cloud, container > Proxmox VE > Add rule.
2. User checkmk@pve and its password (ideally from Checkmk’s password store rather than typed into the rule).
3. Under Conditions, restrict the rule to the node hosts only. Otherwise Checkmk will try to talk to the Proxmox API on every host in the folder.
4. Leave certificate verification on if the nodes have a valid certificate. Turning it off is the easy way out, but then you’ll trust anything answering on port 8006.

Save, run a service discovery on the nodes and activate the changes. You should now see cluster status, storage and backups.

First trap: DNS

If discovery fails without much of a clue, first check whether the Checkmk server can resolve the node names. Proxmox returns nodes by their short name, and if Checkmk doesn’t know which IP pve-node2 has, the query fails. The proper fix is putting them in DNS. The quick workaround is adding them to /etc/hosts on the Checkmk server (or container):

192.168.10.11  pve-node1.your-domain.com  pve-node1
192.168.10.12  pve-node2.your-domain.com  pve-node2
192.168.10.13  pve-node3.your-domain.com  pve-node3

4. The regular agent and Ceph

Here’s the second trap: the Proxmox API doesn’t tell you how Ceph is doing in the detail you need. To know whether there are OSDs down, degraded placement groups or the cluster sitting in HEALTH_WARN, you need the regular Checkmk agent on every node, with the mk_ceph plugin.

You download the agent package from your own Checkmk server (Setup > Agents > Linux has the exact link for your version):

# On every node; swap server, site and version for yours
wget https://checkmk.your-domain.com/monitor/check_mk/agents/check-mk-agent_2.5.0-1_all.deb
dpkg -i check-mk-agent_2.5.0-1_all.deb

# Register it so the connection is encrypted
cmk-agent-ctl register --hostname pve-node1 \
  --server checkmk.your-domain.com --site monitor --user automation

And the Ceph plugin, which ships with the Checkmk server itself:

wget -O /usr/lib/check_mk_agent/plugins/mk_ceph.py \
  https://checkmk.your-domain.com/monitor/check_mk/agents/plugins/mk_ceph.py
chmod +x /usr/lib/check_mk_agent/plugins/mk_ceph.py

# Check that the agent now returns the Ceph section
check_mk_agent | grep -A3 '<<<ceph'

If Ceph sections show up, run discovery again in Checkmk and the Ceph services will appear.

5. VMs as hosts (piggyback)

The special agent doesn’t just bring node data: it also brings data for every VM and container (CPU, memory, disk, whether it’s running, its last backup). Checkmk stores it as piggyback data, riding on another host. To see it, you need to create a Checkmk host per VM with exactly the same name it has in Proxmox.

• If the VM has no Checkmk agent inside, set Monitoring agents to No API integrations, no Checkmk agent and, under Piggyback, Always use and expect piggyback data, so you get warned if it stops arriving.
• If the names don’t match (web01 in Proxmox, web01.your-domain.com in Checkmk), use the Hostname translation for piggybacked hosts rule: strip the domain, lowercase, or a regex. Apply it to the Proxmox nodes only.
• To see which VMs are sending data nobody picks up:

# On the Checkmk server, as the site user
cmk-piggyback list orphans

If the VM matters, give it its own agent too: piggyback tells you how Proxmox sees it from outside, the agent tells you what’s going on inside (services, system disk, updates).

6. Pulse in an LXC

Pulse installs as a service in an LXC container or with Docker, and listens on port 7655. I’d put it in a small LXC (1 core, 1 GB, 8 GB of disk), just like I did with PegaProx. The official installer is signed. The block in its README downloads it, verifies the signature and only then runs it. Copy it from the project page with the exact version of the latest release; I’m not pasting it here because it changes with every version and you’d end up with a stale one.

Then:

1. Open http://LXC_IP:7655 and paste the one-time setup token (sudo pulse bootstrap-token inside the LXC).
2. Create your admin account.
3. Settings > Infrastructure > Add infrastructure and pick Proxmox VE. Pulse gives you a script that creates a monitoring user with a read-only token on the node. Read it before you paste it: it’s your cluster.

A couple of minutes later you have every node, VM, storage pool and backup on one screen, with seven days of history in the free edition. For Proxmox you don’t need any agent: it reads the API.

One warning from the project itself that’s worth reading: if Pulse queries VMs through the QEMU Guest Agent, that’s the same channel freeze-enabled backups use. Don’t change permissions or restart agents in the middle of a backup.

7. Who alerts on what

With both tools running, the easy move is turning notifications on in both. I wouldn’t: two alerts about the same full disk at 3 a.m. is the fastest way to get someone to mute both.

My split:

• Checkmk notifies: node down, Ceph in HEALTH_WARN or HEALTH_ERR, storage above 80%, failed backup, lost quorum. With escalation if nobody reacts.
• Pulse is for looking: during an incident or before maintenance, to see in ten seconds which VM is where and how much it’s using. At most I’d leave it an alert Checkmk doesn’t cover, like update notices.

Scheduled downtime during maintenance

Before rebooting a node, set Ceph’s noout flag (so it doesn’t start moving data the moment the node disappears) and schedule a downtime in Checkmk for that node and its Ceph services:

ceph osd set noout     # before shutting down the node
# ... maintenance ...
ceph osd unset noout   # once the node is back and the OSDs are "up"

With noout set, Ceph sits in HEALTH_WARN on purpose. Without a scheduled downtime, Checkmk will alert you about something you already know, and the next time it alerts for real nobody will listen. And above all, don’t forget the unset: Checkmk will keep flagging the warning until you remove it, which is exactly what it’s for.

What not to do

• Using root@pam for monitoring. If someone gets the password off the monitoring server, they have your whole cluster. PVEAuditor and move on.
• Relying on the API alone for Ceph. Without the agent and mk_ceph, you can have an OSD down and everything green.
• Creating VM hosts with “similar” names. Piggyback matches on the exact name; if it doesn’t match, the data is orphaned and nobody warns you.
• Turning off certificate checks “because it works that way”. Put a valid certificate on the nodes or import the CA into Checkmk.
• Notifying the same thing from two places. One source of alerts.

Summary

1. User checkmk@pve with the PVEAuditor role on /.
2. Nodes in Checkmk with the special agent + regular agent, and the rule restricted to the nodes.
3. mk_ceph on every node to really know how Ceph is doing.
4. VMs as piggyback hosts with the exact name, and cmk-piggyback list orphans for the missing ones.
5. Pulse in a small LXC, with a read-only token, for the at-a-glance view.
6. Checkmk alerts, Pulse is for looking, and always a scheduled downtime when you touch a node.

Checkmk takes more effort to set up, but it’s the one that wakes you up when it matters. Pulse is the one you open once you’re awake and want to understand what’s going on. Together they cover both without stepping on each other.