Agents & Metrics

The master is healthy but a registered agent isn't reporting.

Agent does not appear on the dashboard

On the agent, confirm services are running: mac status.
Confirm the master is reachable: curl http://<master-ip>:28080/health.
Check the agent registered. mac start agent performs a POST /api/v1/register directly against the master and prints Registered with master <ip>. on success. To confirm from the master side:
```
~$ docker logs mac-api-server 2>&1 | grep '/api/v1/register'
```
From the master, check Prometheus targets:
```
~$ curl -s http://localhost:29090/api/v1/targets | jq '.data.activeTargets[] | {labels, health}'
```
Targets in the down state list a scrape error explaining the failure.

`mac start agent` fails or the node never registers

mac start agent registers synchronously and aborts if it can't reach the master, so a node that never appears is usually one of:

MASTER_IP not set — the CLI prints Error: MASTER_IP is required. Set it and retry:
```
~$ mac env set MASTER_IP=<master-ip>      # saved to ~/.mac/env (per-user)
~$ mac start agent
```
On clusters with a shared (NFS) home directory ~/.mac/env is seen by every node — do not put a per-node value there (and never pin NODE_NAME). Set MASTER_IP in the per-node file /etc/mango/mac/env instead.
Master unreachable — could not register with master at <ip>. Confirm the API port is open from the agent: curl -m5 http://<master-ip>:28080/health should return 200.
NODE_NAME empty — rare; set it explicitly with mac env set NODE_NAME=<name>.
Stale containers from an older deployment — if the agent was once started with a raw docker compose up (Compose project deploy) instead of the CLI (project mac), mac start agent fails with Conflict. The container name "/mac-..." is already in use. Remove the leftovers and start again:
```
~$ docker rm -f $(docker ps -aq --filter 'name=^/mac-')
~$ mac start agent
```

Re-registration is idempotent: the master dedupes by hostname (then IP), so re-running mac start agent reuses the same node and never creates a duplicate, even if the agent's IP changed.

Agent does not come back online after a reboot

Registration survives reboots — the master keeps the inventory in a persistent volume and an agent that merely goes offline stays registered (only a clean mac stop agent deregisters a node). So after a power-cycle a node should return on its own. If it doesn't:

Docker isn't starting at boot. The agent containers are restart: unless-stopped, but that only helps once the Docker service itself starts — the package does not enable it:
```
~$ sudo systemctl enable --now docker
~$ mac status            # containers should be Up
```
IPv4 forwarding was reset — see Agent shows "offline" although its containers are healthy below; ip_forward can revert to 0 at boot or on a NIC carrier event.
You re-ran mac start agent and it failed — that path needs MASTER_IP (see the previous section). A plain reboot does not re-run it and does not need to.

Agent shows "offline" although its containers are healthy (`ip_forward`)

Symptom. mac status on the agent shows every container Up (healthy), but the dashboard lists the node as Offline and most exporters are missing from Prometheus — yet the NIC bandwidth chart (nic-exporter, port 29500) still works.

Cause. The master scrapes each agent exporter over the host's published Docker-bridge ports. If the kernel has IPv4 forwarding disabled (net.ipv4.ip_forward = 0), packets that the host DNATs to an exporter container are never forwarded, so every bridge-networked exporter is unreachable from the master: sys-exporter (29101), bmc-agent (29102), dcgm-exporter (29150), rocm-exporter (29200), sw-exporter (29300). The master's health probe targets sys-exporter (29101); when it times out the node is marked offline and dropped from the Prometheus targets file. nic-exporter keeps working because it runs on the host network (no forwarding needed), which makes the failure look selective.

Diagnose.

# From the master: the host-network exporter answers, the bridge ones time out.
~$ curl -m4 -o /dev/null -w '%{http_code}\n' http://<agent>:29500/metrics   # 200
~$ curl -m4 -o /dev/null -w '%{http_code}\n' http://<agent>:29101/metrics   # 000 (timeout)

# On the agent: forwarding is off, but the exporter is fine locally.
~$ cat /proc/sys/net/ipv4/ip_forward                                        # 0  ⇒ the cause
~$ curl -s -o /dev/null -w '%{http_code}\n' http://<agent-ip>:29101/metrics # 200 from the node itself

Immediate fix.

~$ sudo sysctl -w net.ipv4.ip_forward=1

The node returns to Online on the master's next health poll (≤ 30 s).

Make it durable. Persisting the sysctl is necessary but not sufficient on Kubernetes / DPU hosts: a runtime network reconfiguration (observed on RoCE NICs gaining carrier — systemd-networkd ... Link UP / Gained carrier) resets the global flag after boot, so the /etc/sysctl.d value gets clobbered mid-run.

# 1) Persist across reboot.
~$ echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-mac-ipforward.conf
~$ sudo sysctl --system

# 2) Have systemd-networkd re-assert it on every link (re)configuration. networkd only ever
#    turns ip_forward ON, never off, so enabling it on a managed link self-heals the flag
#    after a carrier event resets it. Use a drop-in so netplan-generated profiles are kept:
~$ sudo mkdir -p /etc/systemd/network/10-netplan-<iface>.network.d
~$ printf '[Network]\nIPForward=yes\n' | \
     sudo tee /etc/systemd/network/10-netplan-<iface>.network.d/ipforward.conf
~$ sudo networkctl reconfigure <iface>

Replace <iface> with the node's primary interface (see networkctl list). On systemd ≥ 256 the key is IPv4Forwarding=yes.

Alternative (MAC-side): running the bridge exporters on the host network — the way nic-exporter already does — removes the dependency on ip_forward entirely.

No GPU metrics on a registered NVIDIA node

The dcgm-exporter container relies on the NVIDIA Container Toolkit. Verify:

~$ docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

If this fails, install the toolkit (sudo apt install nvidia-container-toolkit, sudo nvidia-ctk runtime configure --runtime=docker, sudo systemctl restart docker), then mac restart agent.

No GPU metrics on a registered AMD node

The rocm-exporter requires /dev/kfd and /dev/dri to be available inside the container.

~$ ls -l /dev/kfd /dev/dri
~$ docker exec mac-rocm-exporter rocm-smi

If rocm-smi cannot see the GPUs, the kernel module (amdgpu) may not be loaded. Confirm with lsmod | grep amdgpu.

NIC bandwidth chart is flat

For NVIDIA Mellanox and MangoBoost GPUBoost, RDMA bandwidth is read via ethtool -S. Inside the container, run docker exec mac-nic-exporter ethtool -S <iface> | grep -i rdma and confirm the counters increment under load.
For Broadcom Thor2 (BCM57608), bandwidth is read from the IB hardware counters in /sys/class/infiniband/<ibdev>/ports/1/hw_counters/{rx,tx}_bytes. The container needs read access to /sys; this is set by default in docker-compose.agent.yml.

Agent Health Endpoints

Service	URL
dcgm-exporter	`http://<agent>:29150/metrics`
rocm-exporter	`http://<agent>:29200/metrics`
nic-exporter	`http://<agent>:29500/metrics`
sys-exporter	`http://<agent>:29101/metrics`
sw-exporter	`http://<agent>:29300/metrics`
bmc-agent	`http://<agent>:29102/health`
sys-manager	`http://<agent>:29103/health`
mango-sdk-agent	`http://<agent>:28000/docs` (FastAPI docs page; 200 OK ⇒ alive)

Agent does not appear on the dashboard​

mac start agent fails or the node never registers​

Agent does not come back online after a reboot​

Agent shows "offline" although its containers are healthy (ip_forward)​

No GPU metrics on a registered NVIDIA node​

No GPU metrics on a registered AMD node​

NIC bandwidth chart is flat​

Agent Health Endpoints​