Skip to main content

Agents & Metrics

Charts can keep old data during an outage, so check sample timestamps.

Agent does not appear on the dashboard​

Check on the agent: mac status -v, mac env and curl -m5 http://<master-ip>:28080/health.

Fix:

  • Set MASTER_IP and the master's MAC_NODE_TOKEN node-wide, as in Register Each Agent, then run mac start agent and read its registration result.
  • Give nodes with duplicate hostnames a unique node-wide NODE_NAME.
  • On a container name conflict, remove only containers that docker inspect <container-name> confirms are old MAC containers.

Agent does not come back online after a reboot​

Fix: Enable Docker at boot (systemctl is-enabled docker), then start containers that were stopped before the reboot with mac start <mode>. For "port is already allocated", see stale Docker proxies.

Agent shows "offline" although its containers are healthy (ip_forward)​

Usually only NIC metrics still arrive. Check: cat /proc/sys/net/ipv4/ip_forward on the agent.

Fix: If it is 0, run sudo sysctl -w net.ipv4.ip_forward=1 and persist it in /etc/sysctl.d/ and your network manager. The node returns Online within about 30 seconds.

No GPU metrics on a registered NVIDIA node​

Check: nvidia-smi and mac logs dcgm-exporter.

Fix: Configure NVIDIA Container Toolkit (its Docker restart affects other containers), then run mac restart agent. If mac start agent exits with code 4, the loaded driver and its libraries do not match; fix the host driver first.

No GPU metrics on a registered AMD node​

Check: ls -l /dev/kfd /dev/dri, lsmod | grep amdgpu and mac logs rocm-exporter. Readings a card does not provide stay unavailable rather than zero.

AMD partition monitoring recovery​

After a partition change, MAC restarts its ROCm exporter to match without changing GPU settings. If readings stay wrong, check mac logs sys-manager.

Sanity fails while metrics remain current​

Sanity also checks stopped MAC containers, such as one left from an earlier deployment. Confirm the owner with docker inspect <exact-container-name> before changing it.

NIC bandwidth chart is flat​

The RDMA bandwidth chart shows throughput, not link capacity; idle ports read zero.

No node logs or an empty Dmesg tab​

Check: mac logs sys-exporter, master reachability and the node token. Compare an empty Dmesg tab with the Journal and Syslog tabs. Monitoring → System log needs an operator or admin account.

SDK actions return an error​

  • 401: Match the node tokens (Node Authentication), then restart the master and the agent.
  • 409: apt or dnf on the host is busy; retry later.
  • 502: Check package repository access and mac logs mango-sdk-agent.
  • 503: Check mac status on the agent.

SDK Package or DPU Reads Fail on a RHEL-Family Node​

Fix: Set node-wide SDK_AGENT_IMAGE to the RPM-capable image supplied for your deployment and run mac restart agent. Source-based jobs also need /etc/yum.repos.d/mango-sdk.repo (Software).