Skip to main content

Examples

See also Register Each Agent, mac test, Configuring Switch Access, Default Login and password recovery.

Switch to a Private Mirror​

Mirror the MAC and SDK images, run docker login as the user who pulls, then run:

sudo mac env set --global MAC_REGISTRY=<registry-host>/<mac-path>
sudo mac env set --global SDK_AGENT_IMAGE=<registry-host>/<sdk-image>:<tag>
mac pull all
sudo mac restart all

Prometheus, Loki and vendor exporter images still come from their usual registries.

Back Up Prometheus Before mac clean​

umask 077
docker stop mac-prometheus
docker cp mac-prometheus:/prometheus/. - > prometheus-backup.tar
docker start mac-prometheus
tar tf prometheus-backup.tar

This holds metrics only. A full backup also needs the API data and alert settings, mac-loki-data, configuration and token files. Store backups in a private directory.

Restore Prometheus​

warning

These commands erase the current metrics. Validate the archive first and use them only on the default MAC volume.

If the container was removed, start the master once to create it. Then run:

docker stop mac-prometheus
docker run --rm --pull=never --user 0 -v mac-prometheus-data:/data --entrypoint sh \
  "$(docker inspect -f '{{.Image}}' mac-prometheus)" -c 'rm -rf /data/* /data/.[!.]*'
docker run --rm -i --pull=never --user 0 -v mac-prometheus-data:/data --entrypoint sh \
  "$(docker inspect -f '{{.Image}}' mac-prometheus)" -c 'tar xpf - -C /data' < prometheus-backup.tar
docker start mac-prometheus

Back Up and Restore Accounts, Enrollment and Alert Settings​

On the master, back up the API data, which includes the key that decrypts stored BMC credentials, accounts, audit records and congestion events. Keep hidden files (.bmc_key, .jwt_secret) and the SQLite database with any WAL/SHM companion files together while api-server is stopped. Also back up the mac-prometheus-targets volume, which holds alert rules, alert acknowledgements and fabric links:

umask 077
docker stop mac-api-server
docker cp mac-api-server:/app/data/. - > api-data-backup.tar
docker cp mac-api-server:/etc/prometheus/targets/. - > targets-backup.tar
docker start mac-api-server
tar tf api-data-backup.tar
tar tf targets-backup.tar
warning

Restoring erases current accounts, enrollment, alert rules and other API data, including changes made after the backup. Validate the archive first and use the commands only on the default MAC volume.

If the container was removed, start the master once to create it. Then run:

docker stop mac-api-server
docker run --rm --pull=never --user 0 -v mac-api-data:/app/data --entrypoint sh \
  "$(docker inspect -f '{{.Image}}' mac-api-server)" -c 'rm -rf /app/data/* /app/data/.[!.]*'
docker run --rm -i --pull=never --user 0 -v mac-api-data:/app/data --entrypoint sh \
  "$(docker inspect -f '{{.Image}}' mac-api-server)" -c 'tar xpf - -C /app/data' < api-data-backup.tar
docker run --rm --pull=never --user 0 -v mac-prometheus-targets:/targets --entrypoint sh \
  "$(docker inspect -f '{{.Image}}' mac-api-server)" -c 'rm -rf /targets/* /targets/.[!.]*'
docker run --rm -i --pull=never --user 0 -v mac-prometheus-targets:/targets --entrypoint sh \
  "$(docker inspect -f '{{.Image}}' mac-api-server)" -c 'tar xpf - -C /targets' < targets-backup.tar
docker start mac-api-server

Check login, enrolled nodes, saved BMC connections and alert rules.

Reload Agent After a Driver Upgrade​

After upgrading the GPU driver, run mac restart agent and mac test --expect agent, then check that GPU metrics appear.