Monitoring Docker Containers using Docker stats, Prometheus and cAdvisor

will walk through how to monitor containers using Docker stats, Prometheus and cAdvisor.

1. Log into the Docker server and then become the root user with a sudo su -

    I already have two containers  running in  live environment: nginx and redis.

[root@ip-10-0-1-212 ~]# docker ps CONTAINER ID   IMAGE     COMMAND                  CREATED       STATUS       PORTS                                      NAMES e80fef4dcb1d   nginx     "/docker-entrypoint.…"   2 hours ago   Up 2 hours   0.0.0.0:80->80/tcp, :::80->80/tcp           nginx 753447b96a14   redis     "docker-entrypoint.s…"   2 hours ago   Up 2 hours   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   redis [root@ip-10-0-1-212 ~]#

2. In root's home directory, create prometheus.yml:

[root@ip-10-0-1-212 ~]# vi prometheus.yml [root@ip-10-0-1-212 ~]# cat prometheus.yml scrape_configs: - job_name: cadvisor   scrape_interval: 5s   static_configs:   - targets:     - cadvisor:8080  [root@ip-10-0-1-212 ~]#

3. To set up Prometheus services, we've got to create configuration files. 

We're going to make docker-compose.yml, right here in the same directory

[root@ip-10-0-1-212 ~]# vi docker-compose.yml [root@ip-10-0-1-212 ~]# cat docker-compose.yml version: '3' services:   prometheus:     image: prom/prometheus:latest     container_name: prometheus     ports:       - 9090:9090     command:       - --config.file=/etc/prometheus/prometheus.yml     volumes:       - ./prometheus.yml:/etc/prometheus/prometheus.yml     depends_on:       - cadvisor     cadvisor:     image: google/cadvisor:latest     container_name: cadvisor     ports:       - 8080:8080     volumes:       - /:/rootfs:ro       - /var/run:/var/run:rw       - /sys:/sys:ro       - /var/lib/docker:/var/lib/docker:ro  [root@ip-10-0-1-212 ~]#

4. To create the cadvisor & Prometheus container run 

[root@ip-10-0-1-212 ~]# docker-compose up -d Creating network "root_default" with the default driver Pulling cadvisor (google/cadvisor:latest)... latest: Pulling from google/cadvisor ff3a5c916c92: Pull complete 44a45bb65cdf: Pull complete 0bbe1a2fe2a6: Pull complete Digest: sha256:815386ebbe9a3490f38785ab11bda34ec8dacf4634af77b8912832d4f85dca04 Status: Downloaded newer image for google/cadvisor:latest Pulling prometheus (prom/prometheus:latest)... latest: Pulling from prom/prometheus aa2a8d90b84c: Pull complete b45d31ee2d7f: Pull complete 7579d86a00c9: Pull complete 8583d0bc7e17: Pull completeb32caf1c5e65: Pull complete e53f205885a2: Pull complete6366df248f46: Pull complete a63db3af7b6e: Pull complete 94cd9f02fa61: Pull complete 2511fa13a76c: Pull complete 50c2584d9f31: Pull complete 22749d939f03: Pull complete Digest: sha256:e9620d250b16ffe0eb9e3eac7dd76151848424c17d577632ae9ca61d1328687e Status: Downloaded newer image for prom/prometheus:latest Creating cadvisor ... done Creating prometheus ... done [root@ip-10-0-1-212 ~]#

5. Verify by Docker ps additional new containers should be running status.

[root@ip-10-0-1-212 ~]# docker ps CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS         PORTS            NAMES d9b2f7d472da   prom/prometheus:latest   "/bin/prometheus --c…"   8 seconds ago    Up 6 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus 3d11890e1cd1   google/cadvisor:latest   "/usr/bin/cadvisor -…"   12 seconds ago   Up 7 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp   cadvisor e80fef4dcb1d   nginx                    "/docker-entrypoint.…"   2 hours ago      Up 2 hours     0.0.0.0:80->80/tcp,     :::80->80/tcp       nginx 753447b96a14   redis                                                       "docker-entrypoint.s…"   2 hours ago      Up 2 hours     0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   redis [root@ip-10-0-1-212 ~]#

6. Viewing the Prometheus Web Interface using the correct port number:                       http://<IP_ADDRESS>:9090/graph/

7. Viewing the cAdvisor Web Interface using the correct port number:                           http://<IP_ADDRESS>:8080/docker/nginx/

       ( Refer Below Video for web interface )

8. Stats in Docker various aspects (CPU and memory usage, etc.) of  containers.

 

     Press crtl + c to exit from stats 

[root@ip-10-0-1-212 ~]# vi  ~/stats.sh [root@ip-10-0-1-212 ~]# cat  ~/stats.sh docker stats --format "table {{.Name}} {{.ID}} {{.MemUsage}} {{.CPUPerc}}" [root@ip-10-0-1-212 ~]# ./stats.sh NAME CONTAINER ID MEM USAGE / LIMIT CPU % prometheus d9b2f7d472da 37.7MiB / 962.8MiB 0.00% cadvisor 3d11890e1cd1 54.3MiB / 962.8MiB 1.56% nginx e80fef4dcb1d 2.215MiB / 962.8MiB 0.00% redis 753447b96a14 7.945MiB / 962.8MiB 0.12% ^C[root@ip-10-0-1-212 ~]#

9. We got Prometheus running, and we're able to monitor our containers with it. Congratulations!

Happy Learning!!!!

Apart from this if you need more clarity,  I have made a  tutorial video on this, 

please go through and share your comments. Like and share the knowledge