3

I am using Ubuntu server with a Docker Service installed from the Server Setup (snap) and Prometheus running in a docker container.

Server:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

Docker

sudo snap services
Service                          Startup  Current   Notes
docker.dockerd                   enabled  active    -
docker.nvidia-container-toolkit  enabled  inactive  -
lxd.activate                     enabled  inactive  -
lxd.daemon                       enabled  inactive  socket-activated
lxd.user-daemon                  enabled  inactive  socket-activated

snap

snap    2.61.1
snapd   2.61.1
series  16
ubuntu  22.04
kernel  5.15.0-91-generic

Prometheus docker-compose.yml

version: '3.8'
volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - /home/xxxx/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'


extra_hosts:
  - "host.docker.internal:host-gateway"

ports:
  - "9090:9090/tcp"

Docker IP:

ip addr show docker0
    8: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
        link/ether 02:42:18:5c:09:7f brd ff:ff:ff:ff:ff:ff
        inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
           valid_lft forever preferred_lft forever
        inet6 fe80::42:18ff:fe5c:97f/64 scope link
           valid_lft forever preferred_lft forever

Issue

So what I want to do is get metrics for my Prometheus Server running in a docker container. I am using this documentation: https://docs.docker.com/config/daemon/prometheus/

In the beginning it says: Add the following configuration:

Linux: /etc/docker/daemon.json

{
  "metrics-addr": "127.0.0.1:9323"
}

I also tried with:

{
   "metrics-addr": "172.17.0.1:9323"
}

After some trouble I found following question / answer on askubuntu: https://askubuntu.com/questions/1296892/snap-edit-docker-configuration

So I added the daemon.json to

/var/snap/docker/current/etc/docker

and restarted the service with:

sudo snap restart docker

after the restart I still do not get any metrics from the host. I test this using:

curl http://localhost:9323/metrics
curl: (7) Failed to connect to localhost port 9323 after 0 ms: Connection refused
curl http://172.17.0.1:9323/metrics
curl: (7) Failed to connect to 172.17.0.1 port 9323 after 0 ms: Connection refused

Prometheus Target Error

Docker system info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.4)
  compose: Docker Compose (Docker Inc., v2.17.2)

Server:
 Containers: 20
  Running: 20
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 20.10.24
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-91-generic
 Operating System: Ubuntu Core 22
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.26GiB
 Name: blackholeport
 ID: AMKQ:XQSW:OPFY:ZLWE:VCVI:BKBB:NVO4:5WYJ:W6AZ:JB7M:7T4S:WXNC
 Docker Root Dir: /var/snap/docker/common/var-lib-docker
 Debug Mode: false
 Username: hagbart
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

docker inspect prometheus

[
    {
        "Id": "ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684",
        "Created": "2024-01-19T10:23:03.520072505Z",
        "Path": "/bin/prometheus",
        "Args": [
            "--config.file=/etc/prometheus/prometheus.yml"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 83237,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-01-19T13:00:24.320370241Z",
            "FinishedAt": "2024-01-19T13:00:17.400204943Z"
        },
        "Image": "sha256:2a72b385beafd2b3685ad337d14d23f6851f0ff335c5cd9a04c692b3eb846196",
        "ResolvConfPath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/resolv.conf",
        "HostnamePath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/hostname",
        "HostsPath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/hosts",
        "LogPath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684-json.log",
        "Name": "/prometheus",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/lukas/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "prometheus_default",
            "PortBindings": {
                "9090/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9090"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [
                "host.docker.internal:host-gateway"
            ],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "prometheus_prometheus_data",
                    "Target": "/prometheus",
                    "VolumeOptions": {}
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199-init/diff:/var/snap/docker/common/var-lib-docker/overlay2/0aa7d62082adf6dcde04943d937cac5dae5d0609b12ae7319421cbf63131fca3/diff:/var/snap/docker/common/var-lib-docker/overlay2/59878a50f72e66a191110dadd15bf94d352cb1e0a543cf6ab0e96f210d9a16e5/diff:/var/snap/docker/common/var-lib-docker/overlay2/d7dd0384e6e888b48e94d3f32bf1e504c2ed1d11526ba5caba7a6cbaa43f1304/diff:/var/snap/docker/common/var-lib-docker/overlay2/8943641d04172c565f42e75963910fdf6df436bc60d3929284c657b16d831dd1/diff:/var/snap/docker/common/var-lib-docker/overlay2/6c943a9276d504534ec9c66ade625447ad516058dce64053b899283aac500395/diff:/var/snap/docker/common/var-lib-docker/overlay2/416d42bd54e76f8a2c78da2799ad6e6c5d2c3d78b822ed8e17ada9bdc0fc17a7/diff:/var/snap/docker/common/var-lib-docker/overlay2/89ab8fa214a8141179afa2c21071e565dc841d1e0f5a69dc372d5b6514456cb0/diff:/var/snap/docker/common/var-lib-docker/overlay2/d99204b78e6ddf1279d8f1b099318f921a61e9017d457955e5a53c5c2d17be8a/diff:/var/snap/docker/common/var-lib-docker/overlay2/259c1d3c5a275a0c51fe499c1a8a2857427cf592eb094dea65447b591ec66058/diff:/var/snap/docker/common/var-lib-docker/overlay2/721958698d3add899bd2597d3dab22267c3aaa40dc556eb7f0cdb8c564b14a74/diff:/var/snap/docker/common/var-lib-docker/overlay2/c122e772ebb24d66d663577b099800060d10e1d1fda0b38e75ea73bc2fdf6273/diff:/var/snap/docker/common/var-lib-docker/overlay2/b1a9df93abc1fbba6b8f3644eaf5ec9a76b8b2193ad1d934241242fb54b74cd0/diff",
                "MergedDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199/merged",
                "UpperDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199/diff",
                "WorkDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/lukas/data/config/prometheus/prometheus.yml",
                "Destination": "/etc/prometheus/prometheus.yml",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "prometheus_prometheus_data",
                "Source": "/var/snap/docker/common/var-lib-docker/volumes/prometheus_prometheus_data/_data",
                "Destination": "/prometheus",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "ded87ebe4daf",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "9090/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--config.file=/etc/prometheus/prometheus.yml"
            ],
            "Image": "prom/prometheus:latest",
            "Volumes": {
                "/prometheus": {}
            },
            "WorkingDir": "/prometheus",
            "Entrypoint": [
                "/bin/prometheus"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "f387601dc597da679d6a2290f4192863d07c16791b9d33e29a4e84eb43b2e5df",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:2a72b385beafd2b3685ad337d14d23f6851f0ff335c5cd9a04c692b3eb846196",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "prometheus",
                "com.docker.compose.project.config_files": "/home/lukas/BlackholePort/docker/prometheus/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/lukas/BlackholePort/docker/prometheus",
                "com.docker.compose.service": "prometheus",
                "com.docker.compose.version": "2.17.2",
                "maintainer": "The Prometheus Authors <prometheus-developers@googlegroups.com>"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "b882dba1196b5ba11a81cedda028552f453ed82e42cec9fc8fc767d4192571f2",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "9090/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "9090"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "9090"
                    }
                ]
            },
            "SandboxKey": "/run/snap.docker/netns/b882dba1196b",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "prometheus_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "prometheus",
                        "prometheus",
                        "ded87ebe4daf"
                    ],
                    "NetworkID": "971e8873d46285b63d70fb6feeada747176a5427aa7a19127cb7e8dc8a7d8a37",
                    "EndpointID": "b67826c8e2736e5ac55ab67a8f9d67a6822dd3260a214622f65de968b0fa972e",
                    "Gateway": "192.168.32.1",
                    "IPAddress": "192.168.32.2",
                    "IPPrefixLen": 20,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:c0:a8:20:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

docker network ls

docker network ls
NETWORK ID     NAME                    DRIVER    SCOPE
f124d5db3e0b   bridge                  bridge    local
862535ef9371   deluge_default          bridge    local
86bf6736570f   foundryvtt_default      bridge    local
909b8162cdbe   grafana_default         bridge    local
299ce7db89b5   heimdall_default        bridge    local
92a81645a176   host                    host      local
a5e580aa1810   microbin_default        bridge    local
da8d9cb1d3e7   nginx-pm_default        bridge    local
055308848eba   none                    null      local
93afb48bbba2   o-portainer             bridge    local
460a87dcb670   o-security              bridge    local
ea16af5d1414   pihole_default          bridge    local
250440a093d7   portainer_default       bridge    local
971e8873d462   prometheus_default      bridge    local
dcbfb120b9b2   prometheus_monitoring   bridge    local
a482e951879d   prowlarr_default        bridge    local
9ab6f6734815   radarr_default          bridge    local
f6abd6de38d3   sonarr_default          bridge    local
0e062c280f80   spigotmc_default        bridge    local
640505be1145   teamspeak_default       bridge    local

after adding portainer to host network same issue

docker-compose.yml

version: '3.8'
volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - /home/lukas/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'


extra_hosts:
  - &quot;host.docker.internal:host-gateway&quot;
network_mode: &quot;host&quot;

ports:
  - &quot;9090:9090/tcp

docker inspect prometheus (network part)

 "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "f114aceaadadca19fe9da85b04dbce4fc9aedae6d7f91a2bed8b3290d3186e82",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/run/snap.docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "92a81645a17671cf440926ed83cdb6063790dcf2e18813722d96ccc90614ffb3",
                    "EndpointID": "4a4f2cef2556314c2feb526928b24784005a4617583e10813bae06f56d5db8e2",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
Hagbart Celine
  • 153

1 Answers1

4

Docker Compose is creating a network called "prometheus_default" and it's attaching the Prometheus container to that network. You have Docker daemon set to advertise metrics on the docker0 interface which is generally on the bridge default network Docker creates.

On the host where Docker is running, you can verify Docker daemon (-p, to show proccesses) is listening (-l) on the tcp (-t) metrics port number 9323 (-n) via ss or netstat

$ ss -tlpn | grep :9323
LISTEN 0      4096            127.0.0.1:9323       0.0.0.0:*    users:(("dockerd",pid=199176,fd=17))

Option 1

Try adding this config to docker-compose.yaml to add Prometheus to the bridge network

version: '3.8'
volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - /home/lukas/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'


extra_hosts:
  - &quot;host.docker.internal:host-gateway&quot;
network_mode: bridge

ports:
  - &quot;9090:9090/tcp

while making sure Docker is set to advertise metrics on the default bridge

{
   "metrics-addr": "172.17.0.1:9323"
}

Option 2

Change Docker daemon to listen on prometheus_default network

docker network inspect prometheus_default
[
    {
        "Name": "prometheus_default",
        "Id": "55a40edb42b2f3c1144236a765339e9b1c8dd0542df929430dc07829c5e156d5",
        "Created": "2023-03-31T16:51:48.212544059-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "prometheus",
            "com.docker.compose.version": "1.29.2"
        }
    }
]

Set in daemon.json

{
   "metrics-addr": "172.19.0.1:9323"
}

Option 3

Allow any network to access Docker metrics (keep in mind much more services will have access to the Docker daemon metrics interface which may or may not be desirable)

{
  "metrics-addr": "0.0.0.0:9323"
}
nijave
  • 436