Docker Compose and the danger of copying existing configurations…

March 9, 2024

So, I have this docker server running a bunch of services through Traefik. This is working well for me as the services are picked up by traefik based on labels in the docker-compose for each app (instead of setting things up on the traefik side explicitly). Setting up a new app is painless with respect to setting up ingress as traefik handles everything providing your labels are correct.

I recently setup a couple of new apps which came up and worked perfectly the first try. Which, you know is good… except an existing service of mine broke. Specifically it immediately started returning “bad gateway” as soon as the other 2 services came up.

On looking at the dashboard for traefik I noticed 3 servers (separate IPs and ports) for the one service that was breaking… So that explains why it’s breaking (traefik can’t figure out which IP and port to direct traffic to for this as there isn’t a declared loadbalancing setup in the traefik config) , but when I looked at the docker-compose.yml for the broken service all is well. It’s got one declaration for a server. So what’s going?

I took the broken service down and after that discovered that my broken service wasn’t gone from my traefik dashboard… ok that’s weird… but now there are only 2 servers. Looking around I became suspicious that I’d somehow included this service declaration in a separate docker-compose for one of my new apps since the docker IP and port were identical to one of my new apps. After thinking about it I realized I’d copied and pasted the traefik labels for the new configurations from my existing (breaking) service. hmmm…

On closer examination of the docker-compose files, I’d done this in my traefik labels.

    labels:
      - traefik.enable=true
      - traefik.http.routers.new-app1.rule=Host(`new-app1.mydomain.com`)
      - traefik.http.routers.new-app1.service=new-app1
      - traefik.http.routers.new-app1.entrypoints=sslproxy
      - traefik.http.services.new-app1.loadbalancer.server.port=9180
      - traefik.http.routers.new-app1.tls=true
      - traefik.http.routers.new-app1.tls.certresolver=myresolver
      - traefik.http.services.app-I-copied-from.loadbalancer.passhostheader=true

Note… the stray loadbalancer declaration forces a server into existence in the name of app-I-copied-from giving it the same IP and port of new-app1, which will at least help you find the docker-compose.yml this happened in. This is aggregated with the existing app-I-copied-from configuration confusing traefik by giving it more than one IP to push traffic to for this server.

correction to ensure all labels declaring the service name correctly as new-app1 in the new-app1 docker-compose file brings everything back thankfully.

On to the next stupid thing… sigh.