Kristóf's blog

13 May 2023

Go adventures: Live reload and debugging in a Docker container

  9 minutes

Welcome to part 1 of this series about exploring Go. Today we are going to set up a comfy development environment in Docker containers. Let’s dive in!

Goals

Let’s say we are trying to develop a Go-based backend service that will run on Kubernetes. In practice, this usually means that the service is going to run in a Docker container (though there are alternatives, such as Podman). It would be a logical idea to also want to run the service in a container during development, to keep it as close to the production environment as possible.

This latter idea brings up some problems, though.

Firstly, if we want our code changes to show up, we need to either

  1. rebuild the container (if the source files or the executable is copied to the container)
  2. or restart the container (if the source code is mounted via volumes and the executable is built in the Dockerfile)

Both of those can be circumvented by the use of a live-reload solution.

Secondly, during development we often resort to the debugger to find elusive problems. By running the service in a container, it’s harder to attach a debugger, since we need to reach the service inside somehow.

So here we have our goals:

  • Run a service in a docker container so we don’t need different setups for development and production environments
  • Add live-reload support so we don’t need to constantly restart the container
  • Add debugger support

Let’s tackle the problem step by step!

A basic containerized service

For the purposes of the article, we will just assume the service in question is an HTTP server listening on port 8080. We are also using docker-compose to run the development setup so we don’t have to deal with docker commands directly.

All code can be found in repository kristofaranyos/blog-content. The folder structure of our imaginary web service is going to be simple — and yes, it doesn’t follow the usual Go folder structure, but we only care about the environment here:

1
2
3
4
5
6
7

.
├── docker-compose.yml
├── Dockerfile
├── go.mod
├── go.sum
└── src
    └── main.go

Here is what our main.go file looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte("Hello world!"))
	})

	log.Println("starting server...")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(fmt.Errorf("could not start server: %w", err))
	}
}

The Dockerfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

FROM golang:1.20

WORKDIR /service

COPY go.mod ./
COPY go.sum ./
COPY src ./src

EXPOSE 8080

ENTRYPOINT ["go", "run", "./src/main.go"]

And lastly, the docker-compose.yml:

1
2
3
4
5
6
7
8
9

version: "3.9"

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"

You can see the state of the code here. Now let’s run the service to verify it’s indeed working, by first running $ docker compose build and then $ docker compose up.

Tip: docker compose and docker-compose are two different commands. Compose is integrated into the Docker CLI from V2 onwards.

If everything went well, you should be greeted with the message Hello world! after visiting localhost:8080.

Adding live reload

Now that our basic containerized service is running, we can start tuning the setup. Let’s add live reload! Thanks to the flourishing Go ecosystem, we have a variety of choices to do this. One option would be cosmtrek/air. However, I find it a little convoluted so we’re going to use githubnemo/CompileDaemon. It’s wastly simpler, and it’s much lighter, while it’s still enough for basic development.

We begin by adding CompileDaemon to the Dockerfile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

FROM golang:1.20

WORKDIR /service

COPY go.mod ./
COPY go.sum ./
COPY src ./src

RUN go install github.com/githubnemo/CompileDaemon@latest

EXPOSE 8080

ENTRYPOINT ["/bin/bash", "-c", "CompileDaemon \
    -log-prefix=false \
    -graceful-kill=true \
    -directory=\"/service\" \
    -build=\"go build -o /bin/go_service ./src/main.go\" \
    -command=\"/bin/go_service\" \
"]

You can notice we changed two things. First, we download the latest version of CompileDaemon, and after that, instead of directly building the Go executable, we tell CompileDaemon what to do. There are a few things happening here:

  • With the directory flag, we tell CD where to look out for changes.
  • With the build flag, we tell CD what to do once any .go files change.
  • With the command flag, we tell CD what to do once it has rebuilt the executable.

With these additions, our container is now capable of updating the executable it’s running, if the source code changes. The catch here is that it won’t ever change, since we copy the source code upon building the Dockerfile. To overcome this limitation, we can utilize Docker volumes to keep the contents of the container up to date with the host computer. Let’s add a few lines to the docker-compose.yml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

version: "3.9"

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    volumes:
      - ./go.mod:/service/go.mod
      - ./go.sum:/service/go.sum
      - ./src/:/service/src

This adds three volumes — technically we could get away with one using wildcards, but it’s a good practice not to rely on those — that will keep our code up to date.

Now we can rebuild and run our setup using docker compose the usual way.

Tip: Don’t forget to $ docker compose down before running $ docker compose up to make sure your setup is completely fresh.

Assuming we did everything correctly, we can now go on and change something in main.go and have the changes take effect. Let’s add a simple parameter to our endpoint:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
		writer.Write([]byte(fmt.Sprintf("Hello world! I'm %s.", request.URL.Query().Get("name"))))
	})

	log.Println("starting server...")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(fmt.Errorf("could not start server: %w", err))
	}
}

After changing the file, we should see something like this in the terminal:

1
2
3
4
5
6
7
8
9

web-1  | Running build command!
web-1  | Build ok.
web-1  | Restarting the given command.
web-1  | 2023/05/13 16:47:51 starting server...
web-1  | Running build command!
web-1  | Build ok.
web-1  | Gracefully stopping the current process..
web-1  | Restarting the given command.
web-1  | 2023/05/13 16:48:12 starting server...

Connecting to localhost:8080?name=Anonymous in the browser, we should be greeted with Hello world! I'm Anonymous. You can find the code for the current state here.

Time to debug

We reached the last goal — adding the debugger. For this, we are going to use Delve, the de-facto standard Go debugger.

There are two issues that we need to deal with to get the setup working.

  1. Delve has to interact with CompileDaemon. But which comes first and why does this matter?
  2. Containers insulate the processes within from the outside world. How do we let Delve communicate with the client on the host machine?

Starting with the easier issue — Delve and CompileDaemon cooperating — we need to think about what cases there are. We could either have Delve run CD or vice versa. If we think about it, the former solution won’t work since CD runs commands as external processes, and because of this, Delve won’t be able to debug the service. The latter way, we also don’t waste time running CD with the debugger. We continue by adding Delve to the Dockerfile.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

FROM golang:1.20

WORKDIR /service

COPY go.mod ./
COPY go.sum ./
COPY src ./src

RUN go install github.com/githubnemo/CompileDaemon@latest
RUN go install github.com/go-delve/delve/cmd/dlv@latest

EXPOSE 8080
EXPOSE 9090

ENTRYPOINT ["/bin/bash", "-c", "CompileDaemon \
    -log-prefix=false \
    -graceful-kill=true \
    -directory=\"/service\" \
    -build=\"go build -o /bin/go_service ./src/main.go\" \
    -command=\"dlv exec --headless=true --listen=:9090 --api-version=2 --accept-multiclient --continue /bin/go_service\" \
"]

There are again a few things happening here. We added a RUN command to download the Delve executable. We also exposed port 9090 which we will use to connect the debugger through. And lastly, we changed the command flag of CompileDaemon. Let’s dissect the flags:

  1. -headless=true tells Delve we are not going to use an interactive shell, but connect remotely
  2. --listen=:9090 tells Delve which port to listen on — this should be the same one we are exposing
  3. --api-version=2 tells Delve to use its V2 (currently latest and the only maintained) API. It supports both JSON-RPC and DAP to communicate with clients.
  4. --accept-multiclient tells Delve to expect multiple clients connecting. This lets you reconnect after disconnecting
  5. --continue together tell Delve to start running the process even if we don’t connect to it. By default, Delve waits until you connect the debugger. This is required so the service will be started even if you don’t want to debug it

The second issue we talked about is a little more arcane. We begin with taking a look at the docker-compose.yml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

version: "3.9"

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
      - "9090:9090"
    security_opt:
      - "seccomp:unconfined"
    cap_add:
      - SYS_PTRACE
    volumes:
      - ./go.mod:/service/go.mod
      - ./go.sum:/service/go.sum
      - ./src/:/service/src

The first and obvious change is the added port-forward. But what are these changes?

1
2
3
4

    security_opt:
      - "seccomp:unconfined"
    cap_add:
      - SYS_PTRACE

Essentially, what is happening during the debugging is that Delve traces the service process using ptrace(2). The cap_add directive gives the container permission to do this using capabilities(7). This is a pretty obscure Linux feature, but for our purposes it’s enough to know that we just need to include this directive for the debugging to work.

The other thing is the security_opt directive that tells Docker to run the container with — in this case — no seccomp(2) profile. Secure computing is another little-known Linux kernel feature that we won’t be diving into. It just works. ( ͡° ͜ʖ ͡°)

With all this out of the way, we can just again rebuild and restart out compose setup, and now debugging should work. You can check the final state of the code here.

To test it, we are going to be using Goland because obviously it’s the best IDE for Go development. Add a run configuration called Go remote and set the port to 9090. After running it and adding a breakpoint in our code (click the empty space to the right of the line number), our setup should be live.

Tip: I recommend setting the run configuration to leave Delve running after disconnecting so we don’t need to restart the service if we accidentally disconnect the debugger.

Example breakpoint

We can query the endpoint in our browser as usual to make the handler run. Our the system should stop at the breakpoint and we should see the variables in the scope.

Example breakpoint

Tip: If you have multiple services running locally, you can enable all of the debuggers with one click using the Compound run configuration.

With that, we’ve reached all three goals we set for ourselves. Happy coding!

Afterthoughts

I hope this was an informative article and I could help to make your DX be more comfortable. Don’t hesitate to reach out to me with any questions or thoughts!

Tags

container debugging docker docker-compose go go-adventures live-reload