In this example, we see that two lines have been printed to the logs, and then, as expected, the process stopped at the defined breakpoint we've set in our local IDE. If we then step through the breakpoint, and access the logs again, we'll see the following output:

Remotely debugging a service while it is running in Kubernetes is a complex, time-intensive process, but it is often the only way to fix bugs that can't be reproduced locally, outside of the realistic setup.

Below, we'll walk through the steps required to debug a service written in Go in the JetBrains Goland IDE while that service is deployed in a Kubernetes cluster. Then, we’ll demonstrate the increased ease and efficiency that Velocity brings to this same process.

TLDR; Velocity's free IDE plugins for JetBrains and VSCode make this process simple. Read on for more information, or just download the Velocity plugin for your preferred IDE and get started.

To illustrate what's involved in this process, we have a simple Go app, a related Dockerfile, and a Kubernetes deployment.

What is remote debugging in Kubernetes?

Remote debugging in Kubernetes is the process of debugging an application running within a Kubernetes cluster from a development environment separate from the cluster itself — i.e. your local IDE. It's a technique that allows developers to inspect, diagnose, and troubleshoot application issues without trying to replicate the environment on their local machine or without trying to replicate the whole environment on your local machine.

Update the Dockerfile

In order to remotely debug without Velocity, we’ll need to make some changes to our existing Dockerfile. The original Dockerfile for this service is defined as follows — it is a straightforward, multi-stage build in which we build the app binary in a build stage, and we then copy that compiled binary into a deploy stage for deploying to Kubernetes (read more on multi-stage builds here).

Dockerfile

FROM golang:1.21 AS build
WORKDIR /
COPY . .
RUN CGO_ENABLED=0 go build -gcflags "all=-N -l" -ldflags "-s -w" -o ./app

FROM alpine AS deploy
WORKDIR /
COPY --from=build /app app
ENTRYPOINT ["./app"]

In order to debug this service while it is deployed in Kubernetes — without Velocity — we'll first have to update the Dockerfile as follows:

FROM golang:1.18 AS build
WORKDIR /
COPY . .
RUN CGO_ENABLED=0 go install github.com/go-delve/delve/cmd/dlv@latest
RUN CGO_ENABLED=0 go build -gcflags "all=-N -l" -ldflags "-s -w" -o ./app

FROM alpine AS deploy
WORKDIR /
COPY --from=build /go/bin/dlv dlv
COPY --from=build /app app
ENTRYPOINT [ "/dlv" , "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/app"]

Above, we've added Delve, a popular debugger for Golang, to our image, and we've also updated our ENTRYPOINT such that instead of calling the executable app directly, we start it with Delve, which requires some additional arguments, such as an api version, and a port to listen on.

Build the Docker image

Now that the Dockerfile has been updated to include our debugger, we need to build and push the image, like so:

docker build -t example/my-cool-go-app:latest . 
docker login
...
docker push example/my-cool-go-app:latest

Manage port forwarding in Kubernetes

With the Kubernetes Deployment running, we then need to get the name of the Pod in which our updated container is running, and we then need to run kubectl port-forward with the same port we defined for Delve above, like so:

$ kubectl get pods  -n default  
NAME                            READY   STATUS    RESTARTS   AGE
server-debug-5f59f4d5dd-zpgzn   1/1     Running   0          64s

$ kubectl port-forward pod/server-debug-5f59f4d5dd-zpgzn 40000:40000
Forwarding from 127.0.0.1:40000 -> 40000
Forwarding from [::1]:40000 -> 40000
Handling connection for 40000

Create a remote run configuration in Golang

Now, with the debug-ready service running in Kubernetes and our port-forwarding configured, we'll need to create a new run configuration in GoLand that is specific to debugging in a remote environment. To do so, we can select “Edit Run Configurations” at the top center of the IDE, and then click the “+” sign, and select “Go Remote,” as shown below.

Then, we'll need to set the Host and Port values in the new run configuration:

Begin debugging

With all of the above in place and working, we can now click the “debug” icon in the IDE, and the code should stop at our defined breakpoints, as shown below:

We can then see output from the running service by accessing its logs via kubectl:

kubectl logs pod/server-debug-5f59f4d5dd-zpgzn 
API server listening at: [::]:40000
2023-05-16T17:57:40Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
This is a loop
More loop

In this example, we see that two lines have been printed to the logs, and then, as expected, the process stopped at the defined breakpoint we've set in our local IDE. If we then step through the breakpoint, and access the logs again, we'll see the following output:

kubectl logs pod/server-debug-5f59f4d5dd-zpgzn 
API server listening at: [::]:40000
2023-05-16T17:57:40Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
This is a loop
More loop
This is a loop
More loop

Rinse and Repeat

Update the code, rebuild the image, redeploy to Kubernetes

Once we locate the code that seems like it may be causing the unexpected behavior in the service, we'll need to update the source code and test our fix. And for each change we make to code that is running in Kubernetes, we'll have to repeat most of above steps.

After changing the source code, we'll need to rebuild the Docker image, and deploy the new container to our cluster. With an app as simple as this example, that will take a matter of minutes, but — obviously — in a real service deployed in your Development or Staging cluster, you will probably need to go through a full CI process before you can build and redeploy the new image, which will likely take significantly longer.

And, of course, you'll need to go through this same process for every code change you make as you debug, which means tons of idle time during your development process. This is simply an inefficient way to work.

Enter Velocity

With Velocity's IDE plugin, you can achieve this same functionality and more with a few button clicks directly in your IDE.

Specifically, Velocity makes all the necessary adjustments to your Dockerfile, including the addition of the debugger and the adjustment of the compilation to allow debugging, so you don't have to. And it also configures your IDE and your remote environment to further streamline the remote debugging process.

As a result, with a few button clicks, Velocity watches your local environment for code changes and automatically redeploys to your cluster every time you save your work, you won't need to go through any of the manual build steps outlined above.

Moreover, you won't have to wait for a full CI process to debug a line of code. Instead, you can just start Velocity, connect to your cluster in the IDE, and make changes to and debug remote code exactly as you would if that code was running locally — no time-wasting build steps, no waiting for CI processes. Just developing as you're accustomed to. Only commit and push your code when you’re ready – not as part of the development feedback loop. And Velocity makes the build process super fast with cache optimizations, while the build runs in your cluster.

In concrete terms, this means our demo process, which without Velocity takes ~5 minutes for every iteration in our debugging process, now takes ~30 seconds.

‍