Debug .NetCore Containers in Docker and Kubernetes

Many find it confusing, how to debug application in docker containers. In fact, it is reasonably straight forward, it just isn’t documented properly, especially in .Net land. Let’s go over what’s needed:

• .Net Core application built in debug mode inside the container
• VSDBG installed in the docker container
• OpenSSH and correct port/login setup in the container

For your local machine you have many options:

1. Visual Studio 2017, using in-built SSH functionality
2. Visual Studio Code, using MIEngine and Putty
3. Visual Studio 2017, using MIEngine and Putty

VladimirAkopyan/DockerDotnetDebug

vsdbg vs. clrdbg

Both are debugging tools for .Net Core applications.
As part of the new architecture, clrdbg has been replaced with a new executable named vsdbg. When you see references to clrdbg, just know it’s an out-of-date document.

Configuring Base Container

Our container will run both the OpenSSH server and our application simultaneously. This violates the usual mantra of “one container, one process”, but that is just the cost of doing business. We are setting up the application to run continuously — the debugger does not start the application or kill it when it detaches. I find that such setup is better suited to debugging real-world problems, where your application might run for hours before hitting some invalid state / issue.

We will assume that this particular container we are setting up is just used for testing, and with OpenSSH we prioritise convenience over security.
To configure it, create a file calledsshd_config with contents as follows:

# This is ssh server system wide configuration file.
#
# /etc/sshd_config 
Port                       2222
ListenAddress              0.0.0.0
LoginGraceTime             180
X11Forwarding              yes
Ciphers aes128-cbc,3des-cbc,aes256-cbc
MACs hmac-sha1,hmac-sha1-96
StrictModes                yes
SyslogFacility             DAEMON
PasswordAuthentication     yes
PermitEmptyPasswords       no
PermitRootLogin            yes
# Nessesary for VS2017, but not for VSCode!
Subsystem sftp internal-sftp

Although the rest of this example uses Password Authentication, we will we will additionally create an SSH key that we can use. If you don’t need an SSH key, skip this step and delete the corresponding line form the dockerfile.

Create file authorized_keys and place the ssh key into it. In my case, the file looks like this, the entire key must be in a single line.

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAkeoEOUi0HcXu6QZ0X1Qu4Msq5kVYXRjmNFXSUUbG3hYtgiCT9cq0k45phlKbrafFjEuo7BiaboHRz6JLdkYCBQT9aviTHtElpqmNUtQsaDvVGBBrNZ8vUQT0PXq/lYAi6onUxZLNbykYlBBuIgNVQhWqnmsHNXaFxKsG0BJADQeEzYHjJuEuhAtPgiWSP5ViR5hrZk0qdZMLiS8PHcw0XCQMcM2qU0v10PzSIVUUyAwRRgGWWfXmy2P0XcwAg95LVfosCaeFsaW9rb35/kYPRNlf6QxP/9BOsjwhxVWAV7eYWYZlkwqAsQY1g2qhZ5Nzy1N+JwaPhmirE5xO4zdVwQ==

Now we will create a container that will be used as a base container for our future exploits:

FROM microsoft/aspnetcore:2.0 AS base 
# Install the SSHD server
RUN apt-get update \ 
 && apt-get install -y --no-install-recommends openssh-server \ 
 && mkdir -p /run/sshd
# Set password to 'Docker!'. Change as needed.  
RUN echo "root:Docker!" | chpasswd
#Copy settings file. See elsewhere to find them. 
COPY sshd_config /etc/ssh/sshd_config
COPY authorized_keys  root/.ssh/authorized_keys
# Install Visual Studio Remote Debugger
RUN apt-get install zip unzip
RUN curl -sSL https://aka.ms/getvsdbgsh | bash /dev/stdin -v latest -l ~/vsdbg  
EXPOSE 2222

Configured base container is available at dockerhub as clumsypilot/dotnetdebug: asp-runtime-2.0

Configure Debugable application container

We will assume that you have a debugable application. I have just created an ASP.net core application from a template and left it as-is. I have called it DebugSample and it produces a dotnet executable DebugSample.dll

The first thing we need is a script that will start both the application and OpenSSH like the example give on dockerDocumentation. StartSSHAndApp.sh

#!/bin/bash 
# Start the first process
dotnet /app/DebugSample.dll </dev/null &>/dev/null &s
tatus=$?
if [ $status -ne 0 ]; then
  echo "Failed to start ASP Server: $status"
  exit $status
fi 
/usr/sbin/sshd -D
if [ $status -ne 0 ]; then
  echo "Failed to start Ssh Server: $status"
  exit $status
fi
# Naive check runs to see if either of the processes exited.# This illustrates part of the heavy lifting you need to do if you want to run# more than one service in a container. The container exits with an error# if it detects that either of the processes has exited.# Otherwise it loops forever, waking up every 5 seconds 
while sleep 5; do
  ps aux |grep dotnet |grep -q -v grep
  PROCESS_1_STATUS=$?
  ps aux |grep sshd |grep -q -v grep
  PROCESS_2_STATUS=$?
  # If the greps above find anything, they exit with 0 status
  # If they are not both 0, then something is wrong
  if [ $PROCESS_1_STATUS -ne 0 -o $PROCESS_2_STATUS -ne 0 ]; then
    echo "One of the processes has already exited."
    exit 1 
 fi
done

Following this, create a container that will run this script. You can substitute base container for the we’ve created in previous step.

FROM clumsypilot/dotnetdebug:asp-runtime-2.0 AS base
WORKDIR /app 
MAINTAINER Vladimir Vladimir@akopyan.me
FROM microsoft/aspnetcore-build:2.0 AS build
WORKDIR /src 
COPY ./DebugSample .
RUN dotnet restore  
FROM build AS publish
RUN dotnet publish -c Debug -o /app 
FROM base AS final
COPY --from=publish /app /app
COPY ./StartSSHAndApp.sh /app 
EXPOSE 5000 
CMD /app/StartSSHAndApp.sh
#If you wish to only have SSH running and start #your service when you start debugging#then use just the SSH server, you don't need the script
#CMD ["/usr/sbin/sshd", "-D"]

A container built in this manner is avaliable at dockerhub as clumsypilot/dotnetdebug:asp-debug-sample

Basic Commands and References

Changing root password for SSH

If you want to use the base container, but you want it to be secure, change the root password

echo "root:Docker!" | chpasswd

Key File location

root/.ssh/authorized_keys

If want to secure the base container with your own Key, or use password, replace or delete it. If the key is updated, it is immediately applied — no need to restart OpenSSH.

Inspecting the container

You might want to inspect the container. You can attach to the running container as a separate Bash process

docker exec -it <id of running container> bash

VSDBG doesn’t start

Error from pipe program 'plink.exe': bash: /root/vsdbg/vsdbg: cannot execute binary file: Exec format error

To check executable architecture:

file /root/vsdbg/vsdbg

file is not usually installed in docker containers by default, so you might have to use apt-get.

Starting container in Docker for connect-ability

docker run -d -p 2222:2222 -p 5000:5000 clumsypilot/dotnetdebug:asp-debug-sample

Port 2222 is used for debugging, the port 5000 is for the web-server in asp-debug-sample. You might use a different one.

Connecting to a pod in a Kubernetes cluster

kubectl port-forward <POD-NAME> 2222

Now you can use LocalHost as destination address in your connectivity settings!

Detached Start of a program in Linux

dotnet program.dll </dev/null &>/dev/null &

Starts the program in a detached manner and redirects all Stdin/Stdout to null

Script Doesn’t work

Make sure you have linux line endings — if you create the script file in windows, you will have windwos line endings and bash won’t run the script and start your application and OpenSSH.

VSCODE gets Stuck

If you get password or username wrong, VSCODE gets stuck and hangs on:

Starting: “plink.exe” -l root -pw Docker! localhost -P 2222 “~/vsdbg/vsdbg — interpreter=vscode”

Visual Studio 2017 Debugging

Debugging in VS2017 is the most straight forward way of remote debugging, but also has an annoying issue that’s not documented anywhere (more on that later)

1. Clone my git repo and open the DotnetDebug solution.
2. Start the docker container using command described above or deploy in Kubernetes and port-forward.
3. Go to ‘Debug->Attach to Process’.
4. Choose ‘SSH’ Connection type.
5. Enter localhost in the Connection Target field and hit Enter
6. An extra dialog will pop up where you can enter port, password, etc. If you are struggling with this step, you can setup connection in ‘Tools->Options’ using ConnectionManager.
7. Once the connection is established you will be presented with a list of processes running on the target machine / container. Choose dotnet and in the next dialog choose managed debugger.
8. Set a breakpoint in Pages->Index.cshtml.cs
9. Open Localhost:5000 and you should see the breakpoint hit

If you have a Connectivity Problem

I struggled to get Visual Studio to work because of a connectivity issue.

Connectivity Failure. Please make sure host name and port number are correct.

Thanks for this Github thread I finally realised that my sshd_config lacked

Subsystem sftp internal-sftp

This entry is not required to MiEngine based debugging, described below.

MiEngine based setup

Main Suspect

MiEngine is a more flexible debugging system that can use almost any connection between itself and the target application, and SSH is just one of the options. It requires another tool to establish ‘connection pipe’ and a config file.

We will use putty. Make sure that you have putty installed and environmental variables setup. Check this by opening console and typing in ‘plink’, the response should not be plink is not recognized as an internal or external command

Secondly, before you go any further, make sure you have at least once connected to the container and accepted the host key. If you do not do that, the VS2017 demo will fail silently with no error message whatsoever and you will be sat there scratching your head.

Visual Studio 2017 with MiEngine

Let’s look at the MIEngine script, available in git repo at
DebugSample->MIEngineConfig->MIEngineAttach.json

{
  "version": "0.2.0",
  "adapter": "plink.exe",
  "adapterArgs": "-l root -pw Docker! localhost -P 2222 -batch -T ~/vsdbg/vsdbg --interpreter=vscode",
  "languageMappings": {
    "C#": {
      "languageId": "3F5162F8-07C6-11D3-9053-00C04FA302A1",
      "extensions": [ "*" ]
    }
  },
  "exceptionCategoryMappings": {
    "CLR": "449EC4CC-30D2-4032-9256-EE18EB41B62B",
    "MDA": "6ECE07A9-0EDE-45C4-8296-818D8FC401D4"
  },
  "configurations": [
    {
      "name": ".NET Core Attach",
      "type": "coreclr",
      "request": "attach",
   "processName": "dotnet"
    }
  ]
}

MiEngine comes as part of VS2017. You can access it through commands in

1. View->Other Windows->Command Window
2. Type into the command window DebugAdapterHost.Launch /LaunchJson:”C:\Dev\DockerDotnetDebug\DebugSample\MIEngineConfig\MIEngineAttach.json”

You should be in a debug mode identical to the one used previously. The main difference is that

1. You are using putty to connect to the target system. It’s possible to use other programs to act as pipe for MiEngine, like the local docker agent
2. If putty does not accept the hostkey, your command will fail silently, reporting no error. It will simply do nothing. I opened an issue.
3. You don’t need the line Subsystem sftp internal-sftp in sshd_config for this to work.

Visual Studio Code

The procedure for using Visual Studio Code is similar. We need the following:

1. VSCode installed with OmniSharp plugin
2. Open folder DebugSample, if you open the upper folder, file-paths will be different.
3. Visual Studio Code uses the folder .vscode and files tasks.json and launch.json
4. We are concerned with launch.json — we will create an extra configuration in addition to the ones that already exist there, for remote debugging of the container.

We must create a configuration file that instructs VSCode to use putty and connect to OpenSSH server.

Note that configurations for Linux and MacOS here are just placeholders — on those platforms you need to use appropriate SSH client and commandline arguments.

Once you’ve established a connection you will be presented with a process picker, you want to select dotnet process to debug, as that’s your application.

The VSCode debug experience is actually pretty good.

References:

• Debugging .Net Core over SSH
• MiEngine Repository
• Visual Studio Code Launch.Json documentation
• NodeJS is way easier to debug in kubernetes
• Multirun for managing multiple processes inside containers
• OmniSharp VsCode Attaching to remote process
• Github Discussion, “what is the proper way to debug an application inside docker containers?”
• Github Bug report — Can’t establish SSH connection from VS2017