Windows containers with GitLab CI

Now that preview builds of Windows Server 2025 are available, I finally have the excuse I’ve been looking for to experiment with Windows OS containers and the native Docker Engine port. Neither are new, I’ve just been putting this off for a really long time.

I maintain a Windows server for the few things I need that can “only” be done on Windows. It’s set up with GitLab Runner, which has used the shell executor ever since it was installed back in 2017. Essentially, this means that the CI script, which is defined on a per-project basis in the .gitlab-ci.yml file, runs all of its commands directly on the host without any kind of isolation.

There are some really nasty security implications with this, but since I’m the only one with access to my GitLab instance, and my CI jobs – which should never touch anything outside of the designated build directory – mostly consist of pulling in external dependencies, compiling the project, and uploading artifacts, why would I want to use containers?

There are a few reasons, but if I had to pick only one, it’s that I really don’t want to maintain all the software on the host anymore. I’ve had to install a bunch of stuff from various sources to get all these different projects building over the years. Some of these can easily be installed and updated with tools like WinGet or Chocolatey, but a lot of others cannot.

To make matters worse, some jobs can only run with older versions of software, while others rely on newer features, ultimately requiring multiple instances to be installed. As you can imagine, it’s a mess, but one that is easily solved with containers.

At the time of writing, the only copies of Windows Server 2025 available are timed evaluations, but I’ll be using it here regardless. It can be found officially from the Windows Insider Program for Windows Server, or more conveniently from uupdump.net.

I was surprised to run into my first issue immediately after starting the machine. Must be some kind of new record.

After creating a new virtual machine in VMware ESXi with the recommended default settings for Server 2025, the official install media would not boot any further than the “press any key to boot from DVD” prompt.

2024-03-10T15:15:44.997Z In(05) vcpu-0 - Guest: About to do EFI boot: EFI VMware Virtual SATA CDROM Drive (0.0)
2024-03-10T15:15:45.894Z In(05) vcpu-0 - pvnvram: Failed to retrieve timestamp database data.
2024-03-10T15:15:45.894Z In(05) vcpu-0 - pvnvram: Failed to retrieve timestamp database data.
2024-03-10T15:15:45.894Z In(05) vcpu-0 - SECUREBOOT: Image APPROVED.
2024-03-10T15:15:45.896Z In(05) vcpu-0 - Guest: About to do EFI boot: EFI VMware Virtual SATA CDROM Drive (0.0)
2024-03-10T15:15:46.864Z In(05) vcpu-0 - pvnvram: Failed to retrieve timestamp database data.
2024-03-10T15:15:46.864Z In(05) vcpu-0 - pvnvram: Failed to retrieve timestamp database data.
2024-03-10T15:15:46.864Z In(05) vcpu-0 - SECUREBOOT: Image APPROVED.
2024-03-10T15:15:46.867Z In(05) vcpu-0 - Guest: About to do EFI boot: EFI VMware Virtual SATA CDROM Drive (0.0)
2024-03-10T15:15:47.794Z In(05) vcpu-0 - pvnvram: Failed to retrieve timestamp database data.
2024-03-10T15:15:47.794Z In(05) vcpu-0 - pvnvram: Failed to retrieve timestamp database data.
2024-03-10T15:15:47.794Z In(05) vcpu-0 - SECUREBOOT: Image APPROVED.

Logs aren’t helping out much either. I guess I’ll try using BIOS firmware instead.

Oh, okay then... Let’s just start over. I’ll set it up as Server 2022 with ESXi 7.0 compatibility and upgrade it after it’s installed.

Hooray. Can’t say I’m a fan of this new setup wizard, but maybe it’ll grow on me. At least the old version is still available for now. Better yet, you can still drop into a command prompt with Shift+F10 and apply the install image manually with Dism.

Welcome to the not-so-desktop experience

Okay, what’s next… I’ll enable Remote Desktop, but I really don’t want to use it for times when I only need a shell.

Fortunately, there’s an official built-in OpenSSH server component, that is now installed by default.

(host) PS> Get-Service -Name sshd

Status   Name               DisplayName
------   ----               -----------
Stopped  sshd               OpenSSH SSH Server

(host) PS> Set-Service -Name sshd -StartupType 'Automatic'
(host) PS> Start-Service sshd
(host) PS> Get-NetFirewallRule -Name "OpenSSH-Server-In-TCP" | Select-Object Enabled

Enabled
-------
   True

I wasn’t able to connect to the server with any of the SSH clients I had installed, despite there being an enabled firewall rule to allow inbound traffic on TCP port 22. I tried disabling and re-enabling the rule but that didn’t work either.

(host) PS > Get-NetFirewallRule -Name "OpenSSH-Server-In-TCP" | Get-NetFirewallPortFilter

Protocol      : TCP
LocalPort     : 22
RemotePort    : Any
IcmpType      : Any
DynamicTarget : Any

I “fixed” that by removing and re-creating the rule. If you want to listen on a non-standard port, just edit the server configuration file with (in PowerShell) “notepad $env:ProgramData\ssh\sshd_config”, substitute 22 with something else, then restart.

(host) PS> Remove-NetFirewallRule -Name "OpenSSH-Server-In-TCP"
(host) PS> New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' `
  -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22

I don’t use password authentication, so I also created the “Program Data\ssh\administrators_authorized_keys” file and added my public keys to it. The server is configured to read from here by default, so there’s no need to restart the service.

SSH on Windows, cool! Now to enable the containers feature and install the Docker runtime. Microsoft provides a script that handles all the heavy lifting. The source documents a few interesting extra parameters, but I’m fine with the defaults for now.

(host) PS> Invoke-WebRequest -UseBasicParsing -o install-docker-ce.ps1 `
  "https://raw.githubusercontent.com/microsoft/Windows-Containers/Main/helpful_tools/Install-DockerCE/install-docker-ce.ps1"
(host) PS> .\install-docker-ce.ps1

If necessary, the script will create a scheduled task to perform the final configuration steps after a reboot. Starting a new SSH connection won’t trigger this task, but opening a Remote Desktop session will. Logging in on the physical machine works too.

With Docker installed and configured to start automatically, it’s time to build an image.

Most of my projects are built with the Visual C++ toolchain, so that’s a good starting point. I mostly followed this post about installing Visual Studio Build Tools in a container, substituting the components with some of my own choices.

Here’s a minimal example that I’ve saved to C:\GitLab\Images\cpp-msvc-2022-build\Dockerfile:

# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2022

SHELL ["cmd", "/S", "/C"]

RUN curl --location --output vs_buildtools.exe https://aka.ms/vs/17/release/vs_buildtools.exe `
 && (start /w vs_buildtools.exe --quiet --wait --norestart --nocache `
     --installPath "%ProgramFiles(x86)%\Microsoft Visual Studio\2022\BuildTools" `
     --add Microsoft.VisualStudio.Workload.VCTools `
     --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 `
     --add Microsoft.VisualStudio.Component.Windows11SDK.22621 `
     || IF "%ERRORLEVEL%"=="3010" EXIT 0)

(host) > docker build --tag aixxe.net/cpp-msvc-2022-build .

This part takes a while, and beyond the initial curl download output, there is no further indication of any progress being made while it runs. I had to glance over at my process count and CPU usage occasionally to see if it was still doing something.

Although the image is being built from inside temporary containers, you can still see the various setup related processes in the process list of Task Manager. The Microsoft documentation on isolation modes is a good read if you’re interested in the details.

(host) > docker image ls
REPOSITORY                             TAG        IMAGE ID       CREATED          SIZE
aixxe.net/cpp-msvc-2022-build          latest     7b2311f89a7b   1 minute ago     12.1GB
mcr.microsoft.com/windows/servercore   ltsc2022   1f57f3b65348   4 weeks ago      4.29GB

Finally, the image has finished building. Better do a quick test to see if it’s working before installing the rest of the software.

(host) > docker run --rm --name=test --interactive --tty aixxe.net/cpp-msvc-2022-build

Microsoft Windows [Version 10.0.26063.2322]
(c) Microsoft Corporation. All rights reserved.

(container) C:\>call "%ProgramFiles(x86)%\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.2
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

(container) C:\>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.39.33521 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

Here’s a simple C program. I’ll pass the hostname of the container as BUILD_HOST, which should match the container ID.

// hello.c
#include <stdio.h>

int main() {
    printf("Hello, world from build host %s!\n", BUILD_HOST);
}

I forgot to start the container with a bind mount or volume, so I’ll have to copy this in with docker cp. Note that this won’t affect the image – only the test container, as it is written to its temporary writeable layer, which will be deleted upon exit.

(host) > docker ps
CONTAINER ID   IMAGE                           COMMAND                    CREATED         STATUS         PORTS     NAMES
5c39ac74b461   aixxe.net/cpp-msvc-2022-build   "c:\\windows\\system32…"   1 minute ago    Up 1 minute              test

(host) > docker cp hello.c test:C:\hello.c
Successfully copied 2.05kB to test:C:\hello.c

Switching back to the container shell, the source file should now exist at C:\hello.c.

(container) C:\>cl /DBUILD_HOST=\"%COMPUTERNAME%\" hello.c
Microsoft (R) C/C++ Optimizing Compiler Version 19.39.33521 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

hello.c
Microsoft (R) Incremental Linker Version 14.39.33521.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:hello.exe
hello.obj

So far, so good. All that’s left is to pull out the compiled executable and see if it runs.

(host) > docker cp test:C:\hello.exe hello.exe
Successfully copied 144kB to C:\GitLab\Images\cpp-msvc-2022-build\hello.exe

Now for the not so fun part… adding in all the other build stuff to the image.

Or so I thought, but installing the rest of the software didn’t take nearly as long as installing Build Tools. The only time-consuming part was dealing with all the different installer types, and figuring out which CLI flag corresponded to which GUI wizard option.

Here’s a copy of my Dockerfile, which includes a subset of C++ Build Tools, CMake, Meson, Ninja, PowerShell, and Git.

I also included a helper script called VcVars.ps1, which pulls in environment variables you’d get from the Native Tools command prompts into the current instance of PowerShell, as that’s the default shell in GitLab Runner on Windows now.

Now for GitLab Runner. It’s great to see this is still as simple as it was seven years ago. Just a matter of downloading the latest release, copying it to some directory, installing it as a service, then registering with the GitLab server.

(host) > curl --output gitlab-runner.exe ^
  https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/binaries/gitlab-runner-windows-amd64.exe
(host) > .\gitlab-runner.exe install
(host) > .\gitlab-runner.exe start
(host) > .\gitlab-runner.exe register --url https://dev.aixxe.net --token glrt-k229Rj35dFykUy_Dzmts
Runtime platform                                    arch=amd64 os=windows pid=1076 revision=782c6ecb version=16.9.1
Enter the GitLab instance URL (for example, https://gitlab.com/):
[https://dev.aixxe.net]:
Verifying runner... is valid                        runner=k229Rj35d
Enter a name for the runner. This is stored only in the local config.toml file:
[WIN-VR4LB8F5MC8]:
Enter an executor: virtualbox, docker+machine, docker-autoscaler, custom, parallels, docker, docker-windows, kubernetes, instance, shell, ssh:
docker-windows
Enter the default Docker image (for example, mcr.microsoft.com/windows/servercore:1809):
aixxe.net/cpp-msvc-2022-build
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "C:\\GitLab-Runner\\config.toml"

It’s possible to register multiple runners on the same host, so I’ll have one for 32-bit jobs with tag “x86-windows-msvc” and another with tag “x64-windows-msvc” for 64-bit jobs. Both use the same image, differing only in the call to VcVars.ps1.

For this, I added a pre_build_script line to each of the [[runners]] objects. One for x86, one for x64.

pre_build_script = "C:\\VcVars.ps1 x86"  # or x64

Should all be good to start picking up jobs now, right?

Right… so the latest version of GitLab Runner (at the time of writing, v16.9.1) doesn’t support Windows Server 2025 yet. However, you can work around this by simply adding a new line to the supportedWindowsBuilds map in the source.

I put together a rough set of PowerShell commands to try handle all of this automatically.

That’s all for now. I need to go and finish off my build scripts for all the problematic projects…