Developing on Kubernetes: my workflow for taming K8S on Windows

Developing on Kubernetes can be a difficult task even for seasoned developers if no sysadmin/devops knowledge is added to the mix. In this article I show what my setup and workflow look like when developing on Windows using Kubernetes. I mainly develop using Python and PHP but you can adapt every other language/framework with a bit of work.

Introduction – what is this about?

In this post I will not cover what a container is, how to create a Dockerfile or the innards of Kubernetes, I will assume the reader has a general understanding of these concepts. If you don’t have these concepts in your mind you can start here, you may need a few of them. On the other hand, if you are a developer who knows its way around basic Docker this article is for you. If you’re searching for a Vagrant-like experience using Kubernetes, this is the right place.

Developing against Kubernetes is hard, if you start without a cluster, or without proper knowledge, or without patience, and many other things, things will go wrong. And believe me, things tend to go wrong pretty quickly when developing and Kubernetes meet. When you add a Windows machine to the mix you encounter another bunch of problems that you may very well have avoided were you using a Linux one.

Before we dive deep into the how-to let’s take a look at the benefits you will be getting:

• Environment separation: you will get a clear distinction between your machine and the machine running Docker. In this setup I use a local virtual machine, but you can also connect to a remote cluster, if you wish to.
• Reusing the Docker Daemon: you won’t have to spawn multiple daemons and the same daemon will be used for building images and running containers.
• Set-and-forget: once the environment is ready it will require little-to-no-maintenance.
• Single point of access: you will interact with your application and your cluster within one single shell, you won’t need to jump using ssh or similar tools.
• Fast-paced development: you will be able to sync changes between your local machine and the running application without rebuilding the image, just like Vagrant.
• No pipeline required: you won’t need to set up a CI/CD pipeline to run this setup.
• RAM savings: compared to Vagrant where you need a full-blown virtual machine, you can save memory by running containers. You can also offload the memory requirements if you use a remote cluster!

Before we get started here’s a post of mine that showcases some Kubernetes tools that are useful when developing.

The setup

Here’s a list of what I will be using, followed by a description of why I chose to use that specific solution instead of alternatives:

• Operating System: Windows 10
• Hypervisor: HyperV
• Shell: PowerShell
• Kubernetes: Minikube + Docker for Windows
• Docker registry: Docker Registry (Minikube addon)
• Ingress controller: Traefik
• Main tool: DevSpace

You might be thinking: “Wait is the list really all there is to it?” The answer is yes.

Windows 10 is the platform of choice, HyperV being its native virtualization solution makes sense as a choice. I haven’t tested this setup with VirtualBox, but it should work nonetheless (if not better).

I use PowerShell to manage my environment because Cygwin (Babun) doesn’t play well with DevSpace (the main tool I will be using). Although I managed to keep Minikube working on both PowerShell and Cygwin simultaneously for a while, that is not the case anymore. While I wait for WSL2, I ditched Cygwin for pure PowerShell.

To run Kubernetes I selected Minikube, the other alternative being Docker for Windows. During my tests I have found Minikube to be more reliable and stable compared to Docker for Windows. Also Docker for Windows ships with slightly outdated versions of Kubernetes (1-2 versions lag). In order to use this setup you will also need to install Docker for Windows but you can stop it right after the setup, you only need the docker executable.

In order to develop without setting up a full-blown pipeline, I decided a temporary, local registry would come in handy. While I tried using Harbor and other solutions on top of Kubernetes, I found the Minikube addon to be the easiest and most effective way to set up a local registry (it is also painless).

The ingress controller of choice is Traefik. This might be the first thing that you haven’t heard of as a developer. Don’t worry too much about it, think of it as a “way to set up a web-server + port forwarding”.

And now the main tool, kudos to the guys at DevSpace. This tool will be the piece of software that controls your environment and your setup during development.

Setting up Docker for Windows, Minikube+HyperV and the Registry

I assume you know how to install Docker for Windows. You only need the docker executable, you can shut down the virtual machine produced by Docker for Windows as soon as you finish the installation.

I have written a detailed guide on setting up Minikube on Windows, you can follow it to get Minikube up and running, be sure to specify External as switch if you use HyperV. As a side note I suggest you to use –memory 4G and –disk-size 80G when starting up Minikube, 4G of RAM is pretty reasonable for developing on Kubernetes. While you can change memory according to your need, once disk size is set you can’t really modify it without some serious procedure. When you’re done it’s time to set up the registry, you can enable it by doing:

minikube addons enable registry

This will spawn a registry listening at minikube-ip:5000. In order to be able to push/pull images from this registry using your machine (without ssh-ing into minikube) you need to be able to access it as localhost. That can be accomplished using the following (kudos to the author of this blog post):

$(kubectl get po -n kube-system | grep kube-registry-v0 | \awk '{print $1;}') 5000:5000

The last thing you need to do is set up PowerShell to use the Docker Daemon inside Minikube (Dockerception!). You may need to do this every time you restart your shell/virtual machine so keep this command in mind:

& minikube docker-env | Invoke-Expression

Setting up Traefik

Setting up Traefik (I still use version 1.7) is an easy task:

kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml

Traefik is ready. You may also want to set up Traefik Dashboard. To do so modify this manifest and apply it using kubectl apply -f.

Downloading and installing DevSpace cli

In order to use DevSpace you need to install it first. There are two ways: through NPM or compiled binary. I spent many hours trying to make it work using NPM (as suggested by DevSpace developers) but in the end I deemed it not worth and used the compiled binary. You can get it from here. Once you’ve downloaded the Windows binary you must place it in a folder and add that folder to the PATH variable. I assume you know how to do it.

The workflow

I will keep it short:

• Produce a Dockerfile specific to your application
• Produce a devspace.yaml file specific to your application
• Create an entry in the hosts file
• Run devspace dev from your terminal
• ???
• Profit

While I will not show you how to create a Dockerfile because it depends on your application, I will show you a sample devspace.yaml file. This file must be placed in the same root directory of each application (usually where Dockerfile is). Here’s the sample:

version: v1beta7
images:
  myapp:
    image: localhost:5000/<namespace>/<imagename>
    createPullSecret: true
deployments:
- name: myapp
  component:
    containers:
    - image: localhost:5000/<namespace>/<imagename>
    service:
      ports:
      - port: 80
    ingress:
      rules:
        - host: myapp.example.com
dev:
  ports:
  - labelSelector:
      app.kubernetes.io/component: myapp
    forward:
    - port: 80
  sync:
  - labelSelector:
      app.kubernetes.io/component: myapp
    localSubPath: ./
    containerPath: $WORKDIR

The first important thing is the image path. While you can use DockerHub without much tweaking, if you use localhost:5000/<namespace>/<imagename> Kubernetes won’t complain about the image coming from an insecure source (and this will save you some headaches). You can make DevSpace build multiple images of course! The build will be handled by the Docker Daemon inside Minikube, this makes things really fast (compared to building using Kaniko).

• Line 7: the deployment name will help you keep track of your deployment.
• Line 10: instructs DevSpace to spawn a container using the image you just built.
• Line 13: will create a Kubernetes Service on port 80 for your application (you may add more than one).
• Line 16: this is the fake domain name you will be using to access your application, be sure to point your host file to the minikube ip address.
• Line 20 & 22: will perform port forwarding from the outside to your container (application). You must specify the selector you chose on Line 7.
• Line 25: this will instruct to sync the contents of your working directory with the contents of your container. You must make sure to set up localSubPath and containerPath accordingly. You must specify the selector you chose on Line 7.

Once you’re ready hit devspace dev and you’re ready to do some developing on Kubernetes. The content of your application folder will be synced with the container without rebuilding the image, this allows you to modify your code and see the changes in real time.

Moving your way around DevSpace and your environment

While this guide doesn’t cover DevSpace configuration files and options entirely (there are way too many) here’s a few tips for moving around with ease:

• devspace dev will continue running after the container has started, it will stream logs from your applications and keep the syncing process up.
• devspace enter can be used to open a shell inside your containers
• devspace ui will open up a nice web interface.
• devspace build will trigger the build process as defined in devspace.yaml.
• minikube down will shut down the virtual machine (it will not delete it).
• minikube up will start the virtual machine from scratch or load one if it has already been created.
• minikube ip will show you the IP address of the virtual machine.
• minikube ssh allows you to open a shell inside the virtual machine.
• & minikube docker-env | Invoke-Expression configures the docker executables

I’ve been using DevSpace for quite a while and it has improved a lot over time, developers are really committed to the project and it is easily my favorite tool when it comes to developing on Kubernetes.

Devspace has many configurations and commands that will help you develop your applications, for everything that’s not covered in this guide I suggest you read the official documentation.

Maintenance

The setup requires almost no maintenance, but from time to time you may notice that repeated builds of images performed by DevSpace may fill up the virtual machine disk space. You can clean those images with:

devspace cleanup images

Conclusion

Developing on Kubernetes is hard, even more so if you don’t have the right tools. This post aims to show a complete and fast way to iterate over development without straying too much from old-style development environments (e.g. Vagrant).

Once the environment is set up, Kubernetes is almost transparent to the developer who can use it to develop cloud-ready applications (Dockerfile already created).

Having multiple applications running in separate virtual machines will require resources, running the same workloads within containers on top of a single virtual machine will help reduce resource waste.

• Author
• Recent Posts

mark

The IT guy with a slight look of boredom in his eyes. Freelancer. Current interests: Kubernetes, Tensorflow, shiny new things.

Latest posts by mark (see all)

• 2020 A year in review for Marksei.com - 30 December 2020
• Red Hat pulls the kill switch on CentOS - 16 December 2020
• OpenZFS 2.0 released: unified ZFS for Linux and BSD - 9 December 2020