Kubernetes Jenkins Windows and PowerShell

As more people start adopting more DevOps practices, we start to realize that it is hard and takes time to master these technologies and practices. Usually, you would have all your code in change control by now (hopefully) but yo start to realize that is only the tip of the iceberg. But now throw in the mix you have to DevOps, CI/CD, containers, distributed systems on Windows ecosystems and this just got fun.

On this post I want to focus on following technologies like Kubernetes PowerShell, Docker, Jenkins and groovy. The objective is how to setup Jenkins inside a Kubernetes clusters with linux and Windows workers.

Where to find all the code
You can find all the code in my github repo here

https://github.com/gabrielrojasnyc/KubeJenkinsPowershell

What would you need to implement

Assumptions

  • Have a minimal understanding of Kubernetes
  • Know some basic PowerShell
  • Basic understanding of Linux systems
  • Some knowledge of containers
  • Have virtualbox install on your computer

Let’s start

  1. Lets go to the docker website to download  install it.  https:https://store.docker.com/editions/community/docker-ce-desktop-mac//www.docker.com/docker-mac
    2018-04-03_21-26-14.png
  2. Now drag and drop
    Screen Shot 2017-09-12 at 9.39.21 PM.png
  3. After it is install open the terminal window and type
    docker version2018-04-03_21-29-04
  4. We need to download and install Minikube to run kubernetes. Minikube is a tool that lets you run Kubernetes locally in your laptop. It needs to run inside a VM. This is only needed for local development.
    https://kubernetes.io/docs/getting-started-guides/minikube/

  5. To install Minikube on your laptop follow these instruction depending on your operating system https://kubernetes.io/docs/tasks/tools/install-minikube/

  6. You will also need to install kubectl. This the command line tool that kubernetes uses to interact with the environment
    https://kubernetes.io/docs/tasks/tools/install-kubectl/

  7. In my case as I have a MAC i just need to run the following to install these two components
    Minikube

    curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.25.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
    Kubectl
    curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/darwin/amd64/kubectl
    
    chmod +x ./kubectl
     sudo mv ./kubectl /usr/local/bin/kubect
  8.  Now let see some action. Type
    minikube start
    You will see the VM starting on VirtualBox and also it will create the Kubernetes cluster
    2018-04-05_08-52-342018-04-05_08-52-46
  9. Now that we kubernetes cluster to play with, we have to make sure we can reuse the docker daemon.
    eval $(minikube docker-env)
  10. After you type the command above and not type docker ps you will see some containers running for K8 to work
    2018-04-05_09-04-28
  11. Let’s open the Kubernetes dashboard as we are going to use see the status of our deployments and to manage our images
    minikube dashboard
    2018-04-05_09-09-37
  12. You can create, deploy, and destroy deployments from the console. But what would be the fun with that, we are going to use command line and make sure we are using version control.
  13. We are going to create a DockerFile to start our Jenkins master instance the way we need it
    # From the Jenkins supported version
    FROM jenkins/jenkins:lts
    LABEL version=0.1
    #using root for installation
    USER root
    # Install PowerShellCore
    RUN apt-get update -y
    RUN wget https://github.com/PowerShell/PowerShell/releases/download/v6.1.0-preview.1/powershell_6.1.0-preview.1-1.debian.9_amd64.deb
    RUN dpkg -i powershell_6.1.0-preview.1-1.debian.9_amd64.deb 2> /dev/null ; exit 0
    RUN apt-get install -f -y
    RUN rm powershell_6.1.0-preview.1-1.debian.9_amd64.deb
    #Plug-ins Installation
    #Install Kubernetes pluginin
    RUN /usr/local/bin/install-plugins.sh kubernetes:1.5
    #Install ssh plug-in
    RUN /usr/local/bin/install-plugins.sh ssh-slaves:1.26
    #Install PowerShell plug-in
    RUN /usr/local/bin/install-plugins.sh powershell:1.3
    #Install MSBuild plug-in
    RUN /usr/local/bin/install-plugins.sh msbuild:1.29
    # Install git for version control
    RUN /usr/local/bin/install-plugins.sh git:3.8.0
    # Install docker plug-in
    RUN /usr/local/bin/install-plugins.sh docker-plugin:1.1.3
    #Install xunit plug-in
    RUN /usr/local/bin/install-plugins.sh xUnit:1.102
    # Misc plug-ins
    RUN /usr/local/bin/install-plugins.sh cloudbees-folder:6.4
    RUN /usr/local/bin/install-plugins.sh antisamy-markup-formatter:1.5
    RUN /usr/local/bin/install-plugins.sh credentials-binding:1.16
    RUN /usr/local/bin/install-plugins.sh workflow-aggregator:2.5
    RUN /usr/local/bin/install-plugins.sh github-branch-source:2.3.3
    RUN /usr/local/bin/install-plugins.sh pipeline-github-lib:1.0
    RUN /usr/local/bin/install-plugins.sh pipeline-stage-view:2.10
    #update container one more time
    RUN apt-get update -y
    #Dropping back to jenkins user security reasons
    USER jenkins
  14. With this file we can now build our container. Make sure you are the root of the folder when running the following
    docker build -t jenkinstest .2018-04-05_10-41-08
  15. Now you will be able to see your image if you run
    docker images
    2018-04-05_10-42-46.png
  16. Run the following commands to make sure everything is register clean on a new shell window
    minikube stop
    minikube start

    eval $(minikube docker-env)
    
    
  17. Now that we have a container lets upload it to docker hub. I will not explain how to to this as there is a good explanation on this here https://docs.docker.com/docker-cloud/builds/push-images/
    The simple steps are here

    export DOCKER_ID_USER="username"
     $ docker login

  18. Now we need to tag the image that we created from before and upload
    $ docker tag my_image $DOCKER_ID_USER/my_image
     $ docker push $DOCKER_ID_USER/my_image

    2018-04-05_11-20-41.png

  19. We are doing all these because Kubernetes likes to download images from the container image hub. I am using docker hub as it is free and easy to use. Please keep in mind this repo are public and use at your own risk. Should not be use for production !!!!

  20. If  you go to you hub.docker.com you will see your container
    2018-04-05_11-28-35.png
  21. Run docker pull to download the image and make sure it is working correclty
    docker pull username/jenkinssummit2018-04-05_11-42-33
  22. To check if your image was downloaded you can run docker images and i will show you the current images that you available in your computer
    docker images
    2018-04-05_11-44-33
  23. Now lets get into Kubernetes a little bit. In order to deploy a Jenkins server in Kubernetes we need to create .yaml document. This is going to be used as deployment configuration


    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
    name: jenkins-deployment
    spec:
    replicas: 1
    template:
    metadata:
    labels:
    app: jenkins
    spec:
    containers:
    – name: jenkins
    image: gabrielrojas/jenkinssummit
    ports:
    – name: http-port
    containerPort: 8080
    – name: jnlp-port
    containerPort: 31000
    volumeMounts:
    – name: jenkins-persistent-storge
    mountPath: /var/jenkins_home
    volumes:
    – name: jenkins-persistent-storge
    emptyDir: {}

    I will not explain each attribute but this website has a good explanation and examples on what they mean
    https://www.mirantis.com/blog/introduction-to-yaml-creating-a-kubernetes-deployment/

  24. We have most of the pieces together lets create the POD using the deployment file
    kubectl apply -f JenkinsDeploySummit.yaml
    2018-04-05_12-57-47
  25. If you go back to the console you will see our pod deployed
    2018-04-05_12-59-18
  26. We are almost done and ready to use Jenkins and PowerShell. Before that we need to create service file. This file will allow to expose the POD outside of Kubernetes for our use. the file will look like this


    apiVersion: v1
    kind: Service
    metadata:
    name: jenkins
    spec:
    type: NodePort
    ports:
    – port: 8080
    targetPort: 8080
    name: http-port
    – port: 31000
    targetPort: 31000
    nodePort: 31000
    name: jnlp-port
    selector:
    app: jenkins

    2018-04-05_13-15-32.pngYou can also see the service running in the console
    2018-04-05_13-19-10

  27. If you run kubectl get services you will see the port that was exposed. In this case it is port 32649
    2018-04-05_13-21-46.png
  28. In order to connect to Jenkins we need to get the IP of minikube and then type the port. You can get it from the console url or by typing minikube ip In my case i am going to open new windows pointing to

    192.168.99.101:32649

    2018-04-05_13-27-13

  29. Now you will wonder how to get the administrator password. On the Kubernetes console go to overview and click on the following under pods. This will show you the logs and you will be able to get password2018-04-05_13-31-43.png2018-04-05_13-33-11
    Screen Shot 2017-09-12 at 10.26.11 PM.png
  30. Copy and paste the administrator password from the terminal window and then click on install suggested plug ins. This may take 5 to 10 minutes
    Screen Shot 2017-09-12 at 10.29.12 PM.png
  31. Now create a dummy user name and password as this is Jenkins master is for testing and will be destroy at some pointScreen Shot 2017-09-12 at 10.30.42 PM.png
  32. Let’s now configure Kubernetes plug-in in Jenkins so we can spin workers using Kubernetes backend.
    Go to Manage Jenkins -> Configure System
    Scroll all the way down where it says “Add a new Cloud”
    2018-04-05_13-43-41.png
  33. You will need the following information by running this commands on your shell
    kubectl cluster-info
    2018-04-05_13-48-55.pngGet the internal IP of the Jenkins container. You can either doing from the console or using kubectl describe pod2018-04-05_13-53-44
  34. In our case
    Kubernetes URL is : https://192.168.99.101:8443
    Jenkins URL is : http://172.17.0.4:8080  port that we exposed from DockerFile

    Run a test connection and make sure it is successful
    2018-04-05_13-58-42

  35. there is a small change on jnlp port we want to use. The reason I changed on the configs to 31000 is because that is one of the ports that kubernetes exposes to the world.. Make sure to change this under Manage Jenkins -> Global Security2018-04-06_00-28-04.png
  36. We need to tell Kubernetes what container to use for our workers. We are going to create a new DockerFile that will use powershell container that Microsoft provides and we are going to install Java.Java is needed for Jenkins to pass commands and information into the container


    # Microsoft Powershell Ubuntu image
    FROM microsoft/powershell
    LABEL version=0.1
    WORKDIR /tmp
    #Update and install Java
    RUN apt-get update -y
    RUN apt-get install default-jre -y
    RUN apt-get upgrade powershell -y
    #Files to be copy for demo but this should be downloaded from GIT repo
    COPY test.ps1 /tmp/test.ps1
    #Expose Port
    EXPOSE 22 8500 50000
    CMD /bin/bash

  37. We are going to follow the steps from above to build the container and publish it on Docker hub
    docker build -t powershell1 .
    docker tag powershell1 gabrielrojas/powershellsummit
    docker push gabrielrojas/powershellsummit

    2018-04-05_14-36-25
  38. Now that we have our container uploaded lets go back into Jenkins to configure.
    Manage Jenkins -> Configure. Scroll all the way down and click on Pod Template.
    Fill-out with following information
    2018-04-05_14-40-10
  39. This should give you access to 2 different containers with PowerShell. Now we are going to configure Jenkins Master in Kubernetes to communicate to a Windows 2016 server.
  40. The first thing to do is to create a node on Windows under Manage Jenkins -> New Node
    2018-04-06_00-30-59.png
  41. Now configure the node with this exact parameters
    2018-04-06_00-32-05
  42. Login to the Windows machine and open the Internet Explorer to browse to the jenkins master server and type the following
    http://jenkinsmasterIP:port/windows2016/slave-agent.jnlp
    A new window will pop up, click on file and connect
    2018-04-06_00-49-552018-04-06_00-50-56.png
  43. Now your node should be connected and talking to Master Jenkins container being managed by Kubernetes2018-04-06_00-52-54
  44. Lets create a job to run the following code on the windows box to do our UNIT testing. We would need to use xunit to report the failures properly. The configuration will look like this
    Click on Jenkins -> New Item -> Select FreeStyle
    2018-04-06_08-06-162018-04-06_08-04-442018-04-06_08-03-54
  45. After you use run the the job you will see the failures.
  46. This is code I am running for Tests.ps1 to simulate the failure


    # To simulate failure pester check
    Describe 'Service' {
    it 'is summit installed' {
    get-service Name Summit | Should be $true
    }
    }

    view raw

    summit.Test.ps1

    hosted with ❤ by GitHub

  47. However, the best way to create a job is by creating a pipeline job. The advantage of using this is that all code is version control and it is reusable.
    2018-04-06_08-19-00.png
  48. In this case scroll down and paste your code
    2018-04-06_08-20-56.png


    node ('Windows2016') {
    stage ('Test Pass') {
    powershell 'Invoke-OperationValidation -testtype simple'
    }
    stage ('Test 2'){
    powershell 'invoke-pester c:\test_gabe.Tests.ps1 -OutputFormat NunitXML -OutputFile report.xml -EnableExit'
    }
    }

    ENJOY !!!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s