Custom default backend error pages of kubernetes ingress

The kubernetes nginx ingress controller has a default backend which show error pages of 404, 502 with nginx string in the error page. But some times we need to show a valid custom error page instead of showing the pages served by the default backend.

The process is simple. We need to create a configmap with custom error pages, create deployment with image k8s.gcr.io/ingress-nginx/nginx-errors with mounting the config map in /www. Also we need to create service which will be used as the default backend service of the ingress controller.

Configmap Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L72-L539

Note : update the custom error pages under data with the required error HTML content

Deployment Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L19-L70

Service Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L2-L17

Modification in ingress controller arguments : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/ingress-deploy.yaml#L337

Note: Here update the service name matching the custom error service name

Next thing we need to update the ingress definition file for which we want to use the custom error pages.

We need to add 2 annotations for this :

  1. Pointing to the custom error service name
  2. mention the custom error to be served.

Ingress manifest update Example : https://github.com/divyaimca/my-k8s-test-projects/blob/main/rabbitmq_kustom/rabbitmq-ingress.yaml#L9-L10

Now if you want to access the webpage served by the ingress with some error, the ingress will serve the customised backend error pages instead of the default backend error pages.

Jenkins slave as a service in windows to start automatically

In Jenkins many time we have to add windows machine as slave, where we need the agent to be up and running as windows service.

There are many ways to do it, but I struggled to find the correct configuration steps. I used windows resource toolkit to make it work and adding the steps here.

Configuration Steps:

(A) Adding the windows slave in Jenkins server :

1. Add the agent in Jenkins Master with Launch Method : Launch Via Java Web Start

Screen Shot 2019-09-25 at 4.07.06 PM

(B) Creating the Service in windows Server for starting the slave agent: (In Windows Server 2016)

1. Download and install the Java 8.
2. Down and Install Windows Resource Kit Tools (https://www.microsoft.com/en-us/download/details.aspx?id=17657)
3. Create a blank service called “Jenkins Slave” by running the following from a command prompt

“C:\Program Files (x86)\Windows Resource Kits\Tools\instsrv.exe” “Jenkins Slave” “C:\Program Files (x86)\Windows Resource Kits\Tools\srvany.exe”

4. Open Registry Editor and go to below location :

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Jenkins Slave

Now Follow the below steps carefully.

  1. Create a string value “Description”
  2. Populate it with “Jenkins Continuous Integration Slave”

Screen Shot 2019-09-25 at 2.13.33 PM

  1. Create a new key “Parameters”
  2. Under “Parameters” create a new string value “Application”
  3. Populate it with the full path to java.exe, something like “C:\sapjvm_8\bin\java.exe”
  4. Under “Parameters” Create a new string value “AppParameters”
  5. Populate it with

“-jar E:\jenkins_agent\agent.jar -jnlpUrl http://m1-hostname.lab.saas.company.corp:8080/computer/hostame-of-machone/slave-agent.jnlp -secret <secret-name> -workDir E:\jenkinsWorkSpace”

  1. The slave.jar should point to the correct location
  2. The Jenkins master machine name should be correct
  3. The new Jenkins slave machine should be correct
  4. Make sure you use the secret for this machine that you copied from the master when adding the new node

Screen Shot 2019-09-25 at 2.24.06 PM

Open the Services application from Control Panel – Administrative Tools, find the “Jenkins Slave” service, right click on the service and go to “Properties”.

  1. Go to the “Recovery” tab and change “First failure” and “Second failure” to “Restart the Service” – occasionally we found it wouldn’t start up first time out
  2. Go to the “Log On” tab and set an account and password- we found that using an account with local admin rights on the slave machine worked best but this is probably unnecessary
  3. Go to the “General” tab and change the “Startup type” to “Automatic” – make sure the service
  4. starts up when you restart the slave
  5. Click the “OK” button
  6. Now start the serviceScreen Shot 2019-09-25 at 2.27.33 PM

6. The service will run by default during startup of the windows machine.

7. Now Verify the agent is up and running in Jenkins Web Page.

Screen Shot 2019-09-25 at 4.03.00 PM

Chef Issue – Recover deleted user pivotal

By default “pivotal” is the only chef server superuser who has permission to CREATE users,orgnization, group etc in chef server.  So if by mistake you will delete the “pivotal” user with below command :

# chef-server-ctl user-delete pivotal

Then , further is you run any command(list,create,delete,etc) related to users, organization , it will fail with the following error :

Response:  Failed to authenticate as 'pivotal'. Ensure that your node_name and client key are correct.

 

So to overcome this issue we have to recreate “pivotal” using its with required authorization  in pgdb.

So follow below steps to do it.

create pivotal’s public key from /etc/opscode/pivotal.pem and store in an accessible location

#openssl rsa -in /etc/opscode/pivotal.pem -pubout > /var/opt/opscode/postgresql/9.2/data/pivotal.pub

get the pivotal user’s authz_id and store in an accessible location

# echo "SELECT authz_id FROM auth_actor WHERE id = 1" | su -l opscode-pgsql -c 'psql bifrost -tA' | tr -d '\n' > /var/opt/opscode/postgresql/9.2/data/pivotal.authz_id

create the pivotal user’s record

# echo "INSERT INTO users (id, authz_id, username, email, pubkey_version, public_key, serialized_object, last_updated_by, created_at, updated_at) VALUES (md5(random()::text), pg_read_file('pivotal.authz_id'), 'pivotal', 'kryptonite@opscode.com', 0, pg_read_file('pivotal.pub'), '{\"first_name\":\"Clark\",\"last_name\":\"Kent\",\"display_name\":\"Clark Kent\"}', pg_read_file('pivotal.authz_id'), LOCALTIMESTAMP, LOCALTIMESTAMP);" | su -l opscode-pgsql -c 'psql opscode_chef'

delete the temporary files

# rm /var/opt/opscode/postgresql/9.2/data/pivotal.pub /var/opt/opscode/postgresql/9.2/data/pivotal.authz_id

Docker Private Registry Setup

We can create our own secure private  docker repository where we can store our images and can be accessed from remote machine.

1. Goto /var/lib/docker in server and Create certificate using the domain name:

cd /var/lib/docker && mkdir certs
 mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/sl09vmf0022.us.company.com.key -x509 -days 365 -out certs/sl09vmf0022.us.company.com.crt

2. Delete any old registry if exists:

docker rm  OR docker rmi registry:2

3. Recreate the registry using the newly created certificates by staying in the cert dir:

docker run -d -p 5000:5000 --restart=always --name bkdevregistry -v `pwd`/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sl09vmf0022.us.company.com.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sl09vmf0022.us.company.com.key registry:2

4. Goto docker cert dir and copy the crt file as ca.crt and restart docker service

cd /etc/docker/certs.d/sl09vmf0022.us.company.com\:5000/
 cp /var/lib/docker/sl09vmf0022.us.company.com.crt /etc/docker/certs.d/sl09vmf0022.us.company.com\:5000/ca.crt
 update-ca-trust enable
 service docker restart

5. Now push images to private repository:

docker pull ubuntu
 docker tag ubuntu sl09vmf0022.ua.company.com:5000/ubuntu1404
 docker push sl09vmf0022.ua.company.com:5000/ubuntu1404

6. Client side configuration:

Copy the ca.crt file from docker registry server to local docker cert dir and restart docker service

mkdir -p /etc/docker/certs.d/sl09vmf0022.us.company.com\:5000/
 scp sl09vmf0022.us.company.com:/var/lib/docker/certs/sl09vmf0022.us.company.com.crt /etc/docker/certs.d/sl09vmf0022.us.company.com:5000/ca.crt
 service docker restart

7. Pull image from remote registry :

docker pull sl09vmf0022.us.company.com:5000/oel6u6

8. Check images in remote registry available using the crt file or in insecure mode:

curl -X GET https://sn09vmf0022.us.company.com:5000/v2/_catalog --cacert /etc/docker/certs.d/sn09vmf0022.us.company.com\:5000/ca.crt

OR

curl -X GET https://sl09vmf0022.us.company.com:5000/v2/_catalog --insecure

Using Docker – Part 1

In this part we will go through some simple usage of docker command.

Use -D with docker for debug mode.
Docker images are Immutable and Containers are Ephemeral.

How to get help ??

docker help
docker <command> help

1. Check images:

docker images

2. Run an application in the container:

( We have already downloaded oraclelinux:6.6 image from dockerhub)

-i flag starts an interactive container.

-t flag creates a pseudo-TTY that attaches stdin and stdout

docker run -i -t –name guest companylinux:6.6 /bin/bash

–name -> create an container instance with the name using the image companylinux6.6
execute /bin/bash isinde the container guest

NOTE : Here if image doesnt exist locally it will try to pull it from docker hub

3. Create an image and remove the container once logged out

 

docker run -i -t –rm companylinux:6.6 /bin/bash

 

4. Show all info about running processes in docker

docker ps
docker ps -a

5. Show info of processes running inside a container(here guest)

docker top guest

6. Run additional processes inside (guest here)

docker exec -it guest <command>

7. Create a container with a name that can be started in later time

docker create -it –name guest1 companylinux:7 /bin/bash

8. Start a container instance and Attach current shell to a docker container instance guest1

docker start -ai <container name> OR docker start -ai <container id>

9. stop instance and exit from the container

docker stop <containerid>

10. remove a container instance

docker rm guest1

11. Show all logs currently happening inside

docker logs -f guest

-f > updates the output in realtime

12. Get full information about a container in json format with inspect

docker inspect –format ='{{ .State.running}}’ guest1

13. Relaunch a container:

Look at the docker ps -all output and note down the CONTAINER_ID. If want to relaunch with interactive mode use -i option else just start.

docker start -i cfb007d616b9

OR

docker start cfb007d616b9

14. start/attach to a running Container

docker start <ID of comtainer>

15. Change the behaviour of the containers when exits from the container instance (add the option with run command )

–restart=always

Docker always attempts to restart the container when the container exits.

–restart=no
Docker does not attempt to restart the container when the container exits. This is the default policy.

–restart=on-failure[:max-retry]
Docker attempts to restarts the container if the container returns a non-zero exit code. You can optionally specify the maximum number of times that Docker will try to restart the container.

–rm (use this with run command, so that once you exit from the instance, it will get removed)

16. Local repo creation:(Use registry with tag 2, base host port 5000 mapped to registry container instance port 5000, names with localregistry)

docker run -d -p 5000:5000 –restart=always –name localregistry registry:2

17. Add images to local repository:(pull from docker hub OR create local image, tag it ,push it into local repo, pull it to from localrepo to create instance)

docker pull companylinux:6.6
docker tag companylinux:6.6 localhost:5000/oel6u6
docker push localhost:5000/oel6u6
docker pull localhost:5000/oel6u6

18. Stop and remove any instance

 

docker stop <container id> OR docker stop <instance-name>
docker rm <container id> OR docker rm <instance-name>

19. Remove image from repository(use -f for force remove)

docker rmi <imageid> OR docker rmi <imagereponame>
docker rmi -f <imageid> OR docker rmi -f <imagereponame>

20. Remove dead process entry from (docker ps -all) where any instance is in stopped state

docker rm $(docker ps -a -q)

Docker Concept & Setup

Why Containerization ?

Up to now we have been working with monolithic applications where different components of service are packaged into a single application which is easy to develop, test and deploy.But when it becomes large and complex it’s become difficult as one team to work on it and the risk of failure is high at deploy time.
So to overcome, a new trend has been followed to work with microservices where components of the monolithic application are divided into small microservices. Here every microsevice will have its own API to handle its part of the application.

  • It has advantages like each smaller service can use its own technology stack.
  • The developers will find it easy to understand a single service.
  • It’s also quicker to build and faster to deploy.
  • The application becomes distributed and microservice scales quicker horizontally than vertical and becomes more fault tolerant.

Virtual Machines are too big to transfer and often too slow.

So containerization is the better choice when adopting Microservices architecture.

Container ???

  • Container is all about running an application and not just a VM
  • Container is  a virtualization method at operating system level, that allows running multiple instances of OS running in same kernel.
  • Container is an image that contains apps, library, dependencies and most important kernel space components are provided by host operating systems
    • NameSpace : Global system resources like network, PID, mount points are presented as such a way that container thinks this is only available to it
    • CGroup : Used to reserve and allocate resources to container
    • Union file system : Merge different file systems into one virtual file system.
    • Capabilities : Managing privileges like root/nonroot

 

Docker ??

Docker is one of the most popular container product, that is based on LXC and  is an open platform to build , ship and run distributed applications.

 

  –   Docker Engine : portable, lightweight runtime packaging tool
   –  Docker Hub: A cloud service for sharing application
  • Docker enables application to quick assemble from components
  • It removes the friction between Dev,QA, Prod envs.
  • The same app unchanged can run anywhere (lappy/PC/datacente).

Docker images are built from Dockerfile and the containers are built from images.

:: Setup ::

Installing Docker is easy. All the commands used here are in OEL6 in my workplace.

1. Installation:

Update OS to atleast OEL6_UEK4 repo to use kernel > 4.1 (yum update and confirm kernel version, os > 6.4)
[ol6_UEKR4]
name=Latest Unbreakable Enterprise Kernel Release 4 for company Linux $releasever ($basearch)
baseurl=http://public-yum.company.com/repo/companyLinux/OL6/UEKR4/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-company
gpgcheck=1
enabled=1

yum update and reboot

> use docker repo:

[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/companylinux/6
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

2. Use btrfs filesystem:

yum install btrfs-progs
mkfs.btrfs /dev/sdb ( Add a raw disk and format with brtfs )
(FS tab entry )/dev/sdb /var/lib/docker btrfs defaults 0 0

3. Add proxy (if any to contact docker HUB)

/etc/sysconfig/docker ( If any ) OR add in /etc/default/docker( to use it with CURL)

export HTTP_PROXY=”proxy_URL:port”
export HTTPS_PROXY=”proxy_URL:port”

4. Modify docker config

In /etc/init.d/docker

Update

“$unshare” -m — $exec -d $other_args &>> $logfile &

to

$exec -d $other_args &>> $logfile &

5. Start docker service

# service docker start
# chkconfig docker on

6. Check docker details

service docker status
docker info
docker version

7. Add local user to docker group

groupadd docker
usermod -a -G docker <local docker>
chmod g+rx /var/lob/docker

8. Search images in docker hub:(Before pulling check the availability)

docker search oraclelinux
docker searcg centos
docker searcg registry

9. pull oracle linux6.6 image:

docker pull oraclelinux:6.6

here oraclelinux – image is 6.6 – version

10. Check images:

docker images

11. Add this env variable for authenticity, integrity of images

export DOCKER_CONTENT_TRUST=1

 

Puppet Quick Tutorial with examples

Puppet Version Used : 3.8

Distro Used : RHEL,CentOS, OEL

1. Puppet master :

Get the repo with below link.
rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

install the master
yum install puppet-server

Enter the master server hostname in the puppet.conf file under [main]
dns_alt_names = puppet,puppetmaster01,vmf0270,vmf0270.us.xxx.com

certname = vmf0270.us.xxx.com
server = vmf0270.us.xxx.com
environment = prod
runinterval = 1h
strict_variables = true

If this is the only puppet master in your deployment, or if it will be acting as the CA server :
puppet master –verbose –no-daemonize

2. Install puppet agent;

Get the repo –
rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm
Install the puppet package
yum install puppet /

For upgrade the existing package –
puppet resource package puppet ensure=latest and restart the service

Update puppet.conf and add master server IP.
Update under [main]
certname = vmf0207.us.xxx.com
server = vmf0270.us.xxx.com
environment = prod
runinterval = 1h

start the puppet service
puppet resource service puppet ensure=running enable=true

create a cronjob which will pull the configurations in every 30 mins
puppet resource cron puppet-agent ensure=present user=root minute=30 command=’/usr/bin/puppet agent –onetime –no-daemonize –splay’

3. Keep all the required manifests and modules in the directory server or create an empty site.pp file.
touch /etc/puppet/manifests/site.pp

4. A prod ready webserver is required to operate and manage from GUI Webpage.(else all can be managed from CLI)

5. Sign all the client requests by the server:

To check the list:
puppet cert list

To sign;
puppet cert sign –all OR individually by puppet cert sign

 

I have created few puppet manifests to manage the compute infrastructure.

Can be found here : https://github.com/kumarprd/puppet-manifests