xend issue : Xend has probably crashed! Invalid or missing HTTP status code.

I recently found  some VMs of one OVS node( from 30+ nodes)  went down and not able to start with this error :

Xend has probably crashed!  Invalid or missing HTTP status code.

There are many reasons behind this. And if you try to restart xend , it will not start.

The first place to look for is :

/var/log/xen/xend-debug.log

This log will say where exactly the issue is.

In my case my  / filesystem was running out of space because one log file consumed almost 8 GB . So I have to delete that file and now xend started successfully.

Chef Issue – Recover deleted user pivotal

By default “pivotal” is the only chef server superuser who has permission to CREATE users,orgnization, group etc in chef server.  So if by mistake you will delete the “pivotal” user with below command :

# chef-server-ctl user-delete pivotal

Then , further is you run any command(list,create,delete,etc) related to users, organization , it will fail with the following error :

Response:  Failed to authenticate as 'pivotal'. Ensure that your node_name and client key are correct.

 

So to overcome this issue we have to recreate “pivotal” using its with required authorization  in pgdb.

So follow below steps to do it.

create pivotal’s public key from /etc/opscode/pivotal.pem and store in an accessible location

#openssl rsa -in /etc/opscode/pivotal.pem -pubout > /var/opt/opscode/postgresql/9.2/data/pivotal.pub

get the pivotal user’s authz_id and store in an accessible location

# echo "SELECT authz_id FROM auth_actor WHERE id = 1" | su -l opscode-pgsql -c 'psql bifrost -tA' | tr -d '\n' > /var/opt/opscode/postgresql/9.2/data/pivotal.authz_id

create the pivotal user’s record

# echo "INSERT INTO users (id, authz_id, username, email, pubkey_version, public_key, serialized_object, last_updated_by, created_at, updated_at) VALUES (md5(random()::text), pg_read_file('pivotal.authz_id'), 'pivotal', 'kryptonite@opscode.com', 0, pg_read_file('pivotal.pub'), '{\"first_name\":\"Clark\",\"last_name\":\"Kent\",\"display_name\":\"Clark Kent\"}', pg_read_file('pivotal.authz_id'), LOCALTIMESTAMP, LOCALTIMESTAMP);" | su -l opscode-pgsql -c 'psql opscode_chef'

delete the temporary files

# rm /var/opt/opscode/postgresql/9.2/data/pivotal.pub /var/opt/opscode/postgresql/9.2/data/pivotal.authz_id

Docker Concept & Setup

Why Containerization ?

Up to now we have been working with monolithic applications where different components of service are packaged into a single application which is easy to develop, test and deploy.But when it becomes large and complex it’s become difficult as one team to work on it and the risk of failure is high at deploy time.
So to overcome, a new trend has been followed to work with microservices where components of the monolithic application are divided into small microservices. Here every microsevice will have its own API to handle its part of the application.

  • It has advantages like each smaller service can use its own technology stack.
  • The developers will find it easy to understand a single service.
  • It’s also quicker to build and faster to deploy.
  • The application becomes distributed and microservice scales quicker horizontally than vertical and becomes more fault tolerant.

Virtual Machines are too big to transfer and often too slow.

So containerization is the better choice when adopting Microservices architecture.

Container ???

  • Container is all about running an application and not just a VM
  • Container is  a virtualization method at operating system level, that allows running multiple instances of OS running in same kernel.
  • Container is an image that contains apps, library, dependencies and most important kernel space components are provided by host operating systems
    • NameSpace : Global system resources like network, PID, mount points are presented as such a way that container thinks this is only available to it
    • CGroup : Used to reserve and allocate resources to container
    • Union file system : Merge different file systems into one virtual file system.
    • Capabilities : Managing privileges like root/nonroot

 

Docker ??

Docker is one of the most popular container product, that is based on LXC and  is an open platform to build , ship and run distributed applications.

 

  –   Docker Engine : portable, lightweight runtime packaging tool
   –  Docker Hub: A cloud service for sharing application
  • Docker enables application to quick assemble from components
  • It removes the friction between Dev,QA, Prod envs.
  • The same app unchanged can run anywhere (lappy/PC/datacente).

Docker images are built from Dockerfile and the containers are built from images.

:: Setup ::

Installing Docker is easy. All the commands used here are in OEL6 in my workplace.

1. Installation:

Update OS to atleast OEL6_UEK4 repo to use kernel > 4.1 (yum update and confirm kernel version, os > 6.4)
[ol6_UEKR4]
name=Latest Unbreakable Enterprise Kernel Release 4 for company Linux $releasever ($basearch)
baseurl=http://public-yum.company.com/repo/companyLinux/OL6/UEKR4/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-company
gpgcheck=1
enabled=1

yum update and reboot

> use docker repo:

[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/companylinux/6
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

2. Use btrfs filesystem:

yum install btrfs-progs
mkfs.btrfs /dev/sdb ( Add a raw disk and format with brtfs )
(FS tab entry )/dev/sdb /var/lib/docker btrfs defaults 0 0

3. Add proxy (if any to contact docker HUB)

/etc/sysconfig/docker ( If any ) OR add in /etc/default/docker( to use it with CURL)

export HTTP_PROXY=”proxy_URL:port”
export HTTPS_PROXY=”proxy_URL:port”

4. Modify docker config

In /etc/init.d/docker

Update

“$unshare” -m — $exec -d $other_args &>> $logfile &

to

$exec -d $other_args &>> $logfile &

5. Start docker service

# service docker start
# chkconfig docker on

6. Check docker details

service docker status
docker info
docker version

7. Add local user to docker group

groupadd docker
usermod -a -G docker <local docker>
chmod g+rx /var/lob/docker

8. Search images in docker hub:(Before pulling check the availability)

docker search oraclelinux
docker searcg centos
docker searcg registry

9. pull oracle linux6.6 image:

docker pull oraclelinux:6.6

here oraclelinux – image is 6.6 – version

10. Check images:

docker images

11. Add this env variable for authenticity, integrity of images

export DOCKER_CONTENT_TRUST=1

 

Fix SSH Issue : Offending key for IP in ~/.ssh/known_hosts:n

When ever we connect to another host with ssh client, it asks the user to add the public key of the remote host in the ~/.ssh/known_hosts file of that user. This is by default enabled by “StrictHostKeyChecking yes” which can be altered to no in /etc/ssh/ssh_config file.

But sometimes what happened, the host name remain same, but the key value might have been changed, due to some reason like : may be a new system with same host name OR the system may be compromised OR any other reason, the error will come as the fingerprint will mismatch:

Offending key for IP in ~/.ssh/known_hosts:n

Where ‘n’ is the line number in that file.

So in this case manually open the   ~/.ssh/known_hosts and remove the line number “n”.

Or a simply perl pie will search and replace the line with an empty.

  perl -pi -e ‘s/\Q$_// if ($. == n);’ ~/.ssh/known_hosts