linux

Chef Recipe: Oracle DB 11gR2 EE silent deploy

Chef provides a lot of flexibility and greater choice for infrastructure automation and I prefer it over others.

We should design our recipe in such a way that the our recipes without being modified can be used in any environment by maximizing the use of  attributes.

I was working on a deployment project on Linux x86-64 platform, where I had to automate all the infra components. Oracle 11g R2 EE is one of them. I will share the cookbook  here that can help many other. The recipes written here are used for silent installation of the DB using a response file after pulling the media files from a remote system.

Also the recipes are made idempotent, so that rerunning the cookbook again and again never do any damage. It automatically sets an attribute for DB installed / DB running in chef server after a successful compile -> run of the recipes.

Also the username/passwords are pulled stored and pulled from Encrypted Databag to make it more secure.

Here is the cookbook : https://github.com/kumarprd/Ora11gR2-EE-Silent-Install-Chef-Recipe

The recipes involved use below steps in sequence :

  1. setupenv.rb (It create the environment that will be used by rest of the recipes)
  2. oradb.rb (It checks the default attributes to fresh install/patch install and go further for any operations)
  3. install_oradb.rb ( Install the oracle database in ideompotent manner and sets the attributes in the server)
  4. create_schema.rb (This is application specific, but I will provide the template that can be modifed)

NOTE : Here create an encryoted databag with below json props  which are accessed inside recipes.

Follow  my other post : https://thegnulinuxguy.com/2016/08/09/chef-create-encrypted-data-bag-and-keep-secrets/

{

“id”: “apppass”,
“ora_db_passwd”: “dbpass”,
“oracle_pass”: “orapass”

}

Any issue/suggestion are welcome.

Docker Supervisord – Way to run multiple Demon process in a container

The docker was released keeping in mind, one daemon  per container which makes the container lightweight. Like suppose for running a web application, one container will serve database, one container will server as web server, one container  will server as  caching server connecting to DB.

So while writing a Dockerfile, the limitation  is : only one CMD  parameter can be be used inside it to run a single foreground process, the exit of which will stop the container.

But sometime we may face situations like to run more than one daemon process in a single container that is to setup the complete stack in a single container.

For this we can have two approaches:

  1. A bash script that will run all the processes in backened in sequence but the last one should run with only & to run as a foreground process.
  2. Using supervisor : Easy templatized way of managing multiple process in the container.

UseCase : I faced a situation where I have to run ssh,httpd,mysql in a single container and here is how I approached it with supervisor.

Also using the stdout of supervisor we can redirect the logs in terminal.

The three config file used here:

  1. Dockerfile
  2. supervisor.conf
  3. docker-compose.yml

These files can be accessed from my gitrepo :

https://github.com/kumarprd/docker-supervisor

Next run below commands:

  1. docker-compose build (It will build the image by reading the files)

#docker-compose build

Building web
Step 1 : FROM oraclelinux:6.8
—> 7187d444f0ce
Step 2 : ENV container docker
—> Running in 8cff18dabcc4
—> 655b5004777a

……..

……..

Step 20 : CMD /usr/bin/supervisord -c /etc/supervisor.conf
—> Running in 4ffed54b078f
—> dfb974e07bfb
Removing intermediate container 4ffed54b078f
Successfully built dfb974e07bfb

2. docker-compose up

# docker-compose up
Creating supervisord_web_1
Attaching to supervisord_web_1
web_1 | 2016-10-01 05:57:55,357 CRIT Supervisor running as root (no user in config file)
web_1 | 2016-10-01 05:57:55,357 WARN For [program:sshd], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
web_1 | 2016-10-01 05:57:55,357 WARN For [program:mysqld], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
web_1 | 2016-10-01 05:57:55,357 WARN For [program:httpd], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
web_1 | 2016-10-01 05:57:55,364 INFO supervisord started with pid 1
web_1 | 2016-10-01 05:57:56,369 INFO spawned: ‘httpd’ with pid 7
web_1 | 2016-10-01 05:57:56,373 INFO spawned: ‘sshd’ with pid 8
web_1 | 2016-10-01 05:57:56,377 INFO spawned: ‘mysqld’ with pid 9
web_1 | Could not load host key: /etc/ssh/ssh_host_rsa_key
web_1 | Could not load host key: /etc/ssh/ssh_host_dsa_key
web_1 | 161001 05:57:56 mysqld_safe Logging to ‘/var/log/mysqld.log’.
web_1 | 161001 05:57:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
web_1 | httpd: Could not reliably determine the server’s fully qualified domain name, using 172.18.0.2 for ServerName
web_1 | 2016-10-01 05:57:57,649 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
web_1 | 2016-10-01 05:57:57,649 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
web_1 | 2016-10-01 05:57:57,649 INFO success: mysqld entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

3. check the ps table

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
edd870f7e3ca testimg.supervisor “/usr/bin/supervisord” 19 minutes ago Up 19 minutes 0.0.0.0:5002->22/tcp, 0.0.0.0:5000->80/tcp, 0.0.0.0:5001->3306/tcp supervisord_web_1

 

4.  Connect to the container and check the services:

ssh -p 5002 root@<FQDN of the host where docker engine is running>

root@<FQDN>’s password:
Last login: Sat Oct 1 06:07:43 2016 from <FQDN>

[root@edd870f7e3ca ~]# /etc/init.d/mysqld status
mysqld (pid 101) is running…
[root@edd870f7e3ca ~]# /etc/init.d/httpd status
httpd (pid 7) is running…
[root@edd870f7e3ca ~]# /etc/init.d/sshd status
openssh-daemon (pid 8) is running…
[root@edd870f7e3ca ~]#

 

xend Error: Acquire running lock failed: 256

Recently I encountered this issue in OVMM 3.2.9  while starting a vm with

xm create <vm.cfg path>

The reason behind this found was : the vm was not shutdown properly  and the lock file is still there even if VM is down.

So the places to look at :

/var/log/xen/xend-debug.log
/var/run/ovs-agent/vm-*.lock
Look at the log file and if the lock file is present under /var/run/ovs-agent/ with the id of the vm which is not starting, just delete the lock file and then VM will start successfully.

xend issue : Xend has probably crashed! Invalid or missing HTTP status code.

I recently found  some VMs of one OVS node( from 30+ nodes)  went down and not able to start with this error :

Xend has probably crashed!  Invalid or missing HTTP status code.

There are many reasons behind this. And if you try to restart xend , it will not start.

The first place to look for is :

/var/log/xen/xend-debug.log

This log will say where exactly the issue is.

In my case my  / filesystem was running out of space because one log file consumed almost 8 GB . So I have to delete that file and now xend started successfully.

Using optparse in python

Sometimes we have to create tools that takes input as argument with certain options. We can create such tool with optparse module of python.

Here is a small example of using this.

 

[code language=”python”]

from optparse import OptionParser

parser = OptionParser(usage=’usage: %prog [options] arguments’)

parser.add_option(‘-a’,help="setup/cleanup",action="store", dest="action")
parser.add_option(‘-m’,help="email id",action="store", dest="email")
parser.add_option(‘-i’,help="Input json props",action="store", dest="input")
(options, args) = parser.parse_args()

[/code]

For help, type: ( This will display all the arguments that can be used with their format)

python tool.py -h

Usage: tool.py [options] arguments

Options:
-h, –help show this help message and exit
-a ACTION setup/cleanup
-m EMAIL email id
-i INPUT Input json props

save it in a programme and execute it as :

python tool.py -a setup -i file.json -m pdk@pdk.com

 

Now we can access the above inputs and use them, using below variables inside the programme:

[code language=”python”]

options.input

options.email

options.action

[/code]

 

Send mail using Python’s smtplib module

Python has a built in module to send mail to recipient[s] as to,cc,bcc. Here assumption is that : the smtp is configured in localhost (where the script will run).

[code language=”python”]
import socket
import smtplib
from email.mime.text import MIMEText

def SendMail(file,Email,status):
fp = open(file,’rb’)
msg = MIMEText(fp.read())
fp.close()
to=Email
cc=’def@example.com’
bcc=’123@example.com’
msg[‘Subject’] = ‘MULTINODE SETUP :: ‘+status
msg[‘From’] = ‘abc@example.com’
msg[‘to’] = to
msg[‘cc’] = cc

msg[‘bcc’] = bcc
toaddr=to.split(&quot;,&quot;)+cc.split(&quot;,&quot;)+bcc.split(&quot;,&quot;)
s = smtplib.SMTP(‘localhost’)
s.sendmail(‘something@example.com’,toaddr ,msg.as_string())
s.quit()

file=’/u01/work/tmp/sidtest’
Email=’myname@example.com’
status=’testing’
SendMail(file,Email,status)
[/code]

 

 

 

Chef – Deleting existing attributes

Sometimes we face situation like :

  1. May need to remove some persistent attributes which we set a flag after some work is done
  2. May be we set some attributes wrong so need to remove and reset the existing attribute.

The attribute may be set in attributes/default.rb OR insdie recipe with node normal attribute OR node set OR node override attributes.

There is a nice tool of chef (chef exec)  with which we can transform the attributes.

Usage:

Suppose a node with attribute hierarchy :

“normal” :[

{

“install_oradb” :  [

“is_installed” : “True”,

“is_running” :”True”

}

]

knife exec -E "nodes.transform(:all) {|n| n.normal_attrs[:install_oradb].delete(:is_installed) rescue nil }"

knife exec -E "nodes.transform(:all) {|n| n.normal_attrs[:install_oradb].delete(:is_running) rescue nil }"

 

Here we can replace normal_attrs with override_attrs OR default_attrs as required.

This will remove the attributes from all the nodes.

 

Suppose we want remove the attributes from a particular node. In this case you have to get search the node name from the chef solr index and replace the all with the required node.

e.g.

knife exec -E "nodes.transform(:node2) {|n| n.normal_attrs[:install_oradb].delete(:is_installed) rescue nil }"

knife exec -E "nodes.transform(:node2) {|n| n.normal_attrs[:install_oradb].delete(:is_running) rescue nil }"

Writing chef Library

In many cases we have to reuse same code again and again in our recipes. So to reduce this we can write our own library module and reuse it’s methods whenever required. This can help us use our own custom methods.

Like in my previous post we have to access the environment variables in many recipes. So we can use the same code as our library to create a module (lets say ProdEnvSetup)

 

  1. Create a library file under library directory of cook-book.(cook-book/library/envsetup.rb) and add below code.

 

[code language=”ruby”]module ProdEnvSetup
def setupenv()
hash1 = {}
File.open(“/u01/data/wor/app/conf/conf.prop”) do |fp|
fp.each do |line|
key, value = line.chomp.split(“=”,2)
hash1[key] = value
end
end

hash1.each do |key,value|
skey = “#{key.to_s}”.gsub(/\s|”|’/, ”)
svalue = “#{value.to_s}”.gsub(/\s|”|’/, ”)
ENV[skey] = svalue
end
end
end
end[/code]

2. Using library inside recipe ( 1st way )

So if the same environment variables are required inside a recipe, we can directly include the module inside our recipe.

Use below code inside begining of recipe:

[code language=”ruby”]class Chef::Recipe
include ProdEnvSetup
end
setupenv() [/code]

2. Using library ( 2nd way )

We can add additional encapsulation to our module by keeping it inside a recipe and include that recipe in every recipe, wherever its required.

create a recipe setupenv.rb and add above codes in 1st way(above)

and include it in other recipes like this:

[code language=”ruby”]include “prod-multiode-cookbook:setupenv”

[/code]

 


			

	
	



			
				


			
		

		
send foreground job/running process to run in  background
	


		

		
1.Moving Linux job from Foreground Into background :


    

  • start the program
    • 

  • Ctrl+Z (Will pause the program on the terminal)
    • 

  • jobs (find the job number from current shell)
    • 

  • bg  %jobnum
    • 



 


NOTE : Here we are only sending the process into background but exiting the programe still use the child shell of the current shell. So exiting the shell/terminal will kill the process.


 


2. Moving Linux jobs into background(nohup mode, freeing the shell)


    

  • Start the program
    • 

  • Ctrl+Z (Will pause the program on the terminal)
    • 

  • bg to run it in the background
    • 

  • disown -h ( shell disowns the process and will not get SIGHUP, so its kind of nohup mode)
    • 

  • exit (To exit from the shell)
    • 

  • Check in other terminal if the process is still running
    • 



 


NOTE : Here  running process is moved into background, and exiting the shell will not kill the process and will still run.

			

	
	



			
				


			
		

		
Docker Private Registry Setup
	


		

		
We can create our own secure private  docker repository where we can store our images and can be accessed from remote machine.


1. Goto /var/lib/docker in server and Create certificate using the domain name:


cd /var/lib/docker && mkdir certs
 mkdir -p certs && openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/sl09vmf0022.us.company.com.key -x509 -days 365 -out certs/sl09vmf0022.us.company.com.crt


2. Delete any old registry if exists:


docker rm  OR docker rmi registry:2


3. Recreate the registry using the newly created certificates by staying in the cert dir:


docker run -d -p 5000:5000 --restart=always --name bkdevregistry -v `pwd`/certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/sl09vmf0022.us.company.com.crt -e REGISTRY_HTTP_TLS_KEY=/certs/sl09vmf0022.us.company.com.key registry:2


4. Goto docker cert dir and copy the crt file as ca.crt and restart docker service


cd /etc/docker/certs.d/sl09vmf0022.us.company.com\:5000/
 cp /var/lib/docker/sl09vmf0022.us.company.com.crt /etc/docker/certs.d/sl09vmf0022.us.company.com\:5000/ca.crt
 update-ca-trust enable
 service docker restart


5. Now push images to private repository:


docker pull ubuntu
 docker tag ubuntu sl09vmf0022.ua.company.com:5000/ubuntu1404
 docker push sl09vmf0022.ua.company.com:5000/ubuntu1404


6. Client side configuration:


Copy the ca.crt file from docker registry server to local docker cert dir and restart docker service


mkdir -p /etc/docker/certs.d/sl09vmf0022.us.company.com\:5000/
 scp sl09vmf0022.us.company.com:/var/lib/docker/certs/sl09vmf0022.us.company.com.crt /etc/docker/certs.d/sl09vmf0022.us.company.com:5000/ca.crt
 service docker restart


7. Pull image from remote registry :


docker pull sl09vmf0022.us.company.com:5000/oel6u6


8. Check images in remote registry available using the crt file or in insecure mode:


curl -X GET https://sn09vmf0022.us.company.com:5000/v2/_catalog --cacert /etc/docker/certs.d/sn09vmf0022.us.company.com\:5000/ca.crt


OR


curl -X GET https://sl09vmf0022.us.company.com:5000/v2/_catalog --insecure