Chef Recipe: Oracle DB 11gR2 EE silent deploy

Chef provides a lot of flexibility and greater choice for infrastructure automation and I prefer it over others.

We should design our recipe in such a way that the our recipes without being modified can be used in any environment by maximizing the use of  attributes.

I was working on a deployment project on Linux x86-64 platform, where I had to automate all the infra components. Oracle 11g R2 EE is one of them. I will share the cookbook  here that can help many other. The recipes written here are used for silent installation of the DB using a response file after pulling the media files from a remote system.

Also the recipes are made idempotent, so that rerunning the cookbook again and again never do any damage. It automatically sets an attribute for DB installed / DB running in chef server after a successful compile -> run of the recipes.

Also the username/passwords are pulled stored and pulled from Encrypted Databag to make it more secure.

Here is the cookbook : https://github.com/kumarprd/Ora11gR2-EE-Silent-Install-Chef-Recipe

The recipes involved use below steps in sequence :

  1. setupenv.rb (It create the environment that will be used by rest of the recipes)
  2. oradb.rb (It checks the default attributes to fresh install/patch install and go further for any operations)
  3. install_oradb.rb ( Install the oracle database in ideompotent manner and sets the attributes in the server)
  4. create_schema.rb (This is application specific, but I will provide the template that can be modifed)

NOTE : Here create an encryoted databag with below json props  which are accessed inside recipes.

Follow  my other post : https://thegnulinuxguy.com/2016/08/09/chef-create-encrypted-data-bag-and-keep-secrets/

{

“id”: “apppass”,
“ora_db_passwd”: “dbpass”,
“oracle_pass”: “orapass”

}

Any issue/suggestion are welcome.

Docker Supervisord – Way to run multiple Demon process in a container

The docker was released keeping in mind, one daemon  per container which makes the container lightweight. Like suppose for running a web application, one container will serve database, one container will server as web server, one container  will server as  caching server connecting to DB.

So while writing a Dockerfile, the limitation  is : only one CMD  parameter can be be used inside it to run a single foreground process, the exit of which will stop the container.

But sometime we may face situations like to run more than one daemon process in a single container that is to setup the complete stack in a single container.

For this we can have two approaches:

  1. A bash script that will run all the processes in backened in sequence but the last one should run with only & to run as a foreground process.
  2. Using supervisor : Easy templatized way of managing multiple process in the container.

UseCase : I faced a situation where I have to run ssh,httpd,mysql in a single container and here is how I approached it with supervisor.

Also using the stdout of supervisor we can redirect the logs in terminal.

The three config file used here:

  1. Dockerfile
  2. supervisor.conf
  3. docker-compose.yml

These files can be accessed from my gitrepo :

https://github.com/kumarprd/docker-supervisor

Next run below commands:

  1. docker-compose build (It will build the image by reading the files)

#docker-compose build

Building web
Step 1 : FROM oraclelinux:6.8
—> 7187d444f0ce
Step 2 : ENV container docker
—> Running in 8cff18dabcc4
—> 655b5004777a

……..

……..

Step 20 : CMD /usr/bin/supervisord -c /etc/supervisor.conf
—> Running in 4ffed54b078f
—> dfb974e07bfb
Removing intermediate container 4ffed54b078f
Successfully built dfb974e07bfb

2. docker-compose up

# docker-compose up
Creating supervisord_web_1
Attaching to supervisord_web_1
web_1 | 2016-10-01 05:57:55,357 CRIT Supervisor running as root (no user in config file)
web_1 | 2016-10-01 05:57:55,357 WARN For [program:sshd], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
web_1 | 2016-10-01 05:57:55,357 WARN For [program:mysqld], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
web_1 | 2016-10-01 05:57:55,357 WARN For [program:httpd], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
web_1 | 2016-10-01 05:57:55,364 INFO supervisord started with pid 1
web_1 | 2016-10-01 05:57:56,369 INFO spawned: ‘httpd’ with pid 7
web_1 | 2016-10-01 05:57:56,373 INFO spawned: ‘sshd’ with pid 8
web_1 | 2016-10-01 05:57:56,377 INFO spawned: ‘mysqld’ with pid 9
web_1 | Could not load host key: /etc/ssh/ssh_host_rsa_key
web_1 | Could not load host key: /etc/ssh/ssh_host_dsa_key
web_1 | 161001 05:57:56 mysqld_safe Logging to ‘/var/log/mysqld.log’.
web_1 | 161001 05:57:56 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
web_1 | httpd: Could not reliably determine the server’s fully qualified domain name, using 172.18.0.2 for ServerName
web_1 | 2016-10-01 05:57:57,649 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
web_1 | 2016-10-01 05:57:57,649 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
web_1 | 2016-10-01 05:57:57,649 INFO success: mysqld entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

3. check the ps table

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
edd870f7e3ca testimg.supervisor “/usr/bin/supervisord” 19 minutes ago Up 19 minutes 0.0.0.0:5002->22/tcp, 0.0.0.0:5000->80/tcp, 0.0.0.0:5001->3306/tcp supervisord_web_1

 

4.  Connect to the container and check the services:

ssh -p 5002 root@<FQDN of the host where docker engine is running>

root@<FQDN>’s password:
Last login: Sat Oct 1 06:07:43 2016 from <FQDN>

[root@edd870f7e3ca ~]# /etc/init.d/mysqld status
mysqld (pid 101) is running…
[root@edd870f7e3ca ~]# /etc/init.d/httpd status
httpd (pid 7) is running…
[root@edd870f7e3ca ~]# /etc/init.d/sshd status
openssh-daemon (pid 8) is running…
[root@edd870f7e3ca ~]#

 

xend Error: Acquire running lock failed: 256

Recently I encountered this issue in OVMM 3.2.9  while starting a vm with

xm create <vm.cfg path>

The reason behind this found was : the vm was not shutdown properly  and the lock file is still there even if VM is down.

So the places to look at :

/var/log/xen/xend-debug.log
/var/run/ovs-agent/vm-*.lock
Look at the log file and if the lock file is present under /var/run/ovs-agent/ with the id of the vm which is not starting, just delete the lock file and then VM will start successfully.

xend issue : Xend has probably crashed! Invalid or missing HTTP status code.

I recently found  some VMs of one OVS node( from 30+ nodes)  went down and not able to start with this error :

Xend has probably crashed!  Invalid or missing HTTP status code.

There are many reasons behind this. And if you try to restart xend , it will not start.

The first place to look for is :

/var/log/xen/xend-debug.log

This log will say where exactly the issue is.

In my case my  / filesystem was running out of space because one log file consumed almost 8 GB . So I have to delete that file and now xend started successfully.

Python : Inplace update json and maintain proper order

Some time we have to read one existing json property file and  update some values inplace.

If we don’t use proper approach, the update may lead to breaking the json structure in the file.

We have to hook the json objects by using OrderedDict of collection module in python for remembering the proper order.

Here old_value is updated with new_value :

“head”:

{

“name” : “old_value”

}

 

[code language=”python”]

from collections import OrderedDict

propJson = os.path.dirname(os.path.abspath(__file__))+/props.json
if os.path.isfile(propJson):
with open(propJson,r+) as f:
prop = json.load(f, object_hook=OrderedDict)
prop[head][name] = str(new_value)
f.seek(0)
f.write(json.dumps(prop, f, default=str, indent=4))
f.truncate()

[/code]

Using optparse in python

Sometimes we have to create tools that takes input as argument with certain options. We can create such tool with optparse module of python.

Here is a small example of using this.

 

[code language=”python”]

from optparse import OptionParser

parser = OptionParser(usage=’usage: %prog [options] arguments’)

parser.add_option(‘-a’,help="setup/cleanup",action="store", dest="action")
parser.add_option(‘-m’,help="email id",action="store", dest="email")
parser.add_option(‘-i’,help="Input json props",action="store", dest="input")
(options, args) = parser.parse_args()

[/code]

For help, type: ( This will display all the arguments that can be used with their format)

python tool.py -h

Usage: tool.py [options] arguments

Options:
-h, –help show this help message and exit
-a ACTION setup/cleanup
-m EMAIL email id
-i INPUT Input json props

save it in a programme and execute it as :

python tool.py -a setup -i file.json -m pdk@pdk.com

 

Now we can access the above inputs and use them, using below variables inside the programme:

[code language=”python”]

options.input

options.email

options.action

[/code]

 

Send mail using Python’s smtplib module

Python has a built in module to send mail to recipient[s] as to,cc,bcc. Here assumption is that : the smtp is configured in localhost (where the script will run).

[code language=”python”]
import socket
import smtplib
from email.mime.text import MIMEText

def SendMail(file,Email,status):
fp = open(file,’rb’)
msg = MIMEText(fp.read())
fp.close()
to=Email
cc=’def@example.com’
bcc=’123@example.com’
msg[‘Subject’] = ‘MULTINODE SETUP :: ‘+status
msg[‘From’] = ‘abc@example.com’
msg[‘to’] = to
msg[‘cc’] = cc

msg[‘bcc’] = bcc
toaddr=to.split(&quot;,&quot;)+cc.split(&quot;,&quot;)+bcc.split(&quot;,&quot;)
s = smtplib.SMTP(‘localhost’)
s.sendmail(‘something@example.com’,toaddr ,msg.as_string())
s.quit()

file=’/u01/work/tmp/sidtest’
Email=’myname@example.com’
status=’testing’
SendMail(file,Email,status)
[/code]

 

 

 

Knife remove all recipes from the run_list

There is a simple knife command which can be used to remove all recipes from the run_list of all nodes in a environment.

For this you have to create a dummy role like suppose dummy_role.

#knife role create dummy_role

Once you create the dummy role, assign this role to all the nodes in the environment using the below knife command.

 

#knife exec -E ‘nodes.transform(:all) {|n| n.run_list([“role[dummy_role]”])}'”

 

Now this command would remove all the recipes added to the run_list of the nodes in the environment and add dummy_role to the run_list.

We can remove the dummy_role from the run_list of all the nodes and make it empty.

#knife exec -E ‘nodes.find(“role:dummy_role”) {|n|  n.run_list.remove(“role[dummy_role]”); n.save}’

This is helpful in scenarios where you need to remove all the recipes irrespective of the nodes in the environment and start adding fresh.

Chef – Create encrypted data bag and keep secrets

Sometimes we have to deal with global variables like User passwords, database password, API Keys, middleware boot properties in our chef recipes which shouldn’t be exposed outside.

One solution is we have to keep all the secrets in a data bag and encrypt them using a random secret key and later distribute the key to other node where the secrets are accessed.

diagram_01.png

The other solution if using chef-vault which we will cover in a later topics.

First we have to create  a random encryption key:

openssl rand -base64 512 | tr -d ‘\r\n’ > rev_secret_key 

We have to use this secret key now to encrypt the databag item “revpass” in data bag “rev_secret”.

[code language=”bash”]

export EDITOR=vi
knife data bag create  −−secret-file ./rev_secret_key rev_secret revpass

[/code]

This will open the vi editor with JSON data would be:

{
“id”: “revpass”
}

Now add your secrets here in json format which will be:

{
“id”: “revpass”,

“boot_pass”: “bootpassword”,
“db_pass”: “dbpassword”,
}

Save and exit.

Show the encrypted contents of your databag:

knife data bag show rev_secret revpass

Show the decrypted contents of your databag:

knife data bag show −−secret-file=./rev_secret_key rev_secret revpass

For your chef clients to be able to decrypt the databag when needed, just copy over the secret key (replace client-node with your IP/node name):

scp ./rev_secret_key client-node:/etc/chef/encrypted_data_bag_secret

OR

keep it in ~/.chef directory ad update settings in knife.rb file.

encrypted_data_bag_secret “~/.chef/encrypted_data_bag_secret”

Accessing secret in recipe:

OR Mention the secret key in recipe as below.

In your db recipe, add below line to

secret = Chef::EncryptedDataBagItem.load_secret(“/var/chef/cache/cookbooks/revrec-chef/files/default/revrec_secret_key”)

passwords = Chef::EncryptedDataBagItem.load(“rev_secret “, “revpass”)

dbpasswd = passwds[“db_pass”]

Use it inside a resource:

………

oradb_password = “#{dbpasswd}”

……..

Or keep it in a template how ever its suitable.

Note : If you are using password in a template turn of logging by adding this attribute in template resource:

……

sensitive true

…..