Using optparse in python

Sometimes we have to create tools that takes input as argument with certain options. We can create such tool with optparse module of python.

Here is a small example of using this.

 

[code language=”python”]

from optparse import OptionParser

parser = OptionParser(usage=’usage: %prog [options] arguments’)

parser.add_option(‘-a’,help="setup/cleanup",action="store", dest="action")
parser.add_option(‘-m’,help="email id",action="store", dest="email")
parser.add_option(‘-i’,help="Input json props",action="store", dest="input")
(options, args) = parser.parse_args()

[/code]

For help, type: ( This will display all the arguments that can be used with their format)

python tool.py -h

Usage: tool.py [options] arguments

Options:
-h, –help show this help message and exit
-a ACTION setup/cleanup
-m EMAIL email id
-i INPUT Input json props

save it in a programme and execute it as :

python tool.py -a setup -i file.json -m pdk@pdk.com

 

Now we can access the above inputs and use them, using below variables inside the programme:

[code language=”python”]

options.input

options.email

options.action

[/code]

 

Send mail using Python’s smtplib module

Python has a built in module to send mail to recipient[s] as to,cc,bcc. Here assumption is that : the smtp is configured in localhost (where the script will run).

[code language=”python”]
import socket
import smtplib
from email.mime.text import MIMEText

def SendMail(file,Email,status):
fp = open(file,’rb’)
msg = MIMEText(fp.read())
fp.close()
to=Email
cc=’def@example.com’
bcc=’123@example.com’
msg[‘Subject’] = ‘MULTINODE SETUP :: ‘+status
msg[‘From’] = ‘abc@example.com’
msg[‘to’] = to
msg[‘cc’] = cc

msg[‘bcc’] = bcc
toaddr=to.split(",")+cc.split(",")+bcc.split(",")
s = smtplib.SMTP(‘localhost’)
s.sendmail(‘something@example.com’,toaddr ,msg.as_string())
s.quit()

file=’/u01/work/tmp/sidtest’
Email=’myname@example.com’
status=’testing’
SendMail(file,Email,status)
[/code]

 

 

 

Writing chef Library

In many cases we have to reuse same code again and again in our recipes. So to reduce this we can write our own library module and reuse it’s methods whenever required. This can help us use our own custom methods.

Like in my previous post we have to access the environment variables in many recipes. So we can use the same code as our library to create a module (lets say ProdEnvSetup)

 

  1. Create a library file under library directory of cook-book.(cook-book/library/envsetup.rb) and add below code.

 

[code language=”ruby”]module ProdEnvSetup
def setupenv()
hash1 = {}
File.open(“/u01/data/wor/app/conf/conf.prop”) do |fp|
fp.each do |line|
key, value = line.chomp.split(“=”,2)
hash1[key] = value
end
end

hash1.each do |key,value|
skey = “#{key.to_s}”.gsub(/\s|”|’/, ”)
svalue = “#{value.to_s}”.gsub(/\s|”|’/, ”)
ENV[skey] = svalue
end
end
end
end[/code]

2. Using library inside recipe ( 1st way )

So if the same environment variables are required inside a recipe, we can directly include the module inside our recipe.

Use below code inside begining of recipe:

[code language=”ruby”]class Chef::Recipe
include ProdEnvSetup
end
setupenv() [/code]

2. Using library ( 2nd way )

We can add additional encapsulation to our module by keeping it inside a recipe and include that recipe in every recipe, wherever its required.

create a recipe setupenv.rb and add above codes in 1st way(above)

and include it in other recipes like this:

[code language=”ruby”]include “prod-multiode-cookbook:setupenv”

[/code]

 


			

Chef/ruby way – Read a file and expose as environment variable

Many time we have to read a property file in in which the variable and value are comma separated and we have to set those in our environment variable to execute certain recipes.

 

e.g. property file (/u01/data/wor/app/conf/conf.prop)

ops_home = ‘/u01/data/work/app/ops-home-1.2.30’

node_instance = ‘/u01/data/work/app’

 

So here we have to read the file , create a hash and then save the LHS as key and RHS as value. Then we are good to expose them as environment variable.

Note : This is the approach I used, there may be other solution available.

Here the properties are = separated. It can be any separator.

This is a reusable function and can be called where ever required.

[code language=”ruby”]def setupenv()
hash1 = {}
File.open("/u01/data/wor/app/conf/conf.prop") do |fp|
fp.each do |line|
key, value = line.chomp.split("=",2)
hash1[key] = value
end
end</pre>
hash1.each do |key,value|
skey = "#{key.to_s}".gsub(/\s|"|’/, ”)
svalue = "#{value.to_s}".gsub(/\s|"|’/, ”)
ENV[skey] = svalue
end
end
end
end[/code]

 

Here setupenv( ) can be called anywhere the ENV variables are required.

Note : Here gsub(/\s|”|’/, ”) is used to trim  leading and trailing space, single quote, double quote of the key and value.

 

 

Forking vs Threading

What is Fork/Forking:

Fork is nothing but a new process that looks exactly like the old or the parent process but still it is a different process with different process ID and having  it’s own memory. Parent process creates a separate address space for child. Both parent and child process possess the same code segment, but execute independently from each other.

The simplest example of forking is when you run a command on shell in unix/linux. Each time a user issues a command, the shell forks a child process and the task is done.

When a fork system call is issued, a copy of all the pages corresponding to the parent process is created, loaded into a separate memory location by the OS for the child process, but in certain cases, this is not needed. Like in ‘exec’ system calls, there is not need to copy the parent process pages, as execv replaces the address space of the parent process itself.

Few things to note about forking are:

  • The child process will be having it’s own unique process ID.
  • The child process shall have it’s own copy of parent’s file descriptor.
  • File locks set by parent process shall not be inherited by child process.
  • Any semaphores that are open in the parent process shall also be open in the child process.
  • Child process shall have it’s own copy of message queue descriptors of the parents.
  • Child will have it’s own address space and memory.

Fork is universally accepted than thread because of the following reasons:

  • Development is much easier on fork based implementations.
  • Fork based code a more maintainable.
  • Forking is much safer and more secure because each forked process runs in its own virtual address space. If one process crashes or has a buffer overrun, it does not affect any other process at all.
  • Threads code is much harder to debug than fork.
  • Fork are more portable than threads.
  • Forking is faster than threading on single cpu as there are no locking over-heads or context switching.

Some of the applications in which forking is used are: telnetd(freebsd), vsftpd, proftpd, Apache13, Apache2, thttpd, PostgreSQL.

Pitfalls in Fork:

  • In fork, every new process should have it’s own memory/address space, hence a longer startup and stopping time.
  • If you fork, you have two independent processes which need to talk to each other in some way. This inter-process communication is really costly.
  • When the parent exits before the forked child, you will get a zombie process. That is all much easier with a thread. You can end, suspend and resume threads from the parent easily. And if your parent exits suddenly the thread will be ended automatically.
  • In-sufficient storage space could lead the fork system to fail.

What are Threads/Threading:

Threads are Light Weight Processes (LWPs). Traditionally, a thread is just a CPU (and some other minimal state) state with the process containing the remains (data, stack, I/O, signals). Threads require less overhead than “forking” or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process. While most effective on a multiprocessor system where the process flow can be scheduled to run on another processor thus gaining speed through parallel or distributed processing, gains are also found on uniprocessor systems which exploit latency in I/O and other system functions which may halt process execution.

Threads in the same process share:

  • Process instructions
  • Most data
  • open files (descriptors)
  • signals and signal handlers
  • current working directory
  • User and group id

Each thread has a unique:

  • Thread ID
  • set of registers, stack pointer
  • stack for local variables, return addresses
  • signal mask
  • priority
  • Return value: errno

Few things to note about threading are:

  • Thread are most effective on multi-processor or multi-core systems.
  • For thread – only one process/thread table and one scheduler is needed.
  • All threads within a process share the same address space.
  • A thread does not maintain a list of created threads, nor does it know the thread that created it.
  • Threads reduce overhead by sharing fundamental parts.
  • Threads are more effective in memory management because they uses the same memory block of the parent instead of creating new.

Pitfalls in threads:

  • Race conditions: The big loss with threads is that there is no natural protection from having multiple threads working on the same data at the same time without knowing that others are messing with it. This is called race condition. While the code may appear on the screen in the order you wish the code to execute, threads are scheduled by the operating system and are executed at random. It cannot be assumed that threads are executed in the order they are created. They may also execute at different speeds. When threads are executing (racing to complete) they may give unexpected results (race condition). Mutexes and joins must be utilized to achieve a predictable execution order and outcome.
  • Thread safe code: The threaded routines must call functions which are “thread safe”. This means that there are no static or global variables which other threads may clobber or read assuming single threaded operation. If static or global variables are used then mutexes must be applied or the functions must be re-written to avoid the use of these variables. In C, local variables are dynamically allocated on the stack. Therefore, any function that does not use static data or other shared resources is thread-safe. Thread-unsafe functions may be used by only one thread at a time in a program and the uniqueness of the thread must be ensured. Many non-reentrant functions return a pointer to static data. This can be avoided by returning dynamically allocated data or using caller-provided storage. An example of a non-thread safe function is strtok which is also not re-entrant. The “thread safe” version is the re-entrant version strtok_r.

Advantages in threads:

  • Threads share the same memory space hence sharing data between them is really faster means inter-process communication (IPC) is real fast.
  • If properly designed and implemented threads give you more speed because there aint any process level context switching in a multi threaded application.
  • Threads are really fast to start and terminate.

Some of the applications in which threading is used are: MySQL, Firebird, Apache2, MySQL 323

Read and write json in python

In todays  world json is one of the easiest method of communication to exchange data.

Its used everywhere Cloud APIs, Databases, Configuration files, REST Call.

So below are the steps how to read and write back to a json file using python.

Lets say: test.json contains:

{
“pro”: “Cloud”,
“fname”: “Priyadarshee”,
“yob”: 1988
}

To load and read  this file in python:

>>> import json
>>> config = json.loads(open(“test.json”).read())
>>> config
{u’pro’: u’Cloud’, u’yob’: 1988, u’fname’: u’Priyadarshee’}

>>> config[“pro”]
u’Cloud’

 

Now adding new key-value pair as well as updating existing pair:

>>> config [“lname”] = “Kumar”
>>> config [“pro”] = “DevOps”
>>>
>>> config
{‘lname’: ‘Kumar’, u’pro’: ‘DevOps’, u’yob’: 1988, u’fname’: u’Priyadarshee’}
>>> json.dump(config,open(“test.json”,”w”),indent=4)
>>> config[“pro”]
u’DevOps’

Here indent = 4, makes the indentation with 4 spaces. It can be adjusted as required.

New test.josn will look like:

cat test.json

{
“lname”: “Kumar”,
“pro”: “DevOps”,
“yob”: 1988,
“fname”: “Priyadarshee”
}

We can use this lines in a python script to make a complete code to use as required.

 

 

Pasting codes or any content in VI editor without being commented.

By default VI editor is in normal mode. So we have to enable the paste mode and to activate this before copying the code type below:

:set paste

Now go to insert mode by pressing i and paste your code.

After pasting the codes in VI editor, get out of the paste mode hit ESC and type:

:set nopaste

which will take you back to the normal mode.