Python : Read and Update helm chart

Recently I was working on a release pipeline where the helm chart of 30+ environments need to be updated in git with the new chart versions from Jenkins input.

Here the helm chart was in yaml format and it was a umbrella chart and individual service chart was needed to be updated from Jenkins.

The umbrella chart file looks like this. https://raw.githubusercontent.com/divyaimca/my_projects/main/update_helm_chart/chart.yaml

apiVersion: v2
description: Helm chart to deploy application NG
name:app-main
version: 0.0.1
dependencies:
- name: service-a
  version: 0.1.014bf574
  repository: '@helm-repo'
  tags:
  - application
  enabled: true
- name: service-b
  version: 0.1.014bf575
  repository: '@helm-repo'
  tags:
  - application
  enabled: true
- name: service-c
  version: 0.1.014bf475
  repository: '@helm-repo'
  tags:
  - application
  enabled: true
- name: service-d
  version: 0.1.024bf575
  repository: '@helm-repo'
  tags:
  - application
  enabled: true
- name: service-e
  version: 0.1.014bf559
  repository: '@helm-repo'
  tags:
  - application
  enabled: true

Here you can see there are 5 dependent services and each version needs to be updated from

I used python module pyyaml.

Here is the code that is used in one stage to achieve this task.

https://raw.githubusercontent.com/divyaimca/my_projects/main/update_helm_chart/helmchart_parser.py

The function takes the input as chart.yaml file path and the subchart and versions in keyword arguments format. Refer the full code from the above link.

e.g.

    update_helm_chart("./chart.yaml",service-b="0.2.574",service-d="0.2.585",service-e="0.2.576")

parse expect_output in a variable : TCL Data Structure

Sometimes we have to automate our steps through expect, run some commands in remote machine and capture the output of a command in a variable and use that variable in some other task.

So here is an example, how we can do that. The output from expect is always captured in expect_output(buffer) and we have to parse this to get our expected result.

So first we have store this expect_output(buffer) in a variable and which will have multiple lines along with our expected result.

Now we have to split that variable with "\n" as delimiter , which will create an array with all the lines in it.

Again from that array we can use indexing to extract the result from a certain position.

Here is one example.

[code lang=’bash’]
#!/usr/bin/expect
set password somepass
set cmd “ls -Art /var/lib/docker/path_to_files/ | tail -n 1”

spawn ssh root@10.59.1.150
set prompt “#|%|>|\\\$ $”
expect {
“(yes/no)” {send “yes\r”;exp_continue}
“password: ” {send “$password\r”;exp_continue}
-re $prompt
}
send “$cmd\r”
expect “# ”

set outcome [split $expect_out(buffer) “\n”]
set filename [lindex $outcome 1]

expect eof
puts “##########################”
puts $filename
puts “##########################”
[/code]

python : trace line number with exception

In python we raise exception to catch if anything goes wrong, but sometime with try block we have multiple lines of code and we are not able to track which line exactly throwing the exception.

There is way to catch the line number from which the exception is coming.

try this way:

[code language=”python”]

try:

line1

line2

except Exception as E:

print(‘Error on line {}’.format(sys.exc_info()[-1].tb_lineno), type(E).__name__, E)

[/code]

Now for any exception, you will get the line number for that also.

pexpect alternative in python for remote connection

We generally use python pexpect module to connect system remotely with ssh and execute our tasks. But sometimes pexpect module is not found to be installed in remote systems which create problems. And this problem can be solved with the python select module with poll.

Here is the sample code that can be used.

https://github.com/kumarprd/pexpect-alternate

SNMP Poller tool to monitor any thing on network

Few years ago, I had created one SNMP Poller tool using perl and snmp utilites that can poll OID informations from any network devices, which is kind of passive monitoring mechanism.

Thought to make it Opensource under GNU GPL.

The details of the utility with its usage can be found here, if anyone is interested to use it.

https://github.com/kumarprd/snmp-poller

Python : Inplace update json and maintain proper order

Some time we have to read one existing json property file and update some values inplace.

If we don’t use proper approach, the update may lead to breaking the json structure in the file.

We have to hook the json objects by using OrderedDict of collection module in python for remembering the proper order.

Here old_value is updated with new_value :

“head”:

{

“name” : “old_value”

}

[code language=”python”]

from collections import OrderedDict

propJson = os.path.dirname(os.path.abspath(__file__))+/props.json
if os.path.isfile(propJson):
with open(propJson,r+) as f:
prop = json.load(f, object_hook=OrderedDict)
prop[head][name] = str(new_value)
f.seek(0)
f.write(json.dumps(prop, f, default=str, indent=4))
f.truncate()

[/code]

Send mail using Python’s smtplib module

Python has a built in module to send mail to recipient[s] as to,cc,bcc. Here assumption is that : the smtp is configured in localhost (where the script will run).

[code language=”python”]
import socket
import smtplib
from email.mime.text import MIMEText

def SendMail(file,Email,status):
fp = open(file,’rb’)
msg = MIMEText(fp.read())
fp.close()
to=Email
cc=’def@example.com’
bcc=’123@example.com’
msg[‘Subject’] = ‘MULTINODE SETUP :: ‘+status
msg[‘From’] = ‘abc@example.com’
msg[‘to’] = to
msg[‘cc’] = cc

msg[‘bcc’] = bcc
toaddr=to.split(",")+cc.split(",")+bcc.split(",")
s = smtplib.SMTP(‘localhost’)
s.sendmail(‘something@example.com’,toaddr ,msg.as_string())
s.quit()

file=’/u01/work/tmp/sidtest’
Email=’myname@example.com’
status=’testing’
SendMail(file,Email,status)
[/code]

Chef – Create encrypted data bag and keep secrets

Sometimes we have to deal with global variables like User passwords, database password, API Keys, middleware boot properties in our chef recipes which shouldn’t be exposed outside.

One solution is we have to keep all the secrets in a data bag and encrypt them using a random secret key and later distribute the key to other node where the secrets are accessed.

The other solution if using chef-vault which we will cover in a later topics.

First we have to create a random encryption key:

openssl rand -base64 512 | tr -d ‘\r\n’ > rev_secret_key

We have to use this secret key now to encrypt the databag item “revpass” in data bag “rev_secret”.

[code language=”bash”]

export EDITOR=vi
knife data bag create −−secret-file ./rev_secret_key rev_secret revpass

[/code]

This will open the vi editor with JSON data would be:

{
“id”: “revpass”
}

Now add your secrets here in json format which will be:

{
“id”: “revpass”,

“boot_pass”: “bootpassword”,
“db_pass”: “dbpassword”,
}

Save and exit.

Show the encrypted contents of your databag:

knife data bag show rev_secret revpass

Show the decrypted contents of your databag:

knife data bag show −−secret-file=./rev_secret_key rev_secret revpass

For your chef clients to be able to decrypt the databag when needed, just copy over the secret key (replace client-node with your IP/node name):

scp ./rev_secret_key client-node:/etc/chef/encrypted_data_bag_secret

keep it in ~/.chef directory ad update settings in knife.rb file.

encrypted_data_bag_secret “~/.chef/encrypted_data_bag_secret”

Accessing secret in recipe:

OR Mention the secret key in recipe as below.

In your db recipe, add below line to

secret = Chef::EncryptedDataBagItem.load_secret(“/var/chef/cache/cookbooks/revrec-chef/files/default/revrec_secret_key”)

passwords = Chef::EncryptedDataBagItem.load(“rev_secret “, “revpass”)

dbpasswd = passwds[“db_pass”]

Use it inside a resource:

………

oradb_password = “#{dbpasswd}”

……..

Or keep it in a template how ever its suitable.

Note : If you are using password in a template turn of logging by adding this attribute in template resource:

……

sensitive true

…..

Writing chef Library

In many cases we have to reuse same code again and again in our recipes. So to reduce this we can write our own library module and reuse it’s methods whenever required. This can help us use our own custom methods.

Like in my previous post we have to access the environment variables in many recipes. So we can use the same code as our library to create a module (lets say ProdEnvSetup)

Create a library file under library directory of cook-book.(cook-book/library/envsetup.rb) and add below code.

[code language=”ruby”]module ProdEnvSetup
def setupenv()
hash1 = {}
File.open(“/u01/data/wor/app/conf/conf.prop”) do |fp|
fp.each do |line|
key, value = line.chomp.split(“=”,2)
hash1[key] = value
end
end

hash1.each do |key,value|
skey = “#{key.to_s}”.gsub(/\s|”|’/, ”)
svalue = “#{value.to_s}”.gsub(/\s|”|’/, ”)
ENV[skey] = svalue
end
end
end
end[/code]

2. Using library inside recipe ( 1st way )

So if the same environment variables are required inside a recipe, we can directly include the module inside our recipe.

Use below code inside begining of recipe:

[code language=”ruby”]class Chef::Recipe
include ProdEnvSetup
end
setupenv() [/code]

2. Using library ( 2nd way )

We can add additional encapsulation to our module by keeping it inside a recipe and include that recipe in every recipe, wherever its required.

create a recipe setupenv.rb and add above codes in 1st way(above)

and include it in other recipes like this:

[code language=”ruby”]include “prod-multiode-cookbook:setupenv”

[/code]

Change tab width in vim

By default, inside vim the tab width is 8 spaces, we can reduce it using below steps.

Create a .vimrc file in ~(home dir) and add below content. Save it and now try.

filetype plugin indent on
" show existing tab with 4 spaces width
set tabstop=4
" when indenting with '>', use 4 spaces width
set shiftwidth=4
" On pressing tab, insert 4 spaces
set expandtab

FOSS -THE FUTURE IS HERE

The open source is dominating the IT market as the alternative of every type of proprietary s/w .So why not use your freedom !!

SCRIPTING

parse expect_output in a variable : TCL Data Structure

Like this:

python : trace line number with exception

Like this:

pexpect alternative in python for remote connection

Like this:

SNMP Poller tool to monitor any thing on network

Like this:

Python : Inplace update json and maintain proper order

Like this:

Send mail using Python’s smtplib module

Like this:

Change tab width in vim

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this: