Jenkins slave as a service in windows to start automatically

In Jenkins many time we have to add windows machine as slave, where we need the agent to be up and running as windows service.

There are many ways to do it, but I struggled to find the correct configuration steps. I used windows resource toolkit to make it work and adding the steps here.

Configuration Steps:

(A) Adding the windows slave in Jenkins server :

1. Add the agent in Jenkins Master with Launch Method : Launch Via Java Web Start

Screen Shot 2019-09-25 at 4.07.06 PM

(B) Creating the Service in windows Server for starting the slave agent: (In Windows Server 2016)

1. Download and install the Java 8.
2. Down and Install Windows Resource Kit Tools (https://www.microsoft.com/en-us/download/details.aspx?id=17657)
3. Create a blank service called “Jenkins Slave” by running the following from a command prompt

“C:\Program Files (x86)\Windows Resource Kits\Tools\instsrv.exe” “Jenkins Slave” “C:\Program Files (x86)\Windows Resource Kits\Tools\srvany.exe”

4. Open Registry Editor and go to below location :

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Jenkins Slave

Now Follow the below steps carefully.

  1. Create a string value “Description”
  2. Populate it with “Jenkins Continuous Integration Slave”

Screen Shot 2019-09-25 at 2.13.33 PM

  1. Create a new key “Parameters”
  2. Under “Parameters” create a new string value “Application”
  3. Populate it with the full path to java.exe, something like “C:\sapjvm_8\bin\java.exe”
  4. Under “Parameters” Create a new string value “AppParameters”
  5. Populate it with

“-jar E:\jenkins_agent\agent.jar -jnlpUrl http://m1-hostname.lab.saas.company.corp:8080/computer/hostame-of-machone/slave-agent.jnlp -secret <secret-name> -workDir E:\jenkinsWorkSpace”

  1. The slave.jar should point to the correct location
  2. The Jenkins master machine name should be correct
  3. The new Jenkins slave machine should be correct
  4. Make sure you use the secret for this machine that you copied from the master when adding the new node

Screen Shot 2019-09-25 at 2.24.06 PM

Open the Services application from Control Panel – Administrative Tools, find the “Jenkins Slave” service, right click on the service and go to “Properties”.

  1. Go to the “Recovery” tab and change “First failure” and “Second failure” to “Restart the Service” – occasionally we found it wouldn’t start up first time out
  2. Go to the “Log On” tab and set an account and password- we found that using an account with local admin rights on the slave machine worked best but this is probably unnecessary
  3. Go to the “General” tab and change the “Startup type” to “Automatic” – make sure the service
  4. starts up when you restart the slave
  5. Click the “OK” button
  6. Now start the serviceScreen Shot 2019-09-25 at 2.27.33 PM

6. The service will run by default during startup of the windows machine.

7. Now Verify the agent is up and running in Jenkins Web Page.

Screen Shot 2019-09-25 at 4.03.00 PM

parse expect_output in a variable : TCL Data Structure

Sometimes we have to automate our steps through expect, run some commands in remote machine and capture the output of a command in a variable and use that variable in some other task.

So here is an example, how we can do that. The output from expect is always captured in expect_output(buffer) and we have to parse this to get our expected result.

So first we have store this expect_output(buffer) in a variable and which will have multiple lines along with our expected result.

Now we have to split that variable with "\n" as delimiter , which will create an array with all the lines in it.

Again from that array we can use indexing to extract the result from a certain position.

Here is one example.

[code lang=’bash’]
#!/usr/bin/expect
set password somepass
set cmd “ls -Art /var/lib/docker/path_to_files/ | tail -n 1”

spawn ssh root@10.59.1.150
set prompt “#|%|>|\\\$ $”
expect {
“(yes/no)” {send “yes\r”;exp_continue}
“password: ” {send “$password\r”;exp_continue}
-re $prompt
}
send “$cmd\r”
expect “# ”

set outcome [split $expect_out(buffer) “\n”]
set filename [lindex $outcome 1]

expect eof
puts “##########################”
puts $filename
puts “##########################”
[/code]

python : trace line number with exception

In python we raise exception to catch if anything goes wrong, but sometime with try block we have multiple lines of code and we are not able to track which line exactly throwing the exception.

There is way to catch the line number from which the exception is coming.

try this way:

[code language=”python”]

try:

line1

line2

except Exception as E:

print(‘Error on line {}’.format(sys.exc_info()[-1].tb_lineno), type(E).__name__, E)

[/code]

Now for any exception, you will get the line number for that also.

ORA-12519: TNS:no appropriate service handler found

The number of database processes that can run in Oracle DB is set to 150 as default which can be viewed from v$resource_limit.

select * from v$resource_limit where resource_name = 'processes';


RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_ALLOCATION LIMIT_VALUE
------------- ------------------- --------------- ------------------ -----------
processes     148                 149              150                150

We face this issue – ORA-12519 ,  randomly when the number of DB processes utilized by our application is more that 150. So we need to increase the number of DB processes inorder to avoid the issue. Use below steps to incresae the limit.

[code language=”sql”]

sqlplus / as sysdba
sqlplus>alter system set processes=500 scope=spfile;
sqlplus>shutdown immediate;
sqlplus>startup

[/code]

Now query the view to see the new limits in effect.

select * from v$resource_limit where resource_name = 'processes';


RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION INITIAL_ALLOCATION LIMIT_VALUE
------------- ------------------- --------------- ------------------ -----------
processes      279                  285             500                500

 

 

Accessing Host from Docker Container

Sometime we need to access the services that are running in the host machine to be accessible from the docker container.  e.g. In on of my project, we needed to connect to the oracle db (port 1521) from inside the container within code.

The default behaviour of containers are, they cant access the host network directly unless the firewall of the host machine allows the network interface of docker to ACCEPT the packets.

So the docker container will communicate with the host machine using the gateway ip. First find the gateway ip inside the container.

Run below command inside the container to get the gateway ip and observer I am not able to connect to port 1521.

[code language=”bash”]

# nc -vz dockerhost 1521

dockerhost [172.18.0.1] 1521 (?) : Connection timed out

# ip route | awk ‘/^default via /{print $3}’

172.18.0.1

[/code]

Next  task is to get the interface name of the docker network which is binded with the container.  Most of the cases its docker0.

But it can also be customized, so check ifconfig output which matches the inet addr of the container gateway.

[code language=”bash”]

# ifconfig

br-4e83b57c54cf Link encap:Ethernet  HWaddr 02:42:AF:CD:B5:DA

inet addr:172.18.0.1  Bcast:0.0.0.0  Mask:255.255.0.0

# ip addr show br-4e83b57c54cf

10: br-4e83b57c54cf: mtu 1500 qdisc noqueue state UP

link/ether 02:42:af:cd:b5:da brd ff:ff:ff:ff:ff:ff

inet 172.18.0.1/16 scope global br-4e83b57c54cf

valid_lft forever preferred_lft forever

[/code]

Here the interface name is : br-4e83b57c54cf

Now add a iptables rule in Linux host:

[code language=”bash”]

iptables -A INPUT -i br-4e83b57c54cf -j ACCEPT

[/code]

OR with firewalld

[code language=”bash”]
# firewall-cmd –permanent –zone=trusted –change-interface=br-294e81e5ac31
# firewall-cmd –reload

[/code]

Now try to access the host port from container.

[code language=”bash”]

# nc -vz dockerhost 1521

dockerhost [172.18.0.1] 1521 (?) open

[/code]

There are other ways also available on internet , but I found none of them working.

 

 

 

 

 

 

 

 

 

 

 

My MongoDB cheatsheet

Few years ago, I was working on one mongoDB project.Its a NoSQL database , easy to learn and its pure JSON based.

I learned it myself and had created a cheatsheet for it, so I can easily recall it anytime I want. It can help anyone who is interested in learning it quickly.

Thought to share the sheet. It can be found here :

https://github.com/kumarprd/mongo-guide/blob/master/my-mongo_guide.txt

 

 

 

Python : Inplace update json and maintain proper order

Some time we have to read one existing json property file and  update some values inplace.

If we don’t use proper approach, the update may lead to breaking the json structure in the file.

We have to hook the json objects by using OrderedDict of collection module in python for remembering the proper order.

Here old_value is updated with new_value :

“head”:

{

“name” : “old_value”

}

 

[code language=”python”]

from collections import OrderedDict

propJson = os.path.dirname(os.path.abspath(__file__))+/props.json
if os.path.isfile(propJson):
with open(propJson,r+) as f:
prop = json.load(f, object_hook=OrderedDict)
prop[head][name] = str(new_value)
f.seek(0)
f.write(json.dumps(prop, f, default=str, indent=4))
f.truncate()

[/code]

Using optparse in python

Sometimes we have to create tools that takes input as argument with certain options. We can create such tool with optparse module of python.

Here is a small example of using this.

 

[code language=”python”]

from optparse import OptionParser

parser = OptionParser(usage=’usage: %prog [options] arguments’)

parser.add_option(‘-a’,help="setup/cleanup",action="store", dest="action")
parser.add_option(‘-m’,help="email id",action="store", dest="email")
parser.add_option(‘-i’,help="Input json props",action="store", dest="input")
(options, args) = parser.parse_args()

[/code]

For help, type: ( This will display all the arguments that can be used with their format)

python tool.py -h

Usage: tool.py [options] arguments

Options:
-h, –help show this help message and exit
-a ACTION setup/cleanup
-m EMAIL email id
-i INPUT Input json props

save it in a programme and execute it as :

python tool.py -a setup -i file.json -m pdk@pdk.com

 

Now we can access the above inputs and use them, using below variables inside the programme:

[code language=”python”]

options.input

options.email

options.action

[/code]

 

Send mail using Python’s smtplib module

Python has a built in module to send mail to recipient[s] as to,cc,bcc. Here assumption is that : the smtp is configured in localhost (where the script will run).

[code language=”python”]
import socket
import smtplib
from email.mime.text import MIMEText

def SendMail(file,Email,status):
fp = open(file,’rb’)
msg = MIMEText(fp.read())
fp.close()
to=Email
cc=’def@example.com’
bcc=’123@example.com’
msg[‘Subject’] = ‘MULTINODE SETUP :: ‘+status
msg[‘From’] = ‘abc@example.com’
msg[‘to’] = to
msg[‘cc’] = cc

msg[‘bcc’] = bcc
toaddr=to.split(&quot;,&quot;)+cc.split(&quot;,&quot;)+bcc.split(&quot;,&quot;)
s = smtplib.SMTP(‘localhost’)
s.sendmail(‘something@example.com’,toaddr ,msg.as_string())
s.quit()

file=’/u01/work/tmp/sidtest’
Email=’myname@example.com’
status=’testing’
SendMail(file,Email,status)
[/code]

 

 

 

Knife remove all recipes from the run_list

There is a simple knife command which can be used to remove all recipes from the run_list of all nodes in a environment.

For this you have to create a dummy role like suppose dummy_role.

#knife role create dummy_role

Once you create the dummy role, assign this role to all the nodes in the environment using the below knife command.

 

#knife exec -E ‘nodes.transform(:all) {|n| n.run_list([“role[dummy_role]”])}'”

 

Now this command would remove all the recipes added to the run_list of the nodes in the environment and add dummy_role to the run_list.

We can remove the dummy_role from the run_list of all the nodes and make it empty.

#knife exec -E ‘nodes.find(“role:dummy_role”) {|n|  n.run_list.remove(“role[dummy_role]”); n.save}’

This is helpful in scenarios where you need to remove all the recipes irrespective of the nodes in the environment and start adding fresh.