Using Tcpdump to dump and read network traffic

Another Quick FYI tip.

There are many network analyzer/reader utilities available on both Linux and Windows platform. There is of-course Wireshark, the most preferred GUI network protocol analyzer, but I prefer tcpdump as it is very easy to use.

Lets get started with some examples then. Say if you want to capture all multicast data (from all interfaces), to a file, you run:

$> tcpdump -n "multicast" -vvv -w VideoStreamData.pcap

where:

• -w flag writes the captured traffic to the filepath specified. 
• -n flag avoid DNS look ups to convert host addresses to names. 

Or if you want to capture any packets with a specific destination IP say 172.68.1.1 and destination port 80 or 8080.

$> tcpdump -n "dst host 172.68.1.1 and (dst port 80 or dst port 8080)"

After you have captured network data in a pcap file with the tcpdump command, you can read the data packets in ASCII character sets with the command:

$> tcpdump  -A -r httpServerLogonMessages.pcap 

It comes in handy for reading http/web page traffic.

Or if you want to read the messages both HEX and ASCII along with header data.

$> tcpdump  -x -r MessengerCommunication_11July.pcap 

This comes in handy when you want to convert between different host and network byte order (or the other way round).

Read more about Big Endian to Little Endian conversion and vice-versa here. 

Adding routes on a windows machine

Just a small FYI article.

We have multiple P2P lease lines in our office, connecting our different offices within the city, apart from multiple internet connections.
While trying to access these system its prefered to have them accessible over the leased line network.

All of our networks merge on the single LAN. So we need to add routes on our system to tell them, to direct which traffic through which router (as in the specific router connected to the leased line of a office).

For eg the LAN segment 192.66.222.0/8 (to our office in OfficeA1) is accessible via router 192.168.22.10 (which is the meeting point of one of the P2P from here to OfficeA1), run the following command from an elevated command prompt.

C:\>route add -p 192.66.222.0 mask 255.255.255.0 192.168.22.10

Similarly the LAN segment 192.122.72.0 is accessible via router ip 192.168.22.20

C:\>route add -p 192.122.72.0 mask 255.255.255.0 192.168.56.20

Troubleshooting Packet Drops in SolarFlare Onload 10G PCI Card

If you see lots of packet drops in your onload accelerated application even after going the troubleshooting discussion we did over here, you still see drops and are you are nowhere, you can investigate your application design/OS context scheduling and how the application is reading consuming the data packets. The potential reasons for packet drops could be that,
a) another task interrupting the threads reading the traffic or
b) You have run out of packet buffers because the socket receive buffers aren’t being emptied because read()/recv() isn’t called often enough.

It would be a good idea to monitor how many context switches the application/threads are experience because if this suddenly increases when you have a problem it would indicate another thread is competing for processing time on that core.

You can check this using the following command:

# cat /proc/<pid>/status | grep ctxt_switches
voluntary_ctxt_switches:        58
nonvoluntary_ctxt_switches:     1

Replace “<pid>” with the process ID for your application and the “voluntary” count is the number of times the application blocked and another thread was allowed to run. The “nonvoluntary” count is the number of times the thread was stopped from running by the kernel so something else could run in preference.

You can check the individual threads for the process by monitoring the ‘status’ files in the underlying ‘task’ directory:

# grep ctxt_switches /proc/<pid>/task/*/status
/proc/<pid>/task/<task-n1>/status:voluntary_ctxt_switches:    630
/proc/<pid>/task/<task-n1>/status:nonvoluntary_ctxt_switches: 32
/proc/<pid>/task/<task-n2>/status:voluntary_ctxt_switches:    2
/proc/<pid>/task/<task-n2>/status:nonvoluntary_ctxt_switches: 0
/proc/<pid>/task/<task-n3>/status:voluntary_ctxt_switches:    1
/proc/<pid>/task/<task-n3>/status:nonvoluntary_ctxt_switches: 0

Using Docker Containers to build C++ Projects

With everyone moving to the docker bandwagon, why should you feel left behind. If you have your project hosted on BitBucket.org, make sure that you do make use of the pipeline feature to ensure that the project/product is always in the sane state.

Here is a sample bitbucket-pipelines.yml file to get you going with building C++ projects.

# This is a sample build configuration for C++ – Make.
# Check our guides at https://confluence.atlassian.com/x/5Q4SMw for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: gcc:6.1

pipelines:
  default:
    - step:
        script: # Modify the commands below to build your repository.
          - make
          - make test

Well this is not going to be very useful if you use other libraries like boost, or qt in your project. Because the basic default image that BitBucket uses has tools like gcc, python, maven, npm, java etc(Read more about the image here). It does not back with it other third party libraries (like boost).

What do you do? You have the following options at hand for specifying the image:

1. the default image bitbucket provides(which is not very useful. Read here about the image), 
2. your own image from docker hub (public/private) 
3. specify am image hosted in private registry. (for detailed instruction read here)

Well for our little POC project where we were testing the feasibility of making use of Pipeline features we decided to use ilssim/cmake-boost-qt5  image.

CAUTION: Always check the DockerFile of the image when using someone else's image for security issues. Its always better to build your own image and host it on Docker hub and use it.

Coming back to my example, our POC project had reference to boost threading libraries, so here is my sample bitbucket-pipelines.yml file:

# This is a sample build configuration for C++ – Make.
# Check our guides at https://confluence.atlassian.com/x/5Q4SMw for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: ilssim/cmake-boost-qt5

pipelines:
  default:
    - step:
        script:
          - echo "This script runs on all branches that don't have any specific pipeline assigned in 'branches'."
  branches:
    master:
      - step:
          script:
          - make
          - make test
    feature/*:
      - step:
          script:
          - cd orchestrationService
          - cmake
          - make

You can read more about branch workflows here.

Block UDP Traffic via IPTables

While testing an application to verify all code flow paths, one of the scenarios demanded that the application handle dropped UDP packet stream. Now we had a recorded UDP stream at hand which we could play at will.

To simulate UDP packet drop, we added a rule in IPTables to drop/block all UDP packets destined for our UDP destination port 10222. The following command does it for you. 

To Block all udp traffic destined for port 10222

[root@paragpc ~]# iptables -A INPUT -p udp -i eth1 --dport 10222 -j DROP

where:

• -A is for add/Append iptable rule
• -p is for protocol
• -i  = --in-interface eth1  (there is similarly -o =--out-interface
• -j = JUMP

Refer the man page of iptables for more details. 

However don't forget to remove the rule once your test case is over.

Remove iptable rule dropping udp traffic destined for port 10222 [-D is for Delete iptable rule]

[root@paragpc ~]# iptables -D INPUT -p udp -i eth1 --dport 10222 -j DROP

However if you wish to permanently save the rule run the following command(source).

service iptables save

[Solution] Password less ssh connection not working after upgrading Git Bash in windows

After I had updated git bash with version 2.11.1 on windows my .ssh/config files was overwritten (wiped clean) along with the public-private keys entries for ssh passwordless authentication.

Good thing I had a backup of my .ssh/config and id_pub keys with me.

Simply populating the .ssh/config file, resolved the ssh hostname but still it asked my for my password. Then I added the following entries for IdentityFile at the end of .ssh-config file

Host *

    ServerAliveInterval 300

    ServerAliveCountMax 2

    Compression yes

    CompressionLevel 9

    GSSAPIAuthentication no

    ForwardAgent yes  

    IdentityFile ~/.ssh/id_dsa

    IdentityFile ~/.ssh/vm_private_key

where  id_dsa and vm_private_key were the private ssh keys for ssh authentication.
Again simply running ssh-add <Path to key did not work. It kept complaining about :   

parag@paragpc MINGW64 ~
$ ssh-add -l
Could not open a connection to your authentication agent.

So I ran the following command (source) :

parag@paragpc MINGW64 ~
$ eval $(ssh-agent -s); ssh-add /c/Users/parag/.ssh/id_dsa

[Resolved] 'AmqpClient::AmqpLibraryException' in SimpleAmqpClient

I have an application using SimpleAmqpClient, which was working fine in our dev environment. when I moved it to a staging environment, it started failing with the following exception.

terminate called after throwing an instance of 'AmqpClient::AmqpLibraryException'
 what(): a socket error occurred

Well nothing had changed (don't we hear this all the time from devs, we changed nothing in that RabbitMQ/AMQP module), and the error looked cryptic with no details.
Well it turned out that, a (new) login account had been created for connecting to the RabbitMQ server, and it hadn't been enabled to connect from a remote host. Turned it on via the configuration file.
And viola, things started working :) . Read more about RabbitMQ configuration here. 

Creating tunnels over ssh - Connecting to remote machine ports over internet

Sometimes when I am working from home, I need my application, which generates data, to be able to send it(data) from my laptop to some machine in the office. For eg. Parsing XML files and storing it to a DB on the office LAN, or talking to web-service deployed in our staging environment, etc.

You can use a VPN to connect to your office LAN. Or you can use this simple tunneling solution that comes with ssh and get going.
From one terminal connect to your office machine (paragOfcPc).

parag@paragpc:~# ssh paragOfcPc -L 10000:192.168.42.43:12871
Last login: Tue May 16 14:41:01 2016 from 172.168.XX.YY
[root@paragOfcPc ~]# ssh 192.168.42.43
Last login: Tue May 16 08:08:51 2016 from 172.168.XX.YY
[root@SQL43 ~]# nc -lkv 12871
Connection from 10.1.1.51 port 12871 [tcp/*] accepted

This forwards the local port 10000 to port 12871 on some server on my office LAN: 192.168.42.43, which is different from my office pc : paragOfcPc. The nice thing about using ssh is I get the added benefit of being on a secure encrypted connection.

From another terminal you can test it with :

parag@paragpc:~# cat hello.txt
This is a demo message for the connection
Send more messages to the remote server
parag@paragpc:~# cat hello.txt | nc localhost 10000
parag@paragpc:~#

After you execute the second command you should now see the content of the file hello.txt in nc -lkv output.

In case you want to send data only to your office pc (in my case paragOfcPc), you can substitute 192.168.42.43 with localhost while creating the tunnel with -L parameter

C++ Identifying threads with names

In multi-threaded applications, many a times during debugging or logging, we wish to identify the threads by name[functionality ]. Simply printing the executing method name in logs does not help if the functions are common between threads doing different tasks. Also you may want to see the currently running threads from either top or ps in your application.

Here is sample program to tag your threads with proper names. Use prctl defined in <sys/prctl.h> , to get/set thread names. Keep in mind that the name can be up to 16 bytes long only. 

#include <stdio.h>
#include <pthread.h>
#include <stdio.h>
#include <iostream>
#include <unistd.h>
#include <stdlib.h>
#include <sys/prctl.h>

//compile the program with : -lpthread

using namespace std;

void * f1(void *) {

 try{
  prctl(PR_SET_NAME,"Thread_name_f1",0,0,0);
  printf("f1 : Starting sleep\n");
  // ..
  // Code goes here
  // utils();
  // g();
  // ..
 }
 catch(std::string &s){
  char thread_name[17]={0};
  prctl(PR_GET_NAME,thread_name,0,0,0);
  std::cout << "Thread:" << thread_name <<"crashed. Reason:" << s << std::endl;
 }
}

void * f2(void *) {

 try{
      prctl(PR_SET_NAME,"Thread_name_f2",0,0,0);
  printf("f2 : Starting sleep\n");
  // ..
  // Code goes here
  // a();
  // utils();
  // ..
 }
 catch(std::string &s){

  char thread_name[17]={0};
  prctl(PR_GET_NAME,thread_name,0,0,0);
  std::cout << "Thread:" << thread_name <<"crashed. Reason:" << s << std::endl;
 }
}

int main() {
 pthread_t  f1_thread, f2_thread;
 pthread_create(&f1_thread, NULL, f1, NULL);
 pthread_create(&f2_thread, NULL, f2, NULL);

 printf("Main : Starting sleep\n");
 sleep(50);
 pthread_join( f1_thread, NULL);
 pthread_join( f2_thread, NULL);
 printf("Main : Done sleep\n");
 return 0;

}

Removing deleted remote branches from local repo

Sometimes we have a POC(Proof of concept) branches, which are forked off the main/feature branches. After a while when POC is complete, we/the dev no longer uses it.
Now over time the no of such POC branches increase and the repo doesnt look good.
So during our periodic maintenance and cleaning of GIT repo, we delete all such POC branches.

How to sync your local branch list.
Simple:

[parag@paragpc: ~/ProjectCodeName] git fetch -p

 x [deleted]   (none)  ->origin/POC-webCache

 x [deleted]   (none)  ->origin/POC-CDNChange

[parag@paragpc: ~/ProjectCodeName] 

where -p = --prune
                     After fetching, remove any remote-tracking branches which no longer exist on the remote. 


Similarly, if you are have created a local branch and committed some changes while the remote tracking branch is not present at origin, issue:

[parag@paragpc: ~/GitProject] git branch -d <the_dirty_poc_branch>

where -d = --delete
        Delete a branch. The branch must be fully merged in its upstream branch, or in HEAD if no upstream was set with --track or --set-upstream

PS: Before running any delete related operation on branch its better to double check and understand what you are doing!.

Linux: Flush SWAP memory pages to RAM

On my dev machine, after I had started some services like Samba, MySQL, Jetty and executing some batch SQL queries which churned out a big database, I realized that my system had become slow and sluggish. I immediately checked the RAM usage via the System Task Monitor that comes with CentOS. And I noticed that though my RAM had substantial free space, SWAP was in action.

 If you ever run into such situation you can run the following commands to free you SWAP and move the active pages from SWAP to RAM. Before running the following commands ensure that you have substantial free space in your RAM i.e. free RAM size > SWAP memory size in use.

[root@paragcentosvm ~]# sudo swapoff -a
[root@paragcentosvm ~]# sudo swapon -a

The first command, swapoff, will take some time around 2-3 minutes (or more depending on the amount of SWAP memory in use, which needs to be freed). What is does is disables the SWAP partition for paging and swapping. When the -a flag is given, swapping is disabled on all known swap devices and files (as found in /proc/swaps or /etc/fstab)

The next swapon commands enables the SWAP partition for paging and swapping.

Linux : Yum rpmdb: damaged header Problem

While trying to get the latest chrome working on my machine, the yum process crashed. It just wouldn't work. first I tried to clean the yum cache.

[root@paragcentosvm ~]# yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: base epel extras google-chrome updates
Cleaning up Everything
Cleaning up list of fastest mirrors
[root@paragcentosvm ~]# 

but it did not seem to work. It would just show me the following error:

[parag@paragcentosvm:/home/parag/Downloads]sudo yum update firefox
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
epel/metalink                                                                                                                                                | 4.8 kB     00:00     
 * base: centos.mirror.net.in
 * epel: epel.mirror.net.in
 * extras: centos.mirror.net.in
 * updates: centos.mirror.net.in
base                                                                                                                                                         | 3.7 kB     00:00     
http://centos.mirror.net.in/centos/6.8/os/x86_64/repodata/ad3a307dfd95da4d7a7aad136162378d18ead7271010822806902dfa3edb55f2-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 18 - "transfer closed with 4902072 bytes remaining to read"
Trying other mirror.
base/primary_db                                                                                                                                              | 4.7 MB     00:02     
epel                                                                                                                                                         | 4.3 kB     00:00     
epel/primary_db                                                                                                                                              | 5.8 MB     00:06     
http://centos.mirror.net.in/centos/6.8/extras/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with 3435 bytes remaining to read"
Trying other mirror.
extras                                                                                                                                                       | 3.4 kB     00:00     
extras/primary_db                                                                                                                                            |  36 kB     00:00     
google-chrome                                                                                                                                                |  951 B     00:00     
google-chrome/primary                                                                                                                                        | 1.8 kB     00:00     
google-chrome                                                                                                                                                                   3/3
http://centos.mirror.net.in/centos/6.8/updates/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with 3456 bytes remaining to read"
Trying other mirror.
updates                                                                                                                                                      | 3.4 kB     00:00     
updates/primary_db                                                                                                                                           | 732 kB     00:00     
error: rpmdb: damaged header #1703 retrieved -- skipping.
Resolving Dependencies
--> Running transaction check
---> Package firefox.x86_64 0:38.1.0-1.el6.centos will be updated
---> Package firefox.x86_64 0:45.2.0-1.el6.centos will be an update
--> Processing Dependency: nss >= 3.21.0 for package: firefox-45.2.0-1.el6.centos.x86_64
--> Processing Dependency: nspr >= 4.11.0 for package: firefox-45.2.0-1.el6.centos.x86_64
--> Running transaction check
---> Package nspr.x86_64 0:4.10.8-1.el6_6 will be updated
--> Processing Dependency: nspr = 4.10.8-1.el6_6 for package: nspr-devel-4.10.8-1.el6_6.x86_64
---> Package nspr.x86_64 0:4.11.0-1.el6 will be an update
---> Package nss.x86_64 0:3.19.1-3.el6_6 will be updated
--> Processing Dependency: nss = 3.19.1-3.el6_6 for package: nss-sysinit-3.19.1-3.el6_6.x86_64
--> Processing Dependency: nss = 3.19.1-3.el6_6 for package: nss-devel-3.19.1-3.el6_6.x86_64
--> Processing Dependency: nss(x86-64) = 3.19.1-3.el6_6 for package: nss-tools-3.19.1-3.el6_6.x86_64
---> Package nss.x86_64 0:3.21.0-8.el6 will be an update
--> Processing Dependency: nss-util >= 3.21.0 for package: nss-3.21.0-8.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.21)(64bit) for package: nss-3.21.0-8.el6.x86_64
--> Running transaction check
---> Package nspr-devel.x86_64 0:4.10.8-1.el6_6 will be updated
---> Package nspr-devel.x86_64 0:4.11.0-1.el6 will be an update
---> Package nss-devel.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-devel.x86_64 0:3.21.0-8.el6 will be an update
--> Processing Dependency: pkgconfig(nss-util) >= 3.21.0 for package: nss-devel-3.21.0-8.el6.x86_64
---> Package nss-sysinit.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-sysinit.x86_64 0:3.21.0-8.el6 will be an update
---> Package nss-tools.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-tools.x86_64 0:3.21.0-8.el6 will be an update
---> Package nss-util.x86_64 0:3.19.1-1.el6_6 will be updated
---> Package nss-util.x86_64 0:3.21.0-2.el6 will be an update
--> Running transaction check
---> Package nss-util-devel.x86_64 0:3.19.1-1.el6_6 will be updated
---> Package nss-util-devel.x86_64 0:3.21.0-2.el6 will be an update
error: rpmdb: damaged header #1703 retrieved -- skipping.
error: rpmdb: damaged header #1703 retrieved -- skipping.
error: rpmdb: damaged header #1703 retrieved -- skipping.

To fix the issue I ran the following commands, to repair the corrupted yum  metadata files/db.

[parag@paragcentosvm:/home/parag]sudo rm -f /var/lib/rpm/__db*
[parag@paragcentosvm:/home/parag]sudo rpm --rebuilddb
[parag@paragcentosvm:/home/parag]

And everything just became normal again.

Linux : How to get Network Driver Details

If you are looking for a command to query the network card driver details such as name and version, run the ethtool command  with either -i or --driver option followed by the network interface name.

[parag@clara ~]# ethtool -i eth1
driver: via-rhine
version: 2.4.9
firmware-version: 
bus-info: 0000:03:00.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[parag@clara ~]# ethtool -i eth0
driver: tg345
version: 13.8
firmware-version: nb vs.49
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
[parag@clara ~]# 
[parag@paragcentosvm:/home/parag]ethtool --driver eth0
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Jetty Web Server Giving java.util.zip.ZipException: invalid entry size Exception

Today I ran into a bizarre error on Jetty WebServer. I have a few web-sites and services hosted on a Jetty server version 9.3.6.v20151106 running on a Centos 6.4.
I made some changes in a web service code and deployed the binary to my Jetty webapps directory, as soon as I started my Jetty server, it started showing me errors :

[parag@centosvm:/services/jetty-distribution-9]java -jar start.jar 
2016-05-01 17:41:00.380:INFO::main: Logging initialized @625ms
2016-05-01 17:41:00.493:WARN:oejs.HomeBaseWarning:main: This instance of Jetty is not running from a separate {jetty.base} directory, this is not recommended.  See documentation at http://www.eclipse.org/jetty/documentation/current/startup.html
2016-05-01 17:41:00.911:INFO:oejs.Server:main: jetty-9
2016-05-01 17:41:00.959:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///home/parag/jetty/webapps/] at interval 1
2016-05-01 17:41:01.598:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@6093dd95{/staticpages,file:///home/parag/jetty/webapps/staticpages/,AVAILABLE}{/staticpages}
2016-05-01 17:41:08.060:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@32464a14{/wallsoft-x-web,file:///home/parag/jetty/webapps/reactJSSite/,STARTING}{/abcd}
java.util.zip.ZipException: invalid entry size (expected 11658 but got 11664 bytes)
 at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:384)
 at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
 at java.util.jar.JarInputStream.read(JarInputStream.java:207)
 at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:140)
 at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:118)
 at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:142)
 at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:179)
 at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:939)
 at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:851)
 at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:163)
 at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:548)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
 at java.lang.Thread.run(Thread.java:745)

I did a quick search on the internet and stackoverflow ofcourse for finding the problem cause and resolution and did not come across anything concrete. 

However in this particular answer, people had reported it could be due to a corrupt maven repository jar. How in the world it could have happened I do not know. No recent maven packages were downloaded (just incremental changes to code, no new package dependency). 

And I was able to run zip/unzip commands normally, as well as on .war files. Anyhow after spending close to 30 min, I decided to give the answer a shot and deleted my .m2/repository folder, rebuild my project, copied the binaries again and viola it worked :) 

[parag@centosvm:/home/parag] rm -rf .m2/repository/

Disable/Ban certain users to connect from ssh

Generally after turning on the sshd service on our machines, we fail to authorize or rather restrict a set of users only to access (white-listing) the machine. The following setting in sshd_config helps you to restrict access to a select few users. The settings will take effect after restarting the sshd demon on your machine if already running.

[parag@paragcentos:/home/parag]sudo cat /etc/ssh/sshd_config  | grep -i Allow
...
##Allow the following users only to remote via ssh
AllowUsers root parag
...
...
[parag@paragcentos:/home/parag]

Also please read this post on more tips on how to secure your ssh server. 

Checking Bad Blocks in Hard Disk

If you need to see/check how your hard disk in doing, the first thing you look for bad sectors and read/write speed of your hard disk.
Below are a few commands which can be run on your linux machine to check for any errors/speed related or read-write issues

[parag@centos: /home/parag]# fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd42ad42a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64         587     4194304   82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3             587       19458   151582720   83  Linux


[parag@centos: /home/parag]# badblocks -v -s /dev/sda1
Checking blocks 0 to 511999
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found.

[parag@centos: /home/parag]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   532 MB in  2.00 seconds = 266.05 MB/sec
 Timing buffered disk reads: 248 MB in  3.02 seconds =  82.11 MB/sec
[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   556 MB in  2.00 seconds = 277.54 MB/sec
 Timing buffered disk reads: 236 MB in  3.00 seconds =  78.61 MB/sec

[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   524 MB in  2.01 seconds = 261.21 MB/sec
 Timing buffered disk reads: 174 MB in  3.03 seconds =  57.49 MB/sec

[parag@centos: /home/parag]

Adding Missing Data points to Collections in Java

If you are from the .NET world, there are high chances that you have used LINQ in the .NET world. I was working on some JAVA project where there was some data which had missing datapoints.

This data in question was read from a DB, and was a big stored procedure. Making any more changes to it for filling in missing values was out of question, as it required nested loops which is better dealt in your programming language as it makes the program clean and more readable, plus its up the reader/consumer how he wants to prune/treat the data.
So here is a small snippet of code for filling missing data in a collection.

List<Timestamp> completeSeries= Utils.generateTimestampSequence(startTimestamp, endTimestamp);

completeSeries.stream().forEach((w) -> {

    boolean found = interestingDataCollectionWithMissingPoints.stream().

     anyMatch(x-> (x.getTimestamp().equals(w) ));

    if(!found){

     interestingDataCollectionWithMissingPoints.add(new InterestingDataClass(w, 0, 0, 0));

    }

   } );
//Now the interestingDataCollectionWithMissingPoints contains all the missing data points 

Here, I had a certain data collection which was missing some data for a per minute time intervals. And I needed the complete data for all the point in times for the given time period (startTimeStamp and endTimeStamp).
So the little code snippet above does the trick. Java had no such feature prior to 1.5 version. Of course move to newer Java compiler version  to use this nifty feature.

<build>
  <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.5.1</version>
    <configuration>
     <source>1.8</source>
     <target>1.8</target>
    </configuration>
   </plugin>
                 </plugins>
</build>

Automount a disk at machine start up.

Automount a disk at machine start up.

Recently bought an additional hard disk of 1 TB for my home pc running Centos. After you plugin your disk to the machine, the system does not automounts it. It detects the new hardware and assigns it a dev id(deviceId)

To automount the disk at computer start up follow the following steps:

1. Find the UUID of the newly added disk. You can find it by running the following command

1. 
   [root@paragWS ~]# blkid
   
   /dev/sdb1: UUID="dffb1c9c-29e9-4a00-add1-ad08d90c9469" TYPE="xfs"
   
   /dev/sdb2: UUID="SJhaki-0CLf-GDmR-j8LW-e1pL-hcwP-sQVnZJ" TYPE="LVM2_member"
   
   /dev/mapper/centos-root: UUID="cb3625db-6ee0-431f-884e-022037e66e48" TYPE="xfs"
   
   /dev/sda: UUID="c3d18886-59cc-4aba-aff8-c1f5403af506" TYPE="ext3"
   

2. Create an entry in fstab which looks like this:

2. 
   [root@paragWS ~]# cat /etc/fstab
   
   
   
   
   #
   
   # /etc/fstab
   
   # Created by anaconda on Tue Dec  6 00:17:24 2015
   
   #
   
   # Accessible filesystems, by reference, are maintained under '/dev/disk'
   
   # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
   
   #
   
   /dev/mapper/centos-root /                       xfs     defaults        0 0
   
   …
   
               …
   
   
   
   
               # /dev/sda SATA 1TB
   
                  UUID=c3d18886-59cc-4aba-aff8-c1f5403af506 /mnt/VideoData                   ext3     defaults        0 0
   
   
   
   

Default settings used above are equivalent to rw, suid, dev, exec, auto, nouser, async. Read more here

Ping pong test with Ansible

A quick and sweet command to ping your inventory/servers in ansible: 

[parag@centosVM ops-dir]$ ansible all -m ping -i production
PodAServer18 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodAServer93 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodAServer71 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodAServer73 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodBServer151 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
PodBServer48 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}

The above command pings all the server in your production to check for their availability/accessibility/to check whether they are reachable or not. 

In the output above servers : PodAServer18, PodAServer93, PodAServer71, PodAServer73  are accessible and PodBServer151, PodBServer48 are not reachable. 

This can be modified to check for specific sub-section/groups in the server inventory file. 

The following command limits the ping test to the emailServers group defined in the staging inventory file. 

[parag@centosVM ops-dir]$ ansible all -m ping -i staging--limit emailServers
server43 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
Server35 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
Rack3Server45 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
[parag@centosVM ops-dir]

aa

Troubleshooting Packet drops in Solarflare 10G network card

If you experience packet drops while listening to UDP packet broadcast then the following extracts from the onload_stackdump command utility can help you
identify the Onload stack under memory pressure.

[root@paragpc ~]  onload_stackdump lots

You can monitor the rate of these drops for the Onload stack using the above command, for example in one of my machine the stackdump reported:

[root@paragpc ~] onload_stackdump lots | grep memory_pressure_drops

memory_pressure_drops: 81381

If this is increasing during bursts of data then increasing the value of EF_MAX_PACKETS to 65000 or higher could resolve the issue by providing more buffering but if this value is increasing steadily then it would only delay the point at which drops happen. You can set this by setting “EF_MAX_PACKETS=65000” in the environment or prefix the ‘onload’ command, e.g.:

[root@paragpc ~] EF_MAX_PACKETS=65000 onload <app_cmd_line>

Note that this is further restricted for TX and RX in order to avoid potential deadlocks where all the packet buffers are used on the RX and can’t be freed because data needs to be sent to do this. By default there is a 75% limit on each so if you use the maximum for RX it would leave 25% for TX.

Read more about the how to troubleshoot and configure onload here.