Using Tcpdump to dump and read network traffic

Another Quick FYI tip.

There are many network analyzer/reader utilities available on both Linux and Windows platform. There is of-course Wireshark, the most preferred GUI network protocol analyzer, but I prefer tcpdump as it is very easy to use.

Lets get started with some examples then. Say if you want to capture all multicast data (from all interfaces), to a file, you run:

$> tcpdump -n "multicast" -vvv -w VideoStreamData.pcap

where:

• -w flag writes the captured traffic to the filepath specified. 
• -n flag avoid DNS look ups to convert host addresses to names. 

Or if you want to capture any packets with a specific destination IP say 172.68.1.1 and destination port 80 or 8080.

$> tcpdump -n "dst host 172.68.1.1 and (dst port 80 or dst port 8080)"

After you have captured network data in a pcap file with the tcpdump command, you can read the data packets in ASCII character sets with the command:

$> tcpdump  -A -r httpServerLogonMessages.pcap 

It comes in handy for reading http/web page traffic.

Or if you want to read the messages both HEX and ASCII along with header data.

$> tcpdump  -x -r MessengerCommunication_11July.pcap 

This comes in handy when you want to convert between different host and network byte order (or the other way round).

Read more about Big Endian to Little Endian conversion and vice-versa here. 

Troubleshooting Packet Drops in SolarFlare Onload 10G PCI Card

If you see lots of packet drops in your onload accelerated application even after going the troubleshooting discussion we did over here, you still see drops and are you are nowhere, you can investigate your application design/OS context scheduling and how the application is reading consuming the data packets. The potential reasons for packet drops could be that,
a) another task interrupting the threads reading the traffic or
b) You have run out of packet buffers because the socket receive buffers aren’t being emptied because read()/recv() isn’t called often enough.

It would be a good idea to monitor how many context switches the application/threads are experience because if this suddenly increases when you have a problem it would indicate another thread is competing for processing time on that core.

You can check this using the following command:

# cat /proc/<pid>/status | grep ctxt_switches
voluntary_ctxt_switches:        58
nonvoluntary_ctxt_switches:     1

Replace “<pid>” with the process ID for your application and the “voluntary” count is the number of times the application blocked and another thread was allowed to run. The “nonvoluntary” count is the number of times the thread was stopped from running by the kernel so something else could run in preference.

You can check the individual threads for the process by monitoring the ‘status’ files in the underlying ‘task’ directory:

# grep ctxt_switches /proc/<pid>/task/*/status
/proc/<pid>/task/<task-n1>/status:voluntary_ctxt_switches:    630
/proc/<pid>/task/<task-n1>/status:nonvoluntary_ctxt_switches: 32
/proc/<pid>/task/<task-n2>/status:voluntary_ctxt_switches:    2
/proc/<pid>/task/<task-n2>/status:nonvoluntary_ctxt_switches: 0
/proc/<pid>/task/<task-n3>/status:voluntary_ctxt_switches:    1
/proc/<pid>/task/<task-n3>/status:nonvoluntary_ctxt_switches: 0

Creating tunnels over ssh - Connecting to remote machine ports over internet

Sometimes when I am working from home, I need my application, which generates data, to be able to send it(data) from my laptop to some machine in the office. For eg. Parsing XML files and storing it to a DB on the office LAN, or talking to web-service deployed in our staging environment, etc.

You can use a VPN to connect to your office LAN. Or you can use this simple tunneling solution that comes with ssh and get going.
From one terminal connect to your office machine (paragOfcPc).

parag@paragpc:~# ssh paragOfcPc -L 10000:192.168.42.43:12871
Last login: Tue May 16 14:41:01 2016 from 172.168.XX.YY
[root@paragOfcPc ~]# ssh 192.168.42.43
Last login: Tue May 16 08:08:51 2016 from 172.168.XX.YY
[root@SQL43 ~]# nc -lkv 12871
Connection from 10.1.1.51 port 12871 [tcp/*] accepted

This forwards the local port 10000 to port 12871 on some server on my office LAN: 192.168.42.43, which is different from my office pc : paragOfcPc. The nice thing about using ssh is I get the added benefit of being on a secure encrypted connection.

From another terminal you can test it with :

parag@paragpc:~# cat hello.txt
This is a demo message for the connection
Send more messages to the remote server
parag@paragpc:~# cat hello.txt | nc localhost 10000
parag@paragpc:~#

After you execute the second command you should now see the content of the file hello.txt in nc -lkv output.

In case you want to send data only to your office pc (in my case paragOfcPc), you can substitute 192.168.42.43 with localhost while creating the tunnel with -L parameter

Jetty Web Server Giving java.util.zip.ZipException: invalid entry size Exception

Today I ran into a bizarre error on Jetty WebServer. I have a few web-sites and services hosted on a Jetty server version 9.3.6.v20151106 running on a Centos 6.4.
I made some changes in a web service code and deployed the binary to my Jetty webapps directory, as soon as I started my Jetty server, it started showing me errors :

[parag@centosvm:/services/jetty-distribution-9]java -jar start.jar 
2016-05-01 17:41:00.380:INFO::main: Logging initialized @625ms
2016-05-01 17:41:00.493:WARN:oejs.HomeBaseWarning:main: This instance of Jetty is not running from a separate {jetty.base} directory, this is not recommended.  See documentation at http://www.eclipse.org/jetty/documentation/current/startup.html
2016-05-01 17:41:00.911:INFO:oejs.Server:main: jetty-9
2016-05-01 17:41:00.959:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///home/parag/jetty/webapps/] at interval 1
2016-05-01 17:41:01.598:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@6093dd95{/staticpages,file:///home/parag/jetty/webapps/staticpages/,AVAILABLE}{/staticpages}
2016-05-01 17:41:08.060:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@32464a14{/wallsoft-x-web,file:///home/parag/jetty/webapps/reactJSSite/,STARTING}{/abcd}
java.util.zip.ZipException: invalid entry size (expected 11658 but got 11664 bytes)
 at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:384)
 at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
 at java.util.jar.JarInputStream.read(JarInputStream.java:207)
 at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:140)
 at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:118)
 at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:142)
 at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:179)
 at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:939)
 at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:851)
 at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:163)
 at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:548)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
 at java.lang.Thread.run(Thread.java:745)

I did a quick search on the internet and stackoverflow ofcourse for finding the problem cause and resolution and did not come across anything concrete. 

However in this particular answer, people had reported it could be due to a corrupt maven repository jar. How in the world it could have happened I do not know. No recent maven packages were downloaded (just incremental changes to code, no new package dependency). 

And I was able to run zip/unzip commands normally, as well as on .war files. Anyhow after spending close to 30 min, I decided to give the answer a shot and deleted my .m2/repository folder, rebuild my project, copied the binaries again and viola it worked :) 

[parag@centosvm:/home/parag] rm -rf .m2/repository/

Disable/Ban certain users to connect from ssh

Generally after turning on the sshd service on our machines, we fail to authorize or rather restrict a set of users only to access (white-listing) the machine. The following setting in sshd_config helps you to restrict access to a select few users. The settings will take effect after restarting the sshd demon on your machine if already running.

[parag@paragcentos:/home/parag]sudo cat /etc/ssh/sshd_config  | grep -i Allow
...
##Allow the following users only to remote via ssh
AllowUsers root parag
...
...
[parag@paragcentos:/home/parag]

Also please read this post on more tips on how to secure your ssh server. 

Checking Bad Blocks in Hard Disk

If you need to see/check how your hard disk in doing, the first thing you look for bad sectors and read/write speed of your hard disk.
Below are a few commands which can be run on your linux machine to check for any errors/speed related or read-write issues

[parag@centos: /home/parag]# fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd42ad42a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64         587     4194304   82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3             587       19458   151582720   83  Linux


[parag@centos: /home/parag]# badblocks -v -s /dev/sda1
Checking blocks 0 to 511999
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found.

[parag@centos: /home/parag]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   532 MB in  2.00 seconds = 266.05 MB/sec
 Timing buffered disk reads: 248 MB in  3.02 seconds =  82.11 MB/sec
[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   556 MB in  2.00 seconds = 277.54 MB/sec
 Timing buffered disk reads: 236 MB in  3.00 seconds =  78.61 MB/sec

[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   524 MB in  2.01 seconds = 261.21 MB/sec
 Timing buffered disk reads: 174 MB in  3.03 seconds =  57.49 MB/sec

[parag@centos: /home/parag]

Troubleshooting Packet drops in Solarflare 10G network card

If you experience packet drops while listening to UDP packet broadcast then the following extracts from the onload_stackdump command utility can help you
identify the Onload stack under memory pressure.

[root@paragpc ~]  onload_stackdump lots

You can monitor the rate of these drops for the Onload stack using the above command, for example in one of my machine the stackdump reported:

[root@paragpc ~] onload_stackdump lots | grep memory_pressure_drops

memory_pressure_drops: 81381

If this is increasing during bursts of data then increasing the value of EF_MAX_PACKETS to 65000 or higher could resolve the issue by providing more buffering but if this value is increasing steadily then it would only delay the point at which drops happen. You can set this by setting “EF_MAX_PACKETS=65000” in the environment or prefix the ‘onload’ command, e.g.:

[root@paragpc ~] EF_MAX_PACKETS=65000 onload <app_cmd_line>

Note that this is further restricted for TX and RX in order to avoid potential deadlocks where all the packet buffers are used on the RX and can’t be freed because data needs to be sent to do this. By default there is a 75% limit on each so if you use the maximum for RX it would leave 25% for TX.

Read more about the how to troubleshoot and configure onload here.

Bash Script to Create users in a DB

Many a times when running unit test on a clean build, I want to skip the user creation part, or may be reset the user details/profile for a particular user(s) in Development environment.
Of course there are unit test suite written to automate the flow, but sometimes when you are in the flow or designing the test cases itself, you need something to do the little rub-scrub-and-setup the little tiny jobs.
Here is my small script which deletes two very specific users from the my_dev_db db and creates them again. The details of the user are in file UserProd.txt which is tab delimited.  Enjoy.

[parag@paragcentosvm ~]# cat populateUsers.sh
#!/bin/sh

mysql -u parag_dev -pSOMEPASSWORD -e "DELETE FROM User_Table where UserId in (110,111);" -D my_dev_db;
mysql -u parag_dev -pSOMEPASSWORD -e "LOAD DATA LOCAL INFILE '/home/Parag/UserProd.txt' INTO TABLE User_Table;" -D my_dev_db;

Zipping file and folders in Linux CentOS Fedora

To zip a file (or multiple files) in a zip (compressed format for distribution, saving disk space or emailing as attachment), you can use the zip utility present in allmost all the linux distros.

To check if you have zip installed on your machine type

# zip -h

if it is not present, it can be installed via yum Hence the task

# yum install zip
# yum install unzip

Or if you prefer apt over yum then run the following commands:

# apt-get install zip
# apt-get install unzip

Next you can zip files by running the following command. No need to add .zip file extension as it will be added by the zip utility:

$ zip myZip_FileName *

To compress a directory (and all sub-directories within it) run:

$  zip -r myZipped_FolderName *

To decompress (or deflate or unzip) the zip simply run the following command from your command prompt:

$ unzip  zippedFile.zip

To simply view the contents of a zip and not actually unzip it run:

$ unzip -tq zippedFile.zip

To unzip the contents of a zip in a new directory (for avoiding clutter and confusion) run:

$ unzip zippedFile.zip  -d /Folder_to_Unpack_into

Debugging with GDB: Examining Call Stack and Call Stack Frames

If you are debugging a programming with gdb and want to see the call stack, and the call stack is really big (spanning the whole page), it makes sense to see only the top (innermost) 5-6 stack frames rather than the entire stack.

(gdb) backtrace
#0  0x00007ffff7bc2620 in thirdPartyFFTransform () from ./liblzo2.so
#1  0x00007ffff7108e43 in useMD5Checksum (karlCoef=0x7ffff2cd0ff0, ffCoef=0x7ffff2cd10c0) at ThirdParty/HardFFTrf.c:99
#2  0x00007ffff71091d5 in FastCheckSum (gamma=3426, delta=0x7ffff730e4c8 <acc_no> "x^2+5y  ", capillaryRise=0x7ffff2cd1120, 
    pCheckSumList=0x7ffff2cd10c0) at ThirdParty/HardFFTrf.c:152
#3  0x00007ffff7107349 in compute_fourier (gamma=40127, delta=0x7ffff2cd1120, CoefA=0x7ffff2cd123a) at ThirdPartyFF.c:1125
#4  0x0000000000491c6e in example::MathCalculator::FastFourierHelper::getCoefficient(FastFourier::Coefficient const*) ()
#5  0x0000000000492a44 in example::MathCalculator::FastFourierHelper::createFFDescriptor(std::vector<example::MathCalculator::ComputeIterationN*, std::allocator<example::MathCalculator::ComputeIterationN*> > const&, int) ()
#6  0x0000000000478767 in example::MathCalculator::FTransform::FTransform(example::MathCalculator::MessageQueue<example::MathCalculator::QMessg, 1024ul>*, example::MathCalculator::MessageQueue<example::MathCalculator::QueueMessage, 1024ul>*, std::vector<example::MathCalculator::ComputeIterationN*, std::allocator<example::MathCalculator::ComputeIterationN*> >&, example::MathCalculator::Accumulator*) ()
#7  0x0000000000474f23 in example::MathCalculator::MathEngine::createJobFactory(FastFourier::EquationDescription*) ()
#8  0x0000000000476856 in example::MathCalculator::MathEngine::processQueue() ()
#9  0x00000000004770b1 in example::MathCalculator::MathEngine::checkAssignments() ()
#10 0x00000000004a7d2e in example::MathCalculator::MathComputeEngine::Calculate() ()
#11 0x000000000046072c in example::MathCalculator::MMain::runMathLibThread(int) ()
#12 0x00007ffff7787b53 in thread_proxy () from ./libboost_thread.so.1.48.0
#13 0x00007ffff5ecadc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff63de73d in clone () from /lib64/libc.so.6
(gdb) 
(gdb) 
(gdb) bt 4
#0  0x00007ffff7bc2620 in thirdPartyFFTransform () from ./liblzo2.so
#1  0x00007ffff7108e43 in useMD5Checksum (karlCoef=0x7ffff2cd0ff0, ffCoef=0x7ffff2cd10c0) at ThirdParty/HardFFTrf.c:99
#2  0x00007ffff71091d5 in FastCheckSum (gamma=3426, delta=0x7ffff730e4c8 <acc_no> "x^2+5y  ", capillaryRise=0x7ffff2cd1120, 
    pCheckSumList=0x7ffff2cd10c0) at ThirdParty/HardFFTrf.c:152
#3  0x00007ffff7107349 in compute_fourier (gamma=40127, delta=0x7ffff2cd1120, CoefA=0x7ffff2cd123a) at ThirdPartyFF.c:1125
#4  0x0000000000491c6e in example::MathCalculator::FastFourierHelper::getCoefficient(FastFourier::Coefficient const*) ()
(gdb) 
(gdb) 
(gdb) bt -4
#11 0x000000000046072c in example::MathCalculator::MMain::runMathLibThread(int) ()
#12 0x00007ffff7787b53 in thread_proxy () from ./libboost_thread.so.1.48.0
#13 0x00007ffff5ecadc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff63de73d in clone () from /lib64/libc.so.6

Stack frames

Consider the following stack dump

(gdb) backtrace
#0  0x0000000000401449 in autoPtrExample (rm=3) at MultiUseHeader.hpp:81
#1  0x0000000000401254 in registerQuickExitHandlers (k=3, comment="called from main") at MultiUseHeader.hpp:66
#2  0x00000000004015aa in registeringExitHandlers () at samAutoPtr.cpp:22
#3  0x0000000000401610 in main () at samAutoPtr.cpp:30
(gdb) backtrace -2
#2  0x00000000004015aa in registeringExitHandlers () at samAutoPtr.cpp:22
#3  0x0000000000401610 in main () at samAutoPtr.cpp:30
(gdb) bt 2
#0  0x0000000000401449 in autoPtrExample (rm=3) at MultiUseHeader.hpp:81
#1  0x0000000000401254 in registerQuickExitHandlers (k=3, comment="called from main") at MultiUseHeader.hpp:66
(More stack frames follow...)

You can select your frame of interest using the frame command.

(gdb) frame 0
#0  0x0000000000401449 in autoPtrExample (rm=3) at MultiUseHeader.hpp:81
81  cout << "Printing Rollno: p " << m->roll << " q " << n->roll << "\n"; // <- Seg fault
(gdb) frame 1
#1  0x0000000000401254 in registerQuickExitHandlers (k=3, comment="called from main") at MultiUseHeader.hpp:66
66  autoPtrExample(3);
(gdb) 

Similarly to traverse one frame in either direction one at a time use the up/down command

(gdb) up
#2  0x00000000004015aa in registeringExitHandlers () at samAutoPtr.cpp:22
22  registerQuickExitHandlers(3, "called from main");
(gdb) up
#3  0x0000000000401610 in main () at samAutoPtr.cpp:30
30  registeringExitHandlers();
(gdb) up
Initial frame selected; you cannot go up.
(gdb) down
#2  0x00000000004015aa in registeringExitHandlers () at samAutoPtr.cpp:22
22  registerQuickExitHandlers(3, "called from main");
(gdb) down
#1  0x0000000000401254 in registerQuickExitHandlers (k=3, comment="called from main") at MultiUseHeader.hpp:66
66  autoPtrExample(3);
(gdb) down
#0  0x0000000000401449 in autoPtrExample (rm=3) at MultiUseHeader.hpp:81
81  cout << "Printing Rollno: p " << m->roll << " q " << n->roll << "\n"; // <- Seg fault
(gdb) down
Bottom (innermost) frame selected; you cannot go down.

To see the arguments passed in a particular frame use the info args command once the frame of interest is selected

(gdb) frame
#0  0x0000000000401449 in autoPtrExample (rm=3) at MultiUseHeader.hpp:81
81  cout << "Printing Rollno: p " << m->roll << " q " << n->roll << "\n"; // <- Seg fault
(gdb) up
#1  0x0000000000401254 in registerQuickExitHandlers (k=3, comment="called from main") at MultiUseHeader.hpp:66
66  autoPtrExample(3);
(gdb) info args
k = 3
comment = "called from main"
(gdb) 

To see the local arguments of the selected frame use the info locals command to print the variables:

(gdb) info locals
filename = "./StateWisePopulationRecord.csv"
(gdb)