Troubleshooting Packet Drops in SolarFlare Onload 10G PCI Card

If you see lots of packet drops in your onload accelerated application even after going the troubleshooting discussion we did over here, you still see drops and are you are nowhere, you can investigate your application design/OS context scheduling and how the application is reading consuming the data packets. The potential reasons for packet drops could be that,
a) another task interrupting the threads reading the traffic or
b) You have run out of packet buffers because the socket receive buffers aren’t being emptied because read()/recv() isn’t called often enough.

It would be a good idea to monitor how many context switches the application/threads are experience because if this suddenly increases when you have a problem it would indicate another thread is competing for processing time on that core.

You can check this using the following command:

# cat /proc/<pid>/status | grep ctxt_switches
voluntary_ctxt_switches:        58
nonvoluntary_ctxt_switches:     1

Replace “<pid>” with the process ID for your application and the “voluntary” count is the number of times the application blocked and another thread was allowed to run. The “nonvoluntary” count is the number of times the thread was stopped from running by the kernel so something else could run in preference.

You can check the individual threads for the process by monitoring the ‘status’ files in the underlying ‘task’ directory:

# grep ctxt_switches /proc/<pid>/task/*/status
/proc/<pid>/task/<task-n1>/status:voluntary_ctxt_switches:    630
/proc/<pid>/task/<task-n1>/status:nonvoluntary_ctxt_switches: 32
/proc/<pid>/task/<task-n2>/status:voluntary_ctxt_switches:    2
/proc/<pid>/task/<task-n2>/status:nonvoluntary_ctxt_switches: 0
/proc/<pid>/task/<task-n3>/status:voluntary_ctxt_switches:    1
/proc/<pid>/task/<task-n3>/status:nonvoluntary_ctxt_switches: 0

Linux : Yum rpmdb: damaged header Problem

While trying to get the latest chrome working on my machine, the yum process crashed. It just wouldn't work. first I tried to clean the yum cache.

[root@paragcentosvm ~]# yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: base epel extras google-chrome updates
Cleaning up Everything
Cleaning up list of fastest mirrors
[root@paragcentosvm ~]# 

but it did not seem to work. It would just show me the following error:

[parag@paragcentosvm:/home/parag/Downloads]sudo yum update firefox
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
epel/metalink                                                                                                                                                | 4.8 kB     00:00     
 * base: centos.mirror.net.in
 * epel: epel.mirror.net.in
 * extras: centos.mirror.net.in
 * updates: centos.mirror.net.in
base                                                                                                                                                         | 3.7 kB     00:00     
http://centos.mirror.net.in/centos/6.8/os/x86_64/repodata/ad3a307dfd95da4d7a7aad136162378d18ead7271010822806902dfa3edb55f2-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 18 - "transfer closed with 4902072 bytes remaining to read"
Trying other mirror.
base/primary_db                                                                                                                                              | 4.7 MB     00:02     
epel                                                                                                                                                         | 4.3 kB     00:00     
epel/primary_db                                                                                                                                              | 5.8 MB     00:06     
http://centos.mirror.net.in/centos/6.8/extras/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with 3435 bytes remaining to read"
Trying other mirror.
extras                                                                                                                                                       | 3.4 kB     00:00     
extras/primary_db                                                                                                                                            |  36 kB     00:00     
google-chrome                                                                                                                                                |  951 B     00:00     
google-chrome/primary                                                                                                                                        | 1.8 kB     00:00     
google-chrome                                                                                                                                                                   3/3
http://centos.mirror.net.in/centos/6.8/updates/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with 3456 bytes remaining to read"
Trying other mirror.
updates                                                                                                                                                      | 3.4 kB     00:00     
updates/primary_db                                                                                                                                           | 732 kB     00:00     
error: rpmdb: damaged header #1703 retrieved -- skipping.
Resolving Dependencies
--> Running transaction check
---> Package firefox.x86_64 0:38.1.0-1.el6.centos will be updated
---> Package firefox.x86_64 0:45.2.0-1.el6.centos will be an update
--> Processing Dependency: nss >= 3.21.0 for package: firefox-45.2.0-1.el6.centos.x86_64
--> Processing Dependency: nspr >= 4.11.0 for package: firefox-45.2.0-1.el6.centos.x86_64
--> Running transaction check
---> Package nspr.x86_64 0:4.10.8-1.el6_6 will be updated
--> Processing Dependency: nspr = 4.10.8-1.el6_6 for package: nspr-devel-4.10.8-1.el6_6.x86_64
---> Package nspr.x86_64 0:4.11.0-1.el6 will be an update
---> Package nss.x86_64 0:3.19.1-3.el6_6 will be updated
--> Processing Dependency: nss = 3.19.1-3.el6_6 for package: nss-sysinit-3.19.1-3.el6_6.x86_64
--> Processing Dependency: nss = 3.19.1-3.el6_6 for package: nss-devel-3.19.1-3.el6_6.x86_64
--> Processing Dependency: nss(x86-64) = 3.19.1-3.el6_6 for package: nss-tools-3.19.1-3.el6_6.x86_64
---> Package nss.x86_64 0:3.21.0-8.el6 will be an update
--> Processing Dependency: nss-util >= 3.21.0 for package: nss-3.21.0-8.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.21)(64bit) for package: nss-3.21.0-8.el6.x86_64
--> Running transaction check
---> Package nspr-devel.x86_64 0:4.10.8-1.el6_6 will be updated
---> Package nspr-devel.x86_64 0:4.11.0-1.el6 will be an update
---> Package nss-devel.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-devel.x86_64 0:3.21.0-8.el6 will be an update
--> Processing Dependency: pkgconfig(nss-util) >= 3.21.0 for package: nss-devel-3.21.0-8.el6.x86_64
---> Package nss-sysinit.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-sysinit.x86_64 0:3.21.0-8.el6 will be an update
---> Package nss-tools.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-tools.x86_64 0:3.21.0-8.el6 will be an update
---> Package nss-util.x86_64 0:3.19.1-1.el6_6 will be updated
---> Package nss-util.x86_64 0:3.21.0-2.el6 will be an update
--> Running transaction check
---> Package nss-util-devel.x86_64 0:3.19.1-1.el6_6 will be updated
---> Package nss-util-devel.x86_64 0:3.21.0-2.el6 will be an update
error: rpmdb: damaged header #1703 retrieved -- skipping.
error: rpmdb: damaged header #1703 retrieved -- skipping.
error: rpmdb: damaged header #1703 retrieved -- skipping.

To fix the issue I ran the following commands, to repair the corrupted yum  metadata files/db.

[parag@paragcentosvm:/home/parag]sudo rm -f /var/lib/rpm/__db*
[parag@paragcentosvm:/home/parag]sudo rpm --rebuilddb
[parag@paragcentosvm:/home/parag]

And everything just became normal again.

Checking Bad Blocks in Hard Disk

If you need to see/check how your hard disk in doing, the first thing you look for bad sectors and read/write speed of your hard disk.
Below are a few commands which can be run on your linux machine to check for any errors/speed related or read-write issues

[parag@centos: /home/parag]# fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd42ad42a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64         587     4194304   82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3             587       19458   151582720   83  Linux


[parag@centos: /home/parag]# badblocks -v -s /dev/sda1
Checking blocks 0 to 511999
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found.

[parag@centos: /home/parag]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   532 MB in  2.00 seconds = 266.05 MB/sec
 Timing buffered disk reads: 248 MB in  3.02 seconds =  82.11 MB/sec
[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   556 MB in  2.00 seconds = 277.54 MB/sec
 Timing buffered disk reads: 236 MB in  3.00 seconds =  78.61 MB/sec

[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   524 MB in  2.01 seconds = 261.21 MB/sec
 Timing buffered disk reads: 174 MB in  3.03 seconds =  57.49 MB/sec

[parag@centos: /home/parag]

Zipping file and folders in Linux CentOS Fedora

To zip a file (or multiple files) in a zip (compressed format for distribution, saving disk space or emailing as attachment), you can use the zip utility present in allmost all the linux distros.

To check if you have zip installed on your machine type

# zip -h

if it is not present, it can be installed via yum Hence the task

# yum install zip
# yum install unzip

Or if you prefer apt over yum then run the following commands:

# apt-get install zip
# apt-get install unzip

Next you can zip files by running the following command. No need to add .zip file extension as it will be added by the zip utility:

$ zip myZip_FileName *

To compress a directory (and all sub-directories within it) run:

$  zip -r myZipped_FolderName *

To decompress (or deflate or unzip) the zip simply run the following command from your command prompt:

$ unzip  zippedFile.zip

To simply view the contents of a zip and not actually unzip it run:

$ unzip -tq zippedFile.zip

To unzip the contents of a zip in a new directory (for avoiding clutter and confusion) run:

$ unzip zippedFile.zip  -d /Folder_to_Unpack_into