Removing deleted remote branches from local repo

Sometimes we have a POC(Proof of concept) branches, which are forked off the main/feature branches. After a while when POC is complete, we/the dev no longer uses it.
Now over time the no of such POC branches increase and the repo doesnt look good.
So during our periodic maintenance and cleaning of GIT repo, we delete all such POC branches.

How to sync your local branch list.
Simple:

[parag@paragpc: ~/ProjectCodeName] git fetch -p

 x [deleted]   (none)  ->origin/POC-webCache

 x [deleted]   (none)  ->origin/POC-CDNChange

[parag@paragpc: ~/ProjectCodeName] 

where -p = --prune
                     After fetching, remove any remote-tracking branches which no longer exist on the remote. 


Similarly, if you are have created a local branch and committed some changes while the remote tracking branch is not present at origin, issue:

[parag@paragpc: ~/GitProject] git branch -d <the_dirty_poc_branch>

where -d = --delete
        Delete a branch. The branch must be fully merged in its upstream branch, or in HEAD if no upstream was set with --track or --set-upstream

PS: Before running any delete related operation on branch its better to double check and understand what you are doing!.

Linux: Flush SWAP memory pages to RAM

On my dev machine, after I had started some services like Samba, MySQL, Jetty and executing some batch SQL queries which churned out a big database, I realized that my system had become slow and sluggish. I immediately checked the RAM usage via the System Task Monitor that comes with CentOS. And I noticed that though my RAM had substantial free space, SWAP was in action.

 If you ever run into such situation you can run the following commands to free you SWAP and move the active pages from SWAP to RAM. Before running the following commands ensure that you have substantial free space in your RAM i.e. free RAM size > SWAP memory size in use.

[root@paragcentosvm ~]# sudo swapoff -a
[root@paragcentosvm ~]# sudo swapon -a

The first command, swapoff, will take some time around 2-3 minutes (or more depending on the amount of SWAP memory in use, which needs to be freed). What is does is disables the SWAP partition for paging and swapping. When the -a flag is given, swapping is disabled on all known swap devices and files (as found in /proc/swaps or /etc/fstab)

The next swapon commands enables the SWAP partition for paging and swapping.

Linux : Yum rpmdb: damaged header Problem

While trying to get the latest chrome working on my machine, the yum process crashed. It just wouldn't work. first I tried to clean the yum cache.

[root@paragcentosvm ~]# yum clean all
Loaded plugins: fastestmirror, refresh-packagekit, security
Cleaning repos: base epel extras google-chrome updates
Cleaning up Everything
Cleaning up list of fastest mirrors
[root@paragcentosvm ~]# 

but it did not seem to work. It would just show me the following error:

[parag@paragcentosvm:/home/parag/Downloads]sudo yum update firefox
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
epel/metalink                                                                                                                                                | 4.8 kB     00:00     
 * base: centos.mirror.net.in
 * epel: epel.mirror.net.in
 * extras: centos.mirror.net.in
 * updates: centos.mirror.net.in
base                                                                                                                                                         | 3.7 kB     00:00     
http://centos.mirror.net.in/centos/6.8/os/x86_64/repodata/ad3a307dfd95da4d7a7aad136162378d18ead7271010822806902dfa3edb55f2-primary.sqlite.bz2: [Errno 14] PYCURL ERROR 18 - "transfer closed with 4902072 bytes remaining to read"
Trying other mirror.
base/primary_db                                                                                                                                              | 4.7 MB     00:02     
epel                                                                                                                                                         | 4.3 kB     00:00     
epel/primary_db                                                                                                                                              | 5.8 MB     00:06     
http://centos.mirror.net.in/centos/6.8/extras/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with 3435 bytes remaining to read"
Trying other mirror.
extras                                                                                                                                                       | 3.4 kB     00:00     
extras/primary_db                                                                                                                                            |  36 kB     00:00     
google-chrome                                                                                                                                                |  951 B     00:00     
google-chrome/primary                                                                                                                                        | 1.8 kB     00:00     
google-chrome                                                                                                                                                                   3/3
http://centos.mirror.net.in/centos/6.8/updates/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 18 - "transfer closed with 3456 bytes remaining to read"
Trying other mirror.
updates                                                                                                                                                      | 3.4 kB     00:00     
updates/primary_db                                                                                                                                           | 732 kB     00:00     
error: rpmdb: damaged header #1703 retrieved -- skipping.
Resolving Dependencies
--> Running transaction check
---> Package firefox.x86_64 0:38.1.0-1.el6.centos will be updated
---> Package firefox.x86_64 0:45.2.0-1.el6.centos will be an update
--> Processing Dependency: nss >= 3.21.0 for package: firefox-45.2.0-1.el6.centos.x86_64
--> Processing Dependency: nspr >= 4.11.0 for package: firefox-45.2.0-1.el6.centos.x86_64
--> Running transaction check
---> Package nspr.x86_64 0:4.10.8-1.el6_6 will be updated
--> Processing Dependency: nspr = 4.10.8-1.el6_6 for package: nspr-devel-4.10.8-1.el6_6.x86_64
---> Package nspr.x86_64 0:4.11.0-1.el6 will be an update
---> Package nss.x86_64 0:3.19.1-3.el6_6 will be updated
--> Processing Dependency: nss = 3.19.1-3.el6_6 for package: nss-sysinit-3.19.1-3.el6_6.x86_64
--> Processing Dependency: nss = 3.19.1-3.el6_6 for package: nss-devel-3.19.1-3.el6_6.x86_64
--> Processing Dependency: nss(x86-64) = 3.19.1-3.el6_6 for package: nss-tools-3.19.1-3.el6_6.x86_64
---> Package nss.x86_64 0:3.21.0-8.el6 will be an update
--> Processing Dependency: nss-util >= 3.21.0 for package: nss-3.21.0-8.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.21)(64bit) for package: nss-3.21.0-8.el6.x86_64
--> Running transaction check
---> Package nspr-devel.x86_64 0:4.10.8-1.el6_6 will be updated
---> Package nspr-devel.x86_64 0:4.11.0-1.el6 will be an update
---> Package nss-devel.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-devel.x86_64 0:3.21.0-8.el6 will be an update
--> Processing Dependency: pkgconfig(nss-util) >= 3.21.0 for package: nss-devel-3.21.0-8.el6.x86_64
---> Package nss-sysinit.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-sysinit.x86_64 0:3.21.0-8.el6 will be an update
---> Package nss-tools.x86_64 0:3.19.1-3.el6_6 will be updated
---> Package nss-tools.x86_64 0:3.21.0-8.el6 will be an update
---> Package nss-util.x86_64 0:3.19.1-1.el6_6 will be updated
---> Package nss-util.x86_64 0:3.21.0-2.el6 will be an update
--> Running transaction check
---> Package nss-util-devel.x86_64 0:3.19.1-1.el6_6 will be updated
---> Package nss-util-devel.x86_64 0:3.21.0-2.el6 will be an update
error: rpmdb: damaged header #1703 retrieved -- skipping.
error: rpmdb: damaged header #1703 retrieved -- skipping.
error: rpmdb: damaged header #1703 retrieved -- skipping.

To fix the issue I ran the following commands, to repair the corrupted yum  metadata files/db.

[parag@paragcentosvm:/home/parag]sudo rm -f /var/lib/rpm/__db*
[parag@paragcentosvm:/home/parag]sudo rpm --rebuilddb
[parag@paragcentosvm:/home/parag]

And everything just became normal again.

Linux : How to get Network Driver Details

If you are looking for a command to query the network card driver details such as name and version, run the ethtool command  with either -i or --driver option followed by the network interface name.

[parag@clara ~]# ethtool -i eth1
driver: via-rhine
version: 2.4.9
firmware-version: 
bus-info: 0000:03:00.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
[parag@clara ~]# ethtool -i eth0
driver: tg345
version: 13.8
firmware-version: nb vs.49
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
[parag@clara ~]# 
[parag@paragcentosvm:/home/parag]ethtool --driver eth0
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version: 
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Jetty Web Server Giving java.util.zip.ZipException: invalid entry size Exception

Today I ran into a bizarre error on Jetty WebServer. I have a few web-sites and services hosted on a Jetty server version 9.3.6.v20151106 running on a Centos 6.4.
I made some changes in a web service code and deployed the binary to my Jetty webapps directory, as soon as I started my Jetty server, it started showing me errors :

[parag@centosvm:/services/jetty-distribution-9]java -jar start.jar 
2016-05-01 17:41:00.380:INFO::main: Logging initialized @625ms
2016-05-01 17:41:00.493:WARN:oejs.HomeBaseWarning:main: This instance of Jetty is not running from a separate {jetty.base} directory, this is not recommended.  See documentation at http://www.eclipse.org/jetty/documentation/current/startup.html
2016-05-01 17:41:00.911:INFO:oejs.Server:main: jetty-9
2016-05-01 17:41:00.959:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///home/parag/jetty/webapps/] at interval 1
2016-05-01 17:41:01.598:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@6093dd95{/staticpages,file:///home/parag/jetty/webapps/staticpages/,AVAILABLE}{/staticpages}
2016-05-01 17:41:08.060:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@32464a14{/wallsoft-x-web,file:///home/parag/jetty/webapps/reactJSSite/,STARTING}{/abcd}
java.util.zip.ZipException: invalid entry size (expected 11658 but got 11664 bytes)
 at java.util.zip.ZipInputStream.readEnd(ZipInputStream.java:384)
 at java.util.zip.ZipInputStream.read(ZipInputStream.java:196)
 at java.util.jar.JarInputStream.read(JarInputStream.java:207)
 at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:140)
 at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:118)
 at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:142)
 at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:179)
 at org.eclipse.jetty.annotations.AnnotationParser.parseJar(AnnotationParser.java:939)
 at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:851)
 at org.eclipse.jetty.annotations.AnnotationConfiguration$ParserTask.call(AnnotationConfiguration.java:163)
 at org.eclipse.jetty.annotations.AnnotationConfiguration$1.run(AnnotationConfiguration.java:548)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
 at java.lang.Thread.run(Thread.java:745)

I did a quick search on the internet and stackoverflow ofcourse for finding the problem cause and resolution and did not come across anything concrete. 

However in this particular answer, people had reported it could be due to a corrupt maven repository jar. How in the world it could have happened I do not know. No recent maven packages were downloaded (just incremental changes to code, no new package dependency). 

And I was able to run zip/unzip commands normally, as well as on .war files. Anyhow after spending close to 30 min, I decided to give the answer a shot and deleted my .m2/repository folder, rebuild my project, copied the binaries again and viola it worked :) 

[parag@centosvm:/home/parag] rm -rf .m2/repository/

Disable/Ban certain users to connect from ssh

Generally after turning on the sshd service on our machines, we fail to authorize or rather restrict a set of users only to access (white-listing) the machine. The following setting in sshd_config helps you to restrict access to a select few users. The settings will take effect after restarting the sshd demon on your machine if already running.

[parag@paragcentos:/home/parag]sudo cat /etc/ssh/sshd_config  | grep -i Allow
...
##Allow the following users only to remote via ssh
AllowUsers root parag
...
...
[parag@paragcentos:/home/parag]

Also please read this post on more tips on how to secure your ssh server. 

Checking Bad Blocks in Hard Disk

If you need to see/check how your hard disk in doing, the first thing you look for bad sectors and read/write speed of your hard disk.
Below are a few commands which can be run on your linux machine to check for any errors/speed related or read-write issues

[parag@centos: /home/parag]# fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd42ad42a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64         587     4194304   82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3             587       19458   151582720   83  Linux


[parag@centos: /home/parag]# badblocks -v -s /dev/sda1
Checking blocks 0 to 511999
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found.

[parag@centos: /home/parag]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   532 MB in  2.00 seconds = 266.05 MB/sec
 Timing buffered disk reads: 248 MB in  3.02 seconds =  82.11 MB/sec
[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   556 MB in  2.00 seconds = 277.54 MB/sec
 Timing buffered disk reads: 236 MB in  3.00 seconds =  78.61 MB/sec

[parag@centos: /home/parag]# hdparm -tT /dev/sda1

/dev/sda1:
 Timing cached reads:   524 MB in  2.01 seconds = 261.21 MB/sec
 Timing buffered disk reads: 174 MB in  3.03 seconds =  57.49 MB/sec

[parag@centos: /home/parag]

Adding Missing Data points to Collections in Java

If you are from the .NET world, there are high chances that you have used LINQ in the .NET world. I was working on some JAVA project where there was some data which had missing datapoints.

This data in question was read from a DB, and was a big stored procedure. Making any more changes to it for filling in missing values was out of question, as it required nested loops which is better dealt in your programming language as it makes the program clean and more readable, plus its up the reader/consumer how he wants to prune/treat the data.
So here is a small snippet of code for filling missing data in a collection.

List<Timestamp> completeSeries= Utils.generateTimestampSequence(startTimestamp, endTimestamp);

completeSeries.stream().forEach((w) -> {

    boolean found = interestingDataCollectionWithMissingPoints.stream().

     anyMatch(x-> (x.getTimestamp().equals(w) ));

    if(!found){

     interestingDataCollectionWithMissingPoints.add(new InterestingDataClass(w, 0, 0, 0));

    }

   } );
//Now the interestingDataCollectionWithMissingPoints contains all the missing data points 

Here, I had a certain data collection which was missing some data for a per minute time intervals. And I needed the complete data for all the point in times for the given time period (startTimeStamp and endTimeStamp).
So the little code snippet above does the trick. Java had no such feature prior to 1.5 version. Of course move to newer Java compiler version  to use this nifty feature.

<build>
  <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.5.1</version>
    <configuration>
     <source>1.8</source>
     <target>1.8</target>
    </configuration>
   </plugin>
                 </plugins>
</build>

Automount a disk at machine start up.

Automount a disk at machine start up.

Recently bought an additional hard disk of 1 TB for my home pc running Centos. After you plugin your disk to the machine, the system does not automounts it. It detects the new hardware and assigns it a dev id(deviceId)

To automount the disk at computer start up follow the following steps:

1. Find the UUID of the newly added disk. You can find it by running the following command

1. 
   [root@paragWS ~]# blkid
   
   /dev/sdb1: UUID="dffb1c9c-29e9-4a00-add1-ad08d90c9469" TYPE="xfs"
   
   /dev/sdb2: UUID="SJhaki-0CLf-GDmR-j8LW-e1pL-hcwP-sQVnZJ" TYPE="LVM2_member"
   
   /dev/mapper/centos-root: UUID="cb3625db-6ee0-431f-884e-022037e66e48" TYPE="xfs"
   
   /dev/sda: UUID="c3d18886-59cc-4aba-aff8-c1f5403af506" TYPE="ext3"
   

2. Create an entry in fstab which looks like this:

2. 
   [root@paragWS ~]# cat /etc/fstab
   
   
   
   
   #
   
   # /etc/fstab
   
   # Created by anaconda on Tue Dec  6 00:17:24 2015
   
   #
   
   # Accessible filesystems, by reference, are maintained under '/dev/disk'
   
   # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
   
   #
   
   /dev/mapper/centos-root /                       xfs     defaults        0 0
   
   …
   
               …
   
   
   
   
               # /dev/sda SATA 1TB
   
                  UUID=c3d18886-59cc-4aba-aff8-c1f5403af506 /mnt/VideoData                   ext3     defaults        0 0
   
   
   
   

Default settings used above are equivalent to rw, suid, dev, exec, auto, nouser, async. Read more here

Ping pong test with Ansible

A quick and sweet command to ping your inventory/servers in ansible: 

[parag@centosVM ops-dir]$ ansible all -m ping -i production
PodAServer18 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodAServer93 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodAServer71 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodAServer73 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
PodBServer151 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
PodBServer48 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}

The above command pings all the server in your production to check for their availability/accessibility/to check whether they are reachable or not. 

In the output above servers : PodAServer18, PodAServer93, PodAServer71, PodAServer73  are accessible and PodBServer151, PodBServer48 are not reachable. 

This can be modified to check for specific sub-section/groups in the server inventory file. 

The following command limits the ping test to the emailServers group defined in the staging inventory file. 

[parag@centosVM ops-dir]$ ansible all -m ping -i staging--limit emailServers
server43 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
Server35 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
Rack3Server45 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\n",
    "unreachable": true
}
[parag@centosVM ops-dir]

aa

Troubleshooting Packet drops in Solarflare 10G network card

If you experience packet drops while listening to UDP packet broadcast then the following extracts from the onload_stackdump command utility can help you
identify the Onload stack under memory pressure.

[root@paragpc ~]  onload_stackdump lots

You can monitor the rate of these drops for the Onload stack using the above command, for example in one of my machine the stackdump reported:

[root@paragpc ~] onload_stackdump lots | grep memory_pressure_drops

memory_pressure_drops: 81381

If this is increasing during bursts of data then increasing the value of EF_MAX_PACKETS to 65000 or higher could resolve the issue by providing more buffering but if this value is increasing steadily then it would only delay the point at which drops happen. You can set this by setting “EF_MAX_PACKETS=65000” in the environment or prefix the ‘onload’ command, e.g.:

[root@paragpc ~] EF_MAX_PACKETS=65000 onload <app_cmd_line>

Note that this is further restricted for TX and RX in order to avoid potential deadlocks where all the packet buffers are used on the RX and can’t be freed because data needs to be sent to do this. By default there is a 75% limit on each so if you use the maximum for RX it would leave 25% for TX.

Read more about the how to troubleshoot and configure onload here.

C++: Looking at source code while debugging in GDB

Sometimes when I am debugging my code for issues/bugs, I want to look at the surrounding piece of code where my debugger is executing, or may be I want to look at the some lines of code in a particular file. Its quite time consuming to toggle windows, switch source code editor/IDE locating code(sometimes switch desktops). Worry no more, use the following commands, to help you navigate through the code, while debugging.

Scenario: after you have set your breakpoint and the GDB debugger hits the section, and you want to look around those line, type list

[Parag@paragpc:/home/Parag/Workspace/Utils]gdb sample
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/Parag/Workspace/Utils/sample...done.
(gdb) break countLines
Breakpoint 1 at 0x400cf2: file MultiUseHeader.hpp, line 21.
(gdb) run
Starting program: /home/sid/Parag/Workspace/Utils/samp 
inside registerQuickExitHandlers()

Breakpoint 1, countLines () at MultiUseHeader.hpp:21
warning: Source file is more recent than executable.
21     std::ifstream myfile(strFileName);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libstdc++-4.4.7-11.el6.x86_64
(gdb) list
16  char cArr[2];
17 };
18 
19 int countLines(std::string& strFileName) {
20 
21     std::ifstream myfile(strFileName);
22 
23     // new lines will be skipped unless we stop it from happening:    
24     myfile.unsetf(std::ios_base::skipws);
25 
(gdb) n
24     myfile.unsetf(std::ios_base::skipws);
(gdb) n
30         '\n');
(gdb) list
25 
26     // count the newlines with an algorithm specialized for counting:
27     unsigned line_count = std::count(
28         std::istream_iterator<char>(myfile),
29         std::istream_iterator<char>(), 
30         '\n');
31 
32     std::cout << "Lines: " << line_count << "\n";
33     return 0;
34 }
(gdb) 

Similarly if you want to look at some specific lines in a source file, type:

(gdb) list MultiUseHeader.hpp:32
27     unsigned line_count = std::count(
28         std::istream_iterator<char>(myfile),
29         std::istream_iterator<char>(), 
30         '\n');
31 
32     std::cout << "Lines: " << line_count << "\n";
33     return 0;
34 }
35 
36 
(gdb) 

Example Boost MultiIndex Container Insertion and Iteration

Here is another easy example to get you started with boost multi index containers:

#include <string>
#include <iostream>
#include <boost/multi_index_container.hpp>
#include <boost/multi_index/member.hpp>
#include <boost/multi_index/ordered_index.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>

using boost::multi_index::multi_index_container;
using boost::multi_index::ordered_non_unique;
using boost::multi_index::ordered_unique;
using boost::multi_index::indexed_by;
using boost::multi_index::member;
using boost::multi_index::nth_index;
using boost::multi_index::get;

struct employee_entry
{
 employee_entry( const std::string& first,
   const std::string& last,
   long id):
  first_name_(first),
  last_name_(last),
  id_(id)
 {}
 std::string first_name_;
 std::string last_name_;
 long id_;
};

typedef multi_index_container<
 employee_entry, 
 indexed_by<
  ordered_unique<
       member<employee_entry, std::string, &employee_entry::first_name_> 
  >, 
  ordered_non_unique<
       member<employee_entry, std::string, &employee_entry::last_name_> 
  >, 
  ordered_non_unique<
        member<employee_entry, long, &employee_entry::id_> 
  >
 >
> employee_set;

//employee set.... multi-index
employee_set m_employees;

using namespace std;

//Define the different type of iterators
typedef nth_index<employee_set, 0>::type fname_view;
fname_view& fdv = get <0> (m_employees);

typedef nth_index<employee_set, 1>::type lname_view;
lname_view& lnamev = get <1> (m_employees);


typedef nth_index<employee_set, 2>::type id_view;
id_view& idv = get <2> (m_employees);

void PrintLnameWise()
{
 ///get employees sorted by lname
 cout<<"Printing sorted by Lname... \n";
 for(lname_view::iterator it = lnamev.begin(), it_end(lnamev.end()); it != it_end; ++it)
 {
  std::cout << it->first_name_  <<" "
   << it->last_name_ << ":"
   << it->id_ << std::endl;
 }

   const std::string str(40, '-');
   std::cout << str << std::endl;
}

void PrintFnameWise()
{
 ///get employees sorted by first name
 cout<<"Printing sorted by Fname... \n";

   for(fname_view::iterator it = fdv.begin(), it_end(fdv.end()); it != it_end; ++it)
   {
       std::cout << it->first_name_  <<" "
                 << it->last_name_ << ":"
                 << it->id_ << std::endl;
   }
  
   const std::string str(40, '-');
   std::cout << str << std::endl;
}


void PrintIdWise()
{
   cout<<"Printing sorted by Id... \n";
   
   for(id_view::reverse_iterator it = idv.rbegin(), it_end(idv.rend()); it != it_end; ++it)
   {
       std::cout << it->first_name_  <<" "
                 << it->last_name_ << ":"
                 << it->id_ << std::endl;
   }

   const std::string str(40, '-');
   std::cout << str << std::endl;
}

int main()
{

   typedef nth_index<employee_set, 0>::type first_name_view;
   first_name_view& fnv = get<0>(m_employees);

   fnv.insert(employee_entry("John", "Smith", 110));
   fnv.insert(employee_entry("Fudge", "Hunk", 97));
   fnv.insert(employee_entry("Tolem", "Bathi", 87));
   fnv.insert(employee_entry("Prisha", "Agrawal", 1));
   fnv.insert(employee_entry("Shilpi", "Jain", 25));

   std::cout << "Count Structure after 5 inserts:" << m_employees.size() << std::endl;

   //Inserting via the ID Index iterator
   idv.insert(employee_entry("Carlos", "Linus", 140));
   idv.insert(employee_entry("Donald", "gates", 7));
   std::cout << "Count Structure after 2 more inserts:" << m_employees.size() << std::endl;
  
   //Inserting via the lname iterator/index
   lnamev.insert(employee_entry("Sharad", "Smith", 10));
   lnamev.insert(employee_entry("Parag", "Agrawal", 28));
   std::cout << "Final Count Structure after 2 more inserts:" << m_employees.size() << std::endl;
  
   const std::string str(40, '-');
   std::cout << str << std::endl;

   PrintIdWise();
   PrintLnameWise();
   PrintFnameWise();



   return 0;
}

Example Boost MultiI ndex Container

Here is a quick example to get to you started with boost::multi_index_container. Multi_index containers enables defining containers maintaining one or more indices with different sorting and access semantics.
Here is a sample program to get you started:

#include <boost/multi_index_container.hpp>
#include <boost/multi_index/member.hpp>
#include <boost/multi_index/hashed_index.hpp>

#include <iostream>

class CartItem{

public:

 int64_t cartItemId;
 int barcode;
 char symbol[10];        
 int64_t validityDate;     
 float price;

};


struct CartItemByCartItemIdIndexTag{};
struct CartItemByBarcodeIndexTag{};

typedef boost::multi_index_container<
 CartItem,
 boost::multi_index::indexed_by<
  boost::multi_index::hashed_unique<
   boost::multi_index::tag<CartItemByCartItemIdIndexTag>,
   boost::multi_index::member<CartItem, int64_t, &CartItem::cartItemId>
  >,
  boost::multi_index::hashed_unique<
   boost::multi_index::tag<CartItemByBarcodeIndexTag>,
   boost::multi_index::member<CartItem, int, &CartItem::barcode>  
  >
 >
> CartItemMapContainer;


typedef CartItemMapContainer::index<CartItemByCartItemIdIndexTag>::type CartItemByCartItemIdIndex;
typedef CartItemMapContainer::index<CartItemByBarcodeIndexTag>::type CartItemByBarcodeIndex;

//Global Variable
CartItemMapContainer  itemmultiIndexContainer;

void insertItem(int64_t cartItemId, int barcode){


 CartItem c;
 c.cartItemId = cartItemId;
 c.barcode = barcode;

 //Insert operation
 CartItemByCartItemIdIndex& index = itemmultiIndexContainer.get<CartItemByCartItemIdIndexTag>();

 if(!index.insert(c).second){
  std::cout << "Error inserting item";

 }
 else
  std::cout << "Insert successful\n";

}

void searchByItemId(int64_t cartItemId){

 CartItem d;
 //Find Operation by CartItemId
 CartItemByCartItemIdIndex& idx = itemmultiIndexContainer.get<CartItemByCartItemIdIndexTag>();
 CartItemByCartItemIdIndex::iterator itrb = idx.find(cartItemId);

 if(itrb == idx.end()){
  std::cout << "Look up By CartItemId failed";
  return;
 }
 else
  d = (*itrb);

 std::cout << "CartItemId:" << d.cartItemId << ", Barcode:" << d.barcode << std::endl;
}

void searchByBarcode(int barcode){

 CartItem e;
 CartItemByBarcodeIndex& indexByBarCode = itemmultiIndexContainer.get<CartItemByBarcodeIndexTag>();
 CartItemByBarcodeIndex::iterator itr = indexByBarCode.find(barcode);

 if(itr == indexByBarCode.end()){
  std::cout << "look up By BarCode failed";
  return;
 }
 else
  e = *itr;

 std::cout << "CartItemId:" << e.cartItemId << ", Barcode:" << e.barcode << std::endl;

}

int main(){

 insertItem(1,2);
 insertItem(2,4);
 insertItem(3,6);
 searchByItemId(1);
 searchByBarcode(6);


 return 0;

}

Bash Script to Create users in a DB

Many a times when running unit test on a clean build, I want to skip the user creation part, or may be reset the user details/profile for a particular user(s) in Development environment.
Of course there are unit test suite written to automate the flow, but sometimes when you are in the flow or designing the test cases itself, you need something to do the little rub-scrub-and-setup the little tiny jobs.
Here is my small script which deletes two very specific users from the my_dev_db db and creates them again. The details of the user are in file UserProd.txt which is tab delimited.  Enjoy.

[parag@paragcentosvm ~]# cat populateUsers.sh
#!/bin/sh

mysql -u parag_dev -pSOMEPASSWORD -e "DELETE FROM User_Table where UserId in (110,111);" -D my_dev_db;
mysql -u parag_dev -pSOMEPASSWORD -e "LOAD DATA LOCAL INFILE '/home/Parag/UserProd.txt' INTO TABLE User_Table;" -D my_dev_db;

Boost Mutex scoped_lock example

Here is a quick example showing how mutex helps in doing a synchronous access on shared resource. We are using boost::mutex::scoped_lock here, which ensures that the ownership of the mutex object is relinquished even in case the code following the lock throws an exception, thus preventing possible thread deadlocks.

#include <boost/thread/thread.hpp>
#include <boost/thread/mutex.hpp>
#include <boost/bind.hpp>
#include <iostream>

//Compile using : g++ program.cpp -lboost_thread

boost::mutex io_mutex;

void count(int id)
{
  for (int i = 0; i < 5; ++i)
  {
    boost::mutex::scoped_lock lock(io_mutex);
    std::cout << id << ": " << i << std::endl;
  }
}


void countWithoutMutex(int id)
{
  for (int i = 0; i < 5; ++i)
  {
    std::cout << id << ": " << i << std::endl;
  }
}


int main(int argc, char* argv[])
{
std::cout << "The wrong way\n";
  boost::thread thrd1(
    boost::bind(&countWithoutMutex, 1));
  boost::thread thrd2(
    boost::bind(&countWithoutMutex, 2));
  thrd1.join();
  thrd2.join();
std::cout << "Now the right way\n";
   boost::thread thrd3(
    boost::bind(&count, 3));
  boost::thread thrd4(
    boost::bind(&count, 4));
  thrd3.join();
  thrd4.join();
  

  return 0;
}

Output:

The wrong way
2: 01: 0
1: 1
1: 2
1: 3
1: 4

2: 1
2: 22
: 3
2: 4
Now the right way
3: 0
3: 1
3: 2
3: 3
3: 4
4: 0
4: 1
4: 2
4: 3
4: 4

R : Webcrawler Parser with Try-Catch

Well, I felt the need to do some analysis over content hosted on some initial set of web-sites and then aggregate it, plot. I created a very simple and easy parser which would crawl and parse the data read from these websites. I used the try-catch block for fault tolerance and resilience from error (website down, unavailable or trust error, etc). Here is a sample code, where I have used tryCatch and readLines methods :

  
>myUrlStats <- function(urlToCrawl) {
    statData <- tryCatch(
               {
   dataReadFromUrls <- readLines(con=urlToCrawl)
   myWebCrawlParser(dataReadFromUrls)
  },
        error=function(errorMessageStr) {
            message(paste("URL does not seem to exist:", urlToCrawl))
            message("Error message:")
            message(errorMessageStr)
            return(-1)
        },
        warning=function(warningMessageStr) {
            message(paste("URL caused a warning:", urlToCrawl))
            message("Warning message:")
            message(warningMessageStr)
            # Choose a return value in case of warning
            return(NULL)
        },
        finally={
   ##Clean up code
  }
 )
    return(statData)
}


> myWebCrawlParser <- function(dataReadFromURL){
# Do your analysis parsing here
# also like you can mine other outlinks from this data read for further traversing the web-links
return(1)
}


> urlToCrawl <- c(
  "http://superdevresources.com",
     "http://superbloggingresources.com"
     )
> finalReslt<- mapply(myUrlStats, urlToCrawl)

Happy programming!

R: Alternative and easy approach to Data concatenation

Sometimes in R you have the requirement of concatenating/appending data from two sources to create one Super Set, for eg. You may have a list of countries and economic growth indicators, and you want to create a data set which is a super set of this data (AxB), here is how you can do it via simple apply operation:

> myCustColNames
  MyColNames
1          A
2          B
3          C
4          D
5          E
> growthIndicator
[1] "GDP"        "Inflation"  "Population"
> apply(myCustColNames,1, paste, growthIndicator, sep="_")
     [,1]           [,2]           [,3]           [,4]           [,5]          
[1,] "A_GDP"        "B_GDP"        "C_GDP"        "D_GDP"        "E_GDP"       
[2,] "A_Inflation"  "B_Inflation"  "C_Inflation"  "D_Inflation"  "E_Inflation" 
[3,] "A_Population" "B_Population" "C_Population" "D_Population" "E_Population"
> 

Another example can be:

> myCustColNames<-data.frame(MyColNames=LETTERS[1:5])
> myCustColNames
  MyColNames
1          A
2          B
3          C
4          D
5          E
 > apply(myCustColNames,1, paste,seq(1:6), sep="")
     [,1] [,2] [,3] [,4] [,5]
[1,] "A1" "B1" "C1" "D1" "E1"
[2,] "A2" "B2" "C2" "D2" "E2"
[3,] "A3" "B3" "C3" "D3" "E3"
[4,] "A4" "B4" "C4" "D4" "E4"
[5,] "A5" "B5" "C5" "D5" "E5"
[6,] "A6" "B6" "C6" "D6" "E6"
> 

R: Converting a list to a dataframe

Sometimes a function returns a list data type, which you often need to convert to a data.frame data type. To convert the list to data.frame ran the following statement:

 compositeRowOfDataFrame<-do.call(rbind.data.frame, listData)

R deleting rows in a dataframe

To remove all rows from a data frame in R, you can use any of the following ways:

> b
   x  y   Result
1  1  1   B
2  1  2   B
3  1  3   C
4  1  4   A
5  1  5   B
6  1  6   A
7  1  7   C
8  1  8   C
9  1  9   B
10 1 10   C
> deleteRowsVec=c(FALSE)
> b[deleteRowsVec,]
[1] x   y   Result
<0 rows> (or 0-length row.names)