SVN Commit Fun

I know, I know, I should be using Git. Well, I do. I use it for everything I have under my control. Unfortunately, like many, I still have obligations that require me to use SVN on a regular basis. And when I roll with SVN, I roll command line. No, its not an elitist, terminal God point of view. I happen to like TortoiseSVN, but 90% of my SVN versioned work is on Linux servers.

If you’re like me, you’ll find yourself frequently using this lazy command line syntax inside your project directory to catch all your changes in one commit:

svn commit * -m "you should always leave a message!"

And there’s a very good chance you’ve run into this error, or something similar:

svn: Commit failed (details follow):
svn: '/path/to/unversioned_file' is not under version control

As it turns out, there doesn’t appear to be a way to ignore unversioned files in the svn commit options. Now before you sart leaving comments about the svn:ignore property of the SVN configuration, realize that it only applies to the svn add, svn import, and svn status commands. It has no impact on svn commit calls so we need another solution.

To that end, I put together a simple one-liner shell script to perform an SVN commit on all versioned files within the current directory, ignoring all unversioned files. Here it is:

svn status | grep ^[^?] | awk '{print $2}' | xargs svn commit -m "my commit message"

In case that makes no sense, it performs the following operations:

1. Call svn status to get a full list of relevant project files.
2. The question mark (?) in svn status signifies an unversioned file. The grep regular expression only matches files that do not start with that question mark.
3. Use awk to print the second column from the svn status call, which is the filename.
4. Use xargs to feed the filename list from awk to our svn commit call.

You can take this all one step further (as I have) and turn it into a bash script accepting your commit message as its only argument (signified by the ${1}).

#!/bin/bash
svn status | grep ^[^?] | awk '{print $2}' | xargs svn commit -m "${1}";

There you go, happy SVN’ing.

So I have a Centos server project for which I need to do some C++ development. I thought it would be logical to use Centos as my development desktop, inside VirtualBox. This line of thinking was unfortunately flawed as Centos sucks as a desktop. Don’t get me wrong, I wouldn’t build a server anymore that wasn’t running Centos, but the desktop is just hopeless.

Enter Ubuntu 10.10. It was quick, simple, and straightforward to install and get running. Things actually worked like you would expect they would, you know, like a normal desktop OS. I installed Eclipse and all the necessary C++ libraries I needed. Now it came time to build the Guest Additions in VirtualBox for Ubuntu…

Well, it appeared to work, but I couldn’t use the auto-resize functionality of VirtualBox to change the size of the desktop. Also, the screen resolution was stuck at 800×600. As it turns out, the built-in guest additions for Ubuntu 10.10 don’t work correctly. In order to fix the situation you just need to execute the following on the command line:

sudo apt-get update 
sudo apt-get install build-essential linux-headers-$(uname -r) 
sudo apt-get install virtualbox-ose-guest-x11

This will install the latest VirtualBox guest additions for Ubuntu 10.10. All you have to do now is restart your VirtualBox instance of Ubuntu and you’ll be all set. You can now adjust your screen resolution to your liking and the VirtualBox auto-resize of the Ubuntu guest display will be working.

Happy Ubuntu’ing!

The Overview

At the request of one of my faithful readers in my original article on packet capture with libpcap, I decided to post a guide to offline packet capture processing. Why is this useful? Because popular packet capture programs like Wireshark or tcpdump can save captures to files that can be processed later. You can then apply your specialized code to these previously captured packets.

The Code

NOTE: This program makes use of the http.cap Wireshark packet capture sample.

#include 
#include 
#include 
#include 
#include 
#include 
#include 
 
using namespace std;
 
void packetHandler(u_char *userData, const struct pcap_pkthdr* pkthdr, const u_char* packet);
 
int main() {
        pcap_t *descr;
        char errbuf[PCAP_ERRBUF_SIZE];
 
        // open capture file for offline processing
        descr = pcap_open_offline("http.cap", errbuf);
        if (descr == NULL) {
                cout << "pcap_open_live() failed: " << errbuf << endl;
                return 1;
        }
 
        // start packet processing loop, just like live capture
        if (pcap_loop(descr, 0, packetHandler, NULL) < 0) {
                cout << "pcap_loop() failed: " << pcap_geterr(descr);
                return 1;
        }
 
        cout << "capture finished" << endl;
 
        return 0;
}
 
void packetHandler(u_char *userData, const struct pcap_pkthdr* pkthdr, const u_char* packet) {
        const struct ether_header* ethernetHeader;
        const struct ip* ipHeader;
        const struct tcphdr* tcpHeader;
        char sourceIp[INET_ADDRSTRLEN];
        char destIp[INET_ADDRSTRLEN];
        u_int sourcePort, destPort;
        u_char *data;
        int dataLength = 0;
        string dataStr = "";
 
        ethernetHeader = (struct ether_header*)packet;
        if (ntohs(ethernetHeader->ether_type) == ETHERTYPE_IP) {
                ipHeader = (struct ip*)(packet + sizeof(struct ether_header));
                inet_ntop(AF_INET, &(ipHeader->ip_src), sourceIp, INET_ADDRSTRLEN);
                inet_ntop(AF_INET, &(ipHeader->ip_dst), destIp, INET_ADDRSTRLEN);
 
                if (ipHeader->ip_p == IPPROTO_TCP) {
                        tcpHeader = (tcphdr*)(packet + sizeof(struct ether_header) + sizeof(struct ip));
                        sourcePort = ntohs(tcpHeader->source);
                        destPort = ntohs(tcpHeader->dest);
                        data = (u_char*)(packet + sizeof(struct ether_header) + sizeof(struct ip) + sizeof(struct tcphdr));
                        dataLength = pkthdr->len - (sizeof(struct ether_header) + sizeof(struct ip) + sizeof(struct tcphdr));
 
                        // convert non-printable characters, other than carriage return, line feed,
                        // or tab into periods when displayed.
                        for (int i = 0; i < dataLength; i++) {
                                if ((data[i] >= 32 && data[i] <= 126) || data[i] == 10 || data[i] == 11 || data[i] == 13) {
                                        dataStr += (char)data[i];
                                } else {
                                        dataStr += ".";
                                }
                        }
 
                        // print the results
                        cout << sourceIp << ":" << sourcePort << " -> " << destIp << ":" << destPort << endl;
                        if (dataLength > 0) {
                                cout << dataStr << endl;
                        }
                }
        }
}

The Breakdown

#include 
#include 
#include 
#include 
#include 
#include 
#include 
 
using namespace std;
 
void packetHandler(u_char *userData, const struct pcap_pkthdr* pkthdr, const u_char* packet);

These are the includes and declarations necessary for reading the packet captures. The first 2 are self explanatory, the following 5 includes might be less so. These are used for parsing and transforming data found in packets. The functions and structures included in these headers are integral to packet processing and are available natively on Linux systems (Ubuntu in this case).

    int main() {
        pcap_t *descr;
        char errbuf[PCAP_ERRBUF_SIZE];
 
        // open capture file for offline processing
        descr = pcap_open_offline("http.cap", errbuf);
        if (descr == NULL) {
                cout << "pcap_open_live() failed: " << errbuf << endl;
                return 1;
        }

After entering the main execution, we go straight to opening our target packet capture file, http.cap. To do this we use pcap_open_offline() and give it the capture filename and an error buffer as parameters. If all goes well, we get a pcap_t descriptor returned. If not, check the error buffer for details.

        // start packet processing loop, just like live capture
        if (pcap_loop(descr, 0, packetHandler, NULL) < 0) {
                cout << "pcap_loop() failed: " << pcap_geterr(descr);
                return 1;
        }
 
        cout << "capture finished" << endl;
 
        return 0;
}

Just like in a live packet capture, we use pcap_loop() to set up a handler callback for each packet to be processed. We give it the following:

• descr – the descriptor we just created with pcap_open_offline()
• count – 0 (zero), to indicate there is no limit to the number of packets we want to process
• callback – The name of our packet handler function
• userdata – NULL, to indicate that we will be passing no user defined data to the callack

When the entire file has been processed, we will print the “capture complete” message and then exit.

    void packetHandler(u_char *userData, const struct pcap_pkthdr* pkthdr, const u_char* packet) {
        const struct ether_header* ethernetHeader;
        const struct ip* ipHeader;
        const struct tcphdr* tcpHeader;
        char sourceIp[INET_ADDRSTRLEN];
        char destIp[INET_ADDRSTRLEN];
        u_int sourcePort, destPort;
        u_char *data;
        int dataLength = 0;
        string dataStr = "";

Here we define the packet handler callback, as per the libpcap specifications. For more details, check out my original post on packet capture. The following declarations define variables that will help us parse meaningful data out of the packets. These include packet header data, IP addresses, source/destination ports, and payload data.

There’s LOTS more useful information to be analyzed from the average packet. Check out the structure defined in the network includes at the beginning of the code for more details. Actually, it would probably be a hell of a lot easier to just download and fire up Wireshark. It will give you a greater appreciation for what can be learned from a packet.

        ethernetHeader = (struct ether_header*)packet;
        if (ntohs(ethernetHeader->ether_type) == ETHERTYPE_IP) {
                ipHeader = (struct ip*)(packet + sizeof(struct ether_header));
                inet_ntop(AF_INET, &(ipHeader->ip_src), sourceIp, INET_ADDRSTRLEN);
                inet_ntop(AF_INET, &(ipHeader->ip_dst), destIp, INET_ADDRSTRLEN);

I’m not going to delve to deeply into the specifics of network protocols, as that could be a post… check that… that could be a book of its own. Basically here we are parsing the ethernet header from the packet and using its type to determine if it is an IP packet or not. We use the ntohs() to convert the type from network byte order to host byte order.

If it is an IP packet, we parse out the IP header and use the inet_ntop() function to convert the IP addresses found in the IP header into a human readable format (i.e., xxx.xxx.xxx.xxx). In a lot of older examples you’ll see the use of inet_ntoa(), but this is not thread-safe and is deprecated.

               if (ipHeader->ip_p == IPPROTO_TCP) {
                        tcpHeader = (tcphdr*)(packet + sizeof(struct ether_header) + sizeof(struct ip));
                        sourcePort = ntohs(tcpHeader->source);
                        destPort = ntohs(tcpHeader->dest);
                        data = (u_char*)(packet + sizeof(struct ether_header) + sizeof(struct ip) + sizeof(struct tcphdr));
                        dataLength = pkthdr->len - (sizeof(struct ether_header) + sizeof(struct ip) + sizeof(struct tcphdr));

Similar to above, I use the IP header to determine if this is a TCP packet (they all should be since its a HTTP capture) and then parse out the TCP header. With the TCP header we can then determine the source and destination ports, with ntohs() again, and then determine the contents of the packet payload.

                       // convert non-printable characters, other than carriage return, line feed,
                        // or tab into periods when displayed.
                        for (int i = 0; i < dataLength; i++) {
                                if ((data[i] >= 32 && data[i] <= 126) || data[i] == 10 || data[i] == 11 || data[i] == 13) {
                                        dataStr += (char)data[i];
                                } else {
                                        dataStr += ".";
                                }
                        }
 
                        // print the results
                        cout << sourceIp << ":" << sourcePort << " -> " << destIp << ":" << destPort << endl;
                        if (dataLength > 0) {
                                cout << dataStr << endl;
                        }
                }
        }
}

In the final step of the packet handler we display the results of our rudimentary analysis. First we iterate through the bytes of the payload and save it in a format that is human friendly. If you try to print it out with the non-printable characters in there you will get some very messy results in your console. After this cleanup we simply output the packet data we have extracted and display it in the console.

The Summary

So now that you can process packets offline, what do you want to do with them? I don’t know about you, but aside from obvious applications to network analysis, I’d like to use this data for trending, visualization, or even generative art and sound. But then again I’m weird. What are you gonna do?

As part of a larger project I’m working on I need to refamiliarize myself with some old friends: C and libcap. While I do have a healthy dislike for coding in C, I have a much healthier respect for those who have used it to create some pretty fantastic stuff. It still takes conscious effort to not spend all my time wrapping C code in C++ objects, but I digress.

libpcap is a C library used to capture and analyze network packets off the wire, otherwise known as packet sniffing. Some of the more prominent applications out there using it right now are Snort intrusion detection system, Wireshark network analyzer (formerly known as Ethereal), and the proprietor of libpcap, tcpdump.

Why would you want to be able to analyze packets in code and not use one of these applications? Because packet sniffing, in the hands of someone knowledgeable in network protocols, can be a great, non-intrusive way to gather discrete or statistical information and tie it to other coded business logic. There’s also that little, evil, don-your-black-hat, I-wanna-be-a-hacker kind of feel to seeing packets you have no explicit reason to see. Motives aside, let’s see a simple example of how it works in Linux with a little C++ tacked on.

The Code

#include 
#include 
 
using namespace std;
 
static int packetCount = 0;
 
void packetHandler(u_char *userData, const struct pcap_pkthdr* pkthdr, const u_char* packet) {
        cout << ++packetCount << " packet(s) captured" << endl;
}
 
int main() {
        char *dev;
        pcap_t *descr;
        char errbuf[PCAP_ERRBUF_SIZE];
 
        dev = pcap_lookupdev(errbuf);
        if (dev == NULL) {
                cout << "pcap_lookupdev() failed: " << errbuf << endl;
                return 1;
        }
 
        descr = pcap_open_live(dev, BUFSIZ, 0, -1, errbuf);
        if (descr == NULL) {
                cout << "pcap_open_live() failed: " << errbuf << endl;
                return 1;
        }
 
        if (pcap_loop(descr, 10, packetHandler, NULL) < 0) {
                cout << "pcap_loop() failed: " << pcap_geterr(descr);
                return 1;
        }
 
        cout << "capture finished" << endl;
 
        return 0;
}

The Output

1 packet(s) captured
2 packet(s) captured
3 packet(s) captured
4 packet(s) captured
5 packet(s) captured
6 packet(s) captured
7 packet(s) captured
8 packet(s) captured
9 packet(s) captured
10 packet(s) captured
capture finished

In this very basic example we are simply setting up a network interface to use libpcap, capturing a few packets, and exiting with a message. Pretty vanilla, but there’s a good bit going on here. let’s break it down in more detail.

The Breakdown

#include 
#include 
using namespace std;

Here along with standard includes we also include pcap.h. This is the header file that is necessary for all libpcap operations and constants.

static int packetCount = 0;
void packetHandler(u_char *userData, const struct pcap_pkthdr* pkthdr, const u_char* packet) {
        cout << ++packetCount << " packet(s) captured" << endl;
}

This is the callback packet handler function that will do all the heavy lifting when it comes to network analysis. This callback is referenced later by the pcap_loop() function. Every packet that pcap receives will be passed to this function. Here’s the explanation of the parameters:

• userData - a pointer to user defined data that can be passed into each callback, as defined in the pcap_loop() call’s 4th parameter.
• pkthdr - the pcap packet header that includes relevant information about the packet as it relates to pcap’s capture.
• packet - a pointer to the actual packet that will be analyzed.

For the sake of this example, we won’t be doing any protocol analysis and will simply be counting the number of packets that pass over our listening interface.

int main() {
        char *dev;
        pcap_t *descr;
        char errbuf[PCAP_ERRBUF_SIZE];

The start of our packet capture program.

• dev – this will hold the name of the interface on which we will sniff
• descr – The descriptor, or handle, for the libpcap packet capture
• errbuf – a character buffer to contain any potential errors from libpcap. The max error size is defined by PCAP_ERRBUF_SIZE in pcap.h.

 dev = pcap_lookupdev(errbuf);
        if (dev == NULL) {
                cout << "pcap_lookupdev() failed: " << errbuf << endl;
                return 1;
        }

Here we use pcap_lookupdev() to find an available network interface on which to sniff. The name of the interface is returned in the dev character array. If there is an error or if no interface is available, pcap_lookupdev() returns a NULL value and fills the errbuf with the relevant error information. Returning a failure value then populating an error buffer is common practice for libpcap functions that end in error.

 descr = pcap_open_live(dev, BUFSIZ, 0, -1, errbuf);
        if (descr == NULL) {
                cout << "pcap_open_live() failed: " << errbuf << endl;
                return 1;
        }

Next we create our packet capture descriptor using pcap_open_live(). This is how we will enable our target network interface for sniffing. It will return a valid descriptor on success, a NULL on failure. The parameters are as follows:

• device – the name of the network interface to be used by libpcap for sniffing. We found it using pcap_lookupdev().
• snaplen – the maximum snap length, or packet size, to be handled by this descriptor. We using the system defined BUFSIZ constant.
• promisc – Determines whether or not the interface will listen in promiscuous mode. Promiscuous mode will analyze packets not specifically addressed for your machine, like if you were attached to a hub or a mirrored switch port. 1 turns it on, 0 turns it off.
• to_ms – this is the packet read timeout in milliseconds. Specify -1 for no timeout.
• errbuf – In case of an error, this is where a descriptive message will be left.

 if (pcap_loop(descr, 10, packetHandler, NULL) < 0) {
                cout << "pcap_loop() failed: " << pcap_geterr(descr);
                return 1;
        }
        cout << "capture finished" << endl;
 
        return 0;
}

And here’s the call we’ve been waiting for: pcap_loop(). This will take our pcap descriptor and start sniffing packets, sending them all to our packet handler callback function, aptly named packetHandler(). Here’s pcap_loop()’s parameters:

• descr – the packet capture descriptor that we have initialized in the previous steps.
• count – the number of packets to capture before pcap_loop() exits. Use -1 or 0 to use no limit.
• callback – this is the callback function that is called every time pcap sniffs a packet. As specified above in the packetHandler() function, it receives relevant user data, pcap headers, and full packet data. It is the work horse of the packet analysis process. If creating a callback of your own, be sure to follow the function signature given in packetHandler().
• userData – this is an array of unsigned bytes to be sent in with each packet. You can use it to hold any relevant user data you would like to send to the callback as its first paramater. You can also specify NULL if you wish to pass no user data to the callback.

The Summary

With this basic example you can now use C/C++ to capture network packets. But keeping a count of packets isn’t terribly interesting, is it? No, I don’t think so either. So what should I do next? If you managed to make it all the way to this summary, you can help me decide what aspects of packet capture or C/C++ to show case next. Comment on one of the following or ad your own idea and I’ll probably do it next!

• Learn how to use Berkeley packet filters to focus and make more efficient your packet sniffing.
• Process offline packet capture dumps from some of the other programs I mentioned earlier. It’s a critical skill for testing.
• Use libpcap to learn more about the interfaces on your system?
• Wrap this ugly C code in some much more developer friendly C++ objects.
• Get right into the nitty gritty and start learning how to analyze packet data.

The choice is your’s guys.

Centos isn’t exactly the best Linux desktop distro. Ubuntu, Fedora, or Mint jump to mind as better alternatives. But that’s not Centos’s fault as its core focus is stability. For this reason, Centos is purposely a bit behind the times and only adds proven functionality to its core, as well as its published package repositories.

Unfortunately, though, I need a C++ development environment that will suit a Centos server project I’m working on. Wireless network connectivity was the first hurdle I ran into. No out of the box support for my HP laptop. In fact, when trying to enable the wireless interface on my laptop I got nothing but a very cryptic error message:

[root@localhost ~]# ifup wlan0
Determining IP information for wlan0...SIOCSIFFLAGS: No such file or directory.

Helpful, I know. But with a little bit of research I found the setting up wireless section of the Centos laptop FAQs. In here it states Centos does not come with the required wireless firmware for any laptops that don’t allow its free distribution. This then leaves it up to us, the users, to track it down and install it. Here’s how:

1. Open up your /etc/sysconfig/hwconf and find the entry for your wireless card.  You can speed this up by searching for the device name of your card, wlan0 in my case, or entries that have the class NETWORK. The entry you want will look something like this:

   class: NETWORK
   bus: PCI
   detached: 0
   device: wlan0
   driver: iwl3945
   desc: "Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection"
   network.hwaddr: xx:xx:xx:xx:xx:xx
   vendorId: 8086
   deviceId: 4222
   subVendorId: 103c
   subDeviceId: 135b
   pciType: 1
   pcidom:    0
   pcibus:  2
   pcidev:  0
   pcifn:  0

2. In your wireless card’s entry, locate the name of the driver used.  In my case it was iwl3945.

   driver: iwl3945

3. Goto http://wiki.centos.org/HowTos/Laptops/Wireless and follow the instructions for your particular wireless driver.  In most cases this involves setting up RPMForge for yum, using yum to install driver firmware, then enabling the driver module.
4. (OPTIONAL) Disable your network and wpa_supplicant services and enable the NetworkManager.

   [root@localhost ~]# chkconfig network off
   [root@localhost ~]# service network stop
   [root@localhost ~]# chkconfig wpa_supplicant off
   [root@localhost ~]# service wpa_supplicant stop
   [root@localhost ~]# chkconfig NetworkManager on
   [root@localhost ~]# service NetworkManager start

And there you have it, wireless connectivity on your Centos 5 desktop. Not exactly a breeze, but it’s not rocket science either once you know the steps involved. Why do I have a feeling this will soon become a series or articles related to things that should be easy but aren’t on desktop Centos?