Updated: February 13, 2010
A few weeks ago, we've learned about strace, a simple yet powerful system debugging tool that allows you to trace the system calls during the execution of a process, allowing you to solve many problems quickly and efficiently. It's time to step up the geeky fun a notch and learn about OProfile.
OProfile is a Linux system-wide profiling tool that you can use to, uh, profile and analyze performance and runtime problems with your applications, or even the kernel itself. It's very simple to use and does not require any special preparations. No need to patch the kernel or use debug symbols. Just insert the module and start running.
OProfile uses the hardware performance counters of the CPU to enable profiling of a wide variety of statistics, which you can then use for profiling of the kernel and your applications. In fact, OProfile works with everything, including hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.
After you've collected the data you need, you can run reports against it, even produce graphs showing you a vizualization of the profiled runs.
For more details, please read the Novell Cool Solutions OProfile article and visit the official website, where you can find lots of useful information about the tool, including its numerous features and advantages.
Now, let's use it.
I must warn you though. Officially, OProfile is alpha software. Although it has been tested to work well with a wide range of architecture platforms and kernels, there's no guarantee it will do what's expected of it. You may even break your system.
My experience shows no issues with OProfile, but you may not be so lucky. Now, if you're brave enough, proceed.
Regardless, using OProfile is a very geeky thing that will surely impress everyone and may even provide you with yet another powerful tool for making your environment smarter, faster and safer. Home users will probably never need it, but they just might be piqued enough to give it a try, especially if they're facing severe performance problems. System admin wise, OProfile probably falls into the Level II-III support, so you won't be using it that often, but when you do, it should come quite handy.
The tool comes available in the repositories of many distributions, so you will not have to manually download and compile. If you do, consider using checkinstall to have it registered in your software database.
Now, we need to start using the tool.
The first question you need to ask yourselves is: do you also want to profile the kernel? Your answer will determine the OProfile command line.
If you wish to run OProfile without profiling the kernel, then:
If you do want to profile the kernel, as well, then:
opcontrol --vmlinux=/boot/vmlinux-`uname -r`
The above command will set the Linux kernel to the running version of your kernel in uncompressed form. To make sure such a kernel exists, please take a look under your /boot directory.
Modern Linux distributions ship with the kernel archived (zipped) to conserve space, so you will have to unzip it before it can be used:
So we do have the kernel available, it's vmlinux-<whatever>.gz. We need to unzip it. This is done with the gunzip command:
Now, if needed, rerun the opcontrol command from before to set the kernel. Once you've done this, launch the tool. It will start collecting the data.
Let it run for a while before stopping it and profiling the data. In fact, to make things meaningful, we'll run a little compilation in the background, so that our report contains more data, as well as more meaningful data. A good example is the MPlayer.
After a while you can simply dump the collected data and continue profiling or stop the profiler altogether. To just dump the data, use:
To stop it, use:
If, at any given moment you wish to resrt your profiling counters and start fresh, then you can reset the OProfile deamon:
And to shut it down altogether:
Once you've collected enough data, time for a report. Just run:
To get what you need. This is where the real fun begins, analyzing the report and trying to understand the problems you're facing.
Anyhow, here's a sample of what it may look like. Screenshot taken on another host, so please don't mind the differences in host names and CPU speeds and such.
In the leftmost column, you will get the exact number of samples collected. In the middle one, the percentage of time spent using different libraries. In the rightmost column, the actual process name. Usually, you will see a whole bunch of glibc, perl and other libraries. If you're compiling a tool that also uses GUI, then other interesting bits, too.
Since I was also listening to Youtube in Firefox while compiling and doing a few more things, you will notice calls to Nvidia driver, Flash player and so forth.
Now, opreport may not report everything you want and it may even exit with an error. Sometimes, the amount of time spent profiling may not be enough to produce a meaningful report. At other times, you may need packages with debug symbols or even a complete debug kernel.
In that case, you will want to run opcontrol --symbols or opcontrol -l to get more data.
And you may even want to create a call graph. This is done by running opcontrol -c.
You will need a tool that can read and display call graphs. There are many, many options available. For example, an oldie but goodie Kpl comes to mind:
You may also want to use KCachegrind, but this one requires the call graphs to be in the Valgrind format, which will require a conversion tool. This is where the real hacking kicks in, but this is beyond the scope of this article. We'll talk about advanced system debugging in following articles. Consider strace and OProfile a sort of a long and expensive warmup.
OProfile is a very handy, useful tool. In the hands of a smart system administrator, it can be used to detect application slowness problems, analyze system bottlenecks, optimize system performance and utilization, and resolve resource/usage conflicts. Combined with a range of other admin programs, some of which we've talked about and others we are yet to see, OProfile is a must-have item on the system debugging checklist.
I hope you've enjoyed this article. Many more are yet to come, exposing you to the realm of uber-hack. As to
profiling itself, we'll talk about some other cool programs, including Valgrind and Linux Trace Toolkit (LTT).