Updated: March 6, 2010
Normally, I try to avoid providing distro-centric solutions and go for across-the-board, agnostic answers that should weather the trial of time and space. This is not always possible, as evident in the title of this article.
Why CentOS and why Debug repositories? Well, if you recall, we had two very long tutorials on Kernel crash dump collection, one for the older LKCD utility and one for Kdump. In these two sessions, we learned how to setup the tools, crash the system and dump memory cores to a hard disk. The next step is to analyze these cores.
While the crash tutorial is still a few days off, getting your system prepared is a good idea. A necessary part of the crash analysis procedure is to have the right debug package installed, namely the debuginfo package for your running kernel. Since debug packages are a domain of geeks, they are usually pushed into the corner of obscurity. For example, on openSUSE, the repositories are available, but disabled by default.
You will need to tick the Enabled checkbox under Properties.
On CentOS, it's even worse, with Debug repositories missing entirely. You will have to add them manually. I want to help you bridge this step easily.
Why CentOS? Well, serious work is done using either RedHat or SUSE. We will focus some more on Kdump setup and troubleshooting all sorts of problems in two more preparatory articles, both of which should help you get ready for the crash analysis.
Today, in this short how-to, I will show you how to do that, demonstrating with CentOS 5.4, the latest and greatest release of the phenomenal RedHat server distro, the free version thereof.
For more information about CentOS (and RedHat-based distros), you may want to consider reading my ultra-long 50-page review and tutorial of CentOS 5.3 and an extensive review of Scientific Linux, a RedHat-based desktop distro, with lots of exciting features.
Now, let's take a look at CentOS package management, which uses Pirut front-end for the yum package manager.
The default repository list does not have the Debug repo either included or enabled. We'll need to add it manually, by hand.
Go to CentOS Wiki Additional Resource page and copy the text from the code box into a text editor. Save the file as Centos-Debug.repo under /etc/yum.repos.d.
Here we go:
For those who can't see the image:
name=CentOS-$releasever - DebugInfo
Please pay attention to the two baseurl lines. The official CentOS documentation lists the second, shorter string, currently commented in my screenshot and the code section above, as the right URL for the repository. It does not work. However, commenting it out and enabling the first line, which is commented out by default, solves the problem and you have the Debug repository enabled.
Default, not working (might change in the future):
This is how it ought to be:
Next, run yum (or Pirut), after the packages are indexed, you will have debuginfo available, including kernel-debuginfo packages that are mandatory for crash analysis.
Note: You can also manually download RPM files from the repository, but this is a tedious work and you may miss dependencies.
This is a very short and simple tutorial. But you will realize the real importance thereof in the crash analysis tutorial. debuginfo is the bread and butter of any serious work with the kernel. For example, you may need debug packages when using OProfile.
I hope you find this short tutorial helpful. There's a lot of super-geeky work ahead of us. First, I will show you how to configure Kdump on SUSE 11 and CentOS 5.4. Although we have pretty much everything we can possibly need in the full-blown Kdump guide, there are some small differences in the latest releases of the two distros, which require some attention. After that, we will learn how to process the dumped memory cores. And finally, we will work through the analysis files, kernel logs and backtraces and understand what each line means, learn how to look through the sources and intermix them with Assembly.
Take care and see you around.
This tutorial is a part of my Linux Kernel Crash Book. The book is available for free download, in PDF format. Please check the book article for more details.