Under Linux, libSegFault and addr2line are underrated

Many modern programming languages like Java produce useful error messages when they fail. Some indicate where in the program (e.g., down to the line of source code) the failure occurred. Slowly, the C++ language is moving in this direction.

A particularly nasty problem in C and C++ are segmentation faults: they occur when you are trying to access a memory region that is not allocated. For example, this will occur if you are dereferencing a null pointer. You can mostly avoid segmentation faults in C++ by avoiding pointers, but they are instances when pointers are useful or necessary. And even if you are not causing them yourself, the software that you are using might be triggering them.

Thankfully, you can often get a useful message when you have a segmentation by linking with the libSegFault library under Linux.

Let me illustrate. The following program has a segmentation fault on line six:

#include <stdio.h>
#include <stdlib.h>

int go() {
  int x = 0;
  x += *((int *)NULL);
  return x;
}
int main() { return go(); }

I am calling this program hello. Let us compile the program with the line

cc -g -o hello hello.c -lSegFault

I am linking against SegFault, but not otherwise modifying my code. Importantly, I do not need to modify anything else. I do not need to run my program within the scope of another program (e.g., within a debugger like gdb or ldb). As far as I know, SegFault has been available pretty much all the time with Linux, you do not need to install anything. In the future, it will be removed from the standard C library but, at least under Ubuntu, it is being moved to the glibc-tools package. You can also build glibc-tools from its source code.

I run the program and I get the following…

Backtrace:
./hello[0x401116]
./hello[0x40112c]
/lib64/libc.so.6(+0x3feb0)[0x7faa8cc6eeb0]
/lib64/libc.so.6(__libc_start_main+0x80)[0x7faa8cc6ef60]
./hello[0x401045]

It does not tell me where exactly my program failed, instead I get a mysterious address (0x401116). There is a handy program under Linux called addr2line which will tell me where, in the source code, this address is. The error starts at address 0x401116, so let me type:

$ addr2line -e hello -a 0x401116
hello.c:6

And voilà! I get the exact location of the error. Again, this is lightweight and non-invasive, I only need my original program and the address.

The addr2line utility requires you to compile your code with debug symbols. If you try turning on optimization too much, you may not be able to get back to the line of code. However, you may use some optimization (-O1). The following works in this test:

cc -g -O1 -o hello hello.c -lSegFault

In fact, I recommend you compile your code with the -O1 flag generally when debugging.

My source code is available.

Appendix. Some of you might be wondering, why not wait for the application to dumb cores and then load the core into a debugger like gdb or ldb? That is not always possible. Some systems do not dump cores, or the cores might required privileged access. Furthermore, it may take more steps and more time if you have to redo the steps often.

Daniel Lemire, "Under Linux, libSegFault and addr2line are underrated," in Daniel Lemire's blog, May 1, 2023.

Published by

Daniel Lemire

A computer science professor at the University of Quebec (TELUQ).

10 thoughts on “Under Linux, libSegFault and addr2line are underrated”

  1. in your article it is not clear which compiler you use under Linux, but GCC since v4.8 from 2013/03, see

    https://gcc.gnu.org/gcc-4.8/changes.html

    (and CLang/LLVM since its first version) has implemented support for “address sanitizer”, which in addition to finding seg faults, finds double frees, uses after free, vector overruns, and much more at runtime.
    Just build in with:

    CFLAGS=-O1 -g -fsanitize=address
    LDFLAGS=-fsanitize=address

    and at the price of a binary enlargement and a slight execution slowdown, when the error occurs, it shows you a clear backtrace and the C source line where the error is generated, and where the memory was allocated. I always use it when build a debug binary

  2. yes, C built with ASAN is slow like Java or .Net languages.
    Once cleaned from bugs, rebuild without ASAN and you get all the power of C

  3. For deeply embedded systems sometimes it is really valuable to get the stack back trace on error or for all running thread when there is a thread deadlock. The problem is that -fomit-frame-pointer makes function stack frames chaining unstable so you need to disable this optimization. But sometimes even that is not enough as for example for ARM Thumb2 GCC even with -fno-omit-frame-pointer still doesn’t generate predictable frames chaining to avoid generation of one additional instruction in the function prologue/epilogue. The only solution is to patch GCC or to switch to CLANG if your platform allows it.

    Not sure how this will work with address sanitizing.

  4. Note that you can also compile with -Og which is like -O1 except it prevents certain optimizations that make debugging harder.

    Similarly, GCC supports different levels of debug information, from -g0 (no debug information) to -g3, with -g equivalent to -g2. For maximum debugability the optimal flags are probably something like -Og -g3 , though beware this may slow down compilation and/or make your binaries larger.

  5. One can also achieve this by linking against glog (Google logging library) and installing failure signal handler.

    When writing c or c++ programs I always link against glog, gflags, googletest and Google benchmark by default. These are statically linked and the whole program compiled with “-g -O2”. This is a pretty safe place to start from.

  6. Minor comment: The code in the post and in github do not coincide. Github’s code will indeed trigger a segfault on line 6, but the code in the post does not have an empty line before the definition of the go function and will trigger it in line 5.

  7. the developer of ASAN suggest use “-O1 -g” with -fsanitize=address
    and optionally add -fno-omit-frame-pointer to get better stacktrace

Leave a Reply

Your email address will not be published.

You may subscribe to this blog by email.