Debuging errors in startup code in Linux

Recently I got a Korg M1. When you have a synthesizer like that you want to have a good librarian that helps you organizing your sounds, have a big sound library and so on. A quick research brought nothing helpful. So I decieded to give Ctrlr a try. The downloaded Linux version did not work as I have newer binutils installed and libbfd has no stable ABI.

So I compiled it myself. When I started it in the debugger several assertions broke the program execution. So I had to fix them by myself. I got it running but from the next day on the program hangs in snd_seq_open which openes the ALSA sequencer.
There are several strange things:

• Ctrlr is the only program which hangs in that function, that means snd_seq_open works in all other programs.
• There seems to be a deadlock in bindtextdomain from libintl, that is called in nearly all programs on my computer, and thus well-tested
It turns out the function hangs even in the startup code. So what's going on, here? Is my system corrupt or is it some strange configuration of Ctrlr. I must debug the startup code.

The startup process is well documented on this page. According to this page the following functions are calld at startup:

all pointers in the section .preinit_array

_start

__libc_csu_init

_init

__gmon_start__

frame_dummy

__do_global_ctors_aux

here, constructors are called

Now, all entries from .init_array.

main

exit

all at_exit functions

all functions in .fini_array

all destructors

Setting a break point on _start or __libc_csu_init does not lead to very useful results. First without source code and debug symbols of glibc I had to debug assembler using the stepi and nexti commands.
Second, .init_array has more than 100 entries. For some reason gdb was not consistent with announcing the function calls. So it is easy to miss one.

Next, I tried to examine .init_array with objdump, but I don't know how to do it and I didn't find any information about that in the internet

So I decided to print the addresses from the running program and use gdb to resolve the address. Other aprroaches seem to involve too much debugging of external libraries. I wrote the following class:

#include
#include
# define attribute_hidden __attribute__ ((visibility ("hidden")))

extern "C" {
/* These magic symbols are provided by the linker. */
extern void (*__preinit_array_start []) (int, char **, char **)
attribute_hidden;
extern void (*__preinit_array_end []) (int, char **, char **)
attribute_hidden;
extern void (*__init_array_start []) (int, char **, char **)
attribute_hidden;
extern void (*__init_array_end []) (int, char **, char **)
attribute_hidden;
extern void (*__fini_array_start []) (void) attribute_hidden;
extern void (*__fini_array_end []) (void) attribute_hidden;
}
class testclass {
public:
testclass() {
printinit();
printpreinit();
std::cerr << ("testclass()") << std::endl;
wert = 77;
}

void set(int i) {
wert = i;
std::cerr << ("testclass():: set ") << wert << std::endl;
std::cerr << "returning from set " << std::endl;
}
int get () const {
std::cerr << ("testclass():: get ") << wert << std::endl;
std::cerr << "returning from get " << std::endl;
return wert;
}
protected:
int wert;

void printinit() {
const size_t size = ::__init_array_end - ::__init_array_start;
printf("init_table:\n");
for (size_t i = 0; i < size; i++)
printf("info symbol %p\n", (void*)__init_array_start[i]);
printf("init_table end of %d:\n", size);
//(*__init_array_start [i]) (argc, argv, envp);
}
void printpreinit() {
const size_t size = ::__preinit_array_end - ::__preinit_array_start;
printf("preinit_table:\n");
for (size_t i = 0; i < size; i++)
printf("info symbol %p\n", i, __preinit_array_start[i]);
printf("preinit_table end of %d:\n", size);

//(*__preinit_array_start [i]) (argc, argv, envp);
}
} testobjekt;

Obviously also a global object is generated, so that the constructor is called during startup, before the program crashes. The functions get and set are made so that I can call them somewhere in the main program, so that the linker won't optimise this object away.

Note, this program prints the process local addresses to the functions in .init_array. So we cannot use any static analysis tool like addr2line to resolve the addresses. We must use a debugger. The program already prints a list of commands to stderr. But cut and paste did not work (probably I should add “\r” to the output). So I copied all commands into a text file and loaded it using the source command of gdb. It worked nice. The next step is to save everything for later usage. For this purpose I used gdbs logging feature which is controlled by the
commands set logging on, set logging off and set logging file.

Now I examined the log file: All symbols seem to start with _GLOBAL. We can check it with the simple command
gdb grep -v '^_GLOBAL' gdb.txt |less
This was successful. So we can find out which symbols are stored in that table:
grep '^_GLOBAL' gdb.txt |sort >symbols.init_array

What a surprise: Every function seems to be called trice. And I don't really understand, why.

Nevertheless, we have now all tools in order to examine the call tree.
We can disassemble the binary:
objdump -d build/Ctrlr >Ctrlr.disasm
Then, we reduce this file to the entry points and function calls:
grep '^[^ \t]\|call' Ctrlr.disasm
Now, we have a file which we can process further.

Note: using gprof might work, too, but it does not work as long as the program does not terminate :-(.