Debuging errors in startup code in Linux

Recently I got a Korg M1. When you have a synthesizer like that you want to have a good librarian that helps you organizing your sounds, have a big sound library and so on. A quick research brought nothing helpful. So I decieded to give Ctrlr a try. The downloaded Linux version did not work as I have newer binutils installed and libbfd has no stable ABI.

So I compiled it myself. When I started it in the debugger several assertions broke the program execution. So I had to fix them by myself. I got it running but from the next day on the program hangs in snd_seq_open which openes the ALSA sequencer.
There are several strange things:

  • Ctrlr is the only program which hangs in that function, that means snd_seq_open works in all other programs.
  • There seems to be a deadlock in bindtextdomain from libintl, that is called in nearly all programs on my computer, and thus well-tested
    • It turns out the function hangs even in the startup code. So what's going on, here? Is my system corrupt or is it some strange configuration of Ctrlr. I must debug the startup code.

      The startup process is well documented on this page. According to this page the following functions are calld at startup:

      all pointers in the section .preinit_array
      here, constructors are called

      Now, all entries from .init_array.

      all at_exit functions
      all functions in .fini_array
      all destructors

      Setting a break point on _start or __libc_csu_init does not lead to very useful results. First without source code and debug symbols of glibc I had to debug assembler using the stepi and nexti commands.
      Second, .init_array has more than 100 entries. For some reason gdb was not consistent with announcing the function calls. So it is easy to miss one.

      Next, I tried to examine .init_array with objdump, but I don't know how to do it and I didn't find any information about that in the internet

      So I decided to print the addresses from the running program and use gdb to resolve the address. Other aprroaches seem to involve too much debugging of external libraries. I wrote the following class:

      # define attribute_hidden __attribute__ ((visibility ("hidden")))

      extern "C" {
      /* These magic symbols are provided by the linker. */
      extern void (*__preinit_array_start []) (int, char **, char **)
      extern void (*__preinit_array_end []) (int, char **, char **)
      extern void (*__init_array_start []) (int, char **, char **)
      extern void (*__init_array_end []) (int, char **, char **)
      extern void (*__fini_array_start []) (void) attribute_hidden;
      extern void (*__fini_array_end []) (void) attribute_hidden;
      class testclass {
      testclass() {
      std::cerr << ("testclass()") << std::endl;
      wert = 77;

      void set(int i) {
      wert = i;
      std::cerr << ("testclass():: set ") << wert << std::endl;
      std::cerr << "returning from set " << std::endl;
      int get () const {
      std::cerr << ("testclass():: get ") << wert << std::endl;
      std::cerr << "returning from get " << std::endl;
      return wert;
      int wert;

      void printinit() {
      const size_t size = ::__init_array_end - ::__init_array_start;
      for (size_t i = 0; i < size; i++)
      printf("info symbol %p\n", (void*)__init_array_start[i]);
      printf("init_table end of %d:\n", size);
      //(*__init_array_start [i]) (argc, argv, envp);
      void printpreinit() {
      const size_t size = ::__preinit_array_end - ::__preinit_array_start;
      for (size_t i = 0; i < size; i++)
      printf("info symbol %p\n", i, __preinit_array_start[i]);
      printf("preinit_table end of %d:\n", size);

      //(*__preinit_array_start [i]) (argc, argv, envp);
      } testobjekt;

      Obviously also a global object is generated, so that the constructor is called during startup, before the program crashes. The functions get and set are made so that I can call them somewhere in the main program, so that the linker won't optimise this object away.

      Note, this program prints the process local addresses to the functions in .init_array. So we cannot use any static analysis tool like addr2line to resolve the addresses. We must use a debugger. The program already prints a list of commands to stderr. But cut and paste did not work (probably I should add “\r” to the output). So I copied all commands into a text file and loaded it using the source command of gdb. It worked nice. The next step is to save everything for later usage. For this purpose I used gdbs logging feature which is controlled by the
      commands set logging on, set logging off and set logging file.

      Now I examined the log file: All symbols seem to start with _GLOBAL. We can check it with the simple command
      gdb grep -v '^_GLOBAL' gdb.txt |less
      This was successful. So we can find out which symbols are stored in that table:
      grep '^_GLOBAL' gdb.txt |sort >symbols.init_array

      What a surprise: Every function seems to be called trice. And I don't really understand, why.

      Nevertheless, we have now all tools in order to examine the call tree.
      We can disassemble the binary:
      objdump -d build/Ctrlr >Ctrlr.disasm
      Then, we reduce this file to the entry points and function calls:
      grep '^[^ \t]\|call' Ctrlr.disasm
      Now, we have a file which we can process further.

      Note: using gprof might work, too, but it does not work as long as the program does not terminate :-(.