[compiler-rt] Spring cleaning of compiler-rt symbolizers

[kubamracek]

Summary

Hi everyone,

this is a refactoring, NFC patch that tries to clean up the current symbolizer code in sanitizer_common/sanitizer_symbolizer*.[cc|h]. One of the reasons for that is that I'd like to implement some OS X specific symbolizers that could be available in case llvm-symbolizer isn't (AtosSymbolizer and/or DlAddr symbolizer, see http://reviews.llvm.org/D6588). However, there are other weirdnesses in the current implementation: We have several symbolizers available (libbacktrace, llvm-symbolizer, internal symbolizer, addr2line), but they don't share a common interface. The POSIXSymbolizer which operates on top of the individual symbolizers has a hardcoded format of output that it expects. The symbolizers also form a very weird chain, e.g. if libbacktrace symbolizer is available but cannot symbolize something, we still try to symbolize using an internal or external symbolizer. But when an internal symbolizer returns "false", we don't try an external one. SymbolizerProcess which is supposed to only transfer communication from/to an external process actually contains logic to format commands.

This patch extracts an interface, SymbolizerInterface that all the symbolizers now implement. SymbolizerProcess now only takes a string command and returns a string response. Things in POSIXSymbolizer that were specific to llvm-symbolizer and now extracted into LLVMSymbolizer class. Parsing of llvm-symbolizer output is extracted into ParseSymbolizePCOutput and ParseSymbolizeDataOutput methods, and it's used to parse lines from both llvm-symbolizer and addr2line. The symbolizers now form an explicit chain via the next_symbolizer_ field, and POSIXSymbolizer walks this linked list when symbolizing an address.

Thanks for reviewing!
Kuba

[samsonov]

Thanks for working on this! Let's work on landing this patch in pieces before moving on to http://reviews.llvm.org/D6588. The main comment is: this patch is still way too large to digest. Let's try to split it into bite-size pieces and land them one by one (move Extract routines, introduce and move SymbolizerProcess to header, change the interface, etc. For now a few high-level comments:

• think of splitting the code in sanitizer_symbolizer_posix_libcdep.cc into more files, or moving it around. Some code shouldn't really be POSIX-specific (e.g. Extract routines).
• think of adding more headers. For example, sanitizer_symbolizer.h is the header used by actual sanitizers, and Symbolizer is the class they are interested in. In the meantime, SymbolizerInterface (btw. the naming is really poor, considering we already have Symbolizer, maybe SymbolizerTool?) is an implementation detail, and could be hidden in another header.
• I'm fine with general approach "use the chain of symbolizers - try the first one, fallback to the next if necessary, then to the next etc.". But this approach then should be shared between POSIX and Windows - that is, WinSymbolizer will be no different from, say, InternalSymbolizer - it will be just another symbolizer in a chain. In particular, probably Symbolizer class will not even need to be abstract, and you could move all the logic to common sources.
• please re-use IntrusiveList instead of re-implementing linked lists, if possible.

SymbolizeData family of methods is really confusing. We should have renamed it to SymbolizeGlobal long ago (same for DataInfo -> GlobalInfo), I will do it soon.

[kubamracek]

In D7827#129177, @samsonov wrote:

Let's try to split it into bite-size pieces and land them one by one (move Extract routines, introduce and move SymbolizerProcess to header, change the interface, etc.

Sounds like a plan! Here's the first pieces: http://reviews.llvm.org/D7867, http://reviews.llvm.org/D7868.

[kubamracek]

One more: http://reviews.llvm.org/D7869

[kubamracek]

Another one: http://reviews.llvm.org/D7889

[kubamracek]

http://reviews.llvm.org/D7936: SymbolizerTool and better interface

[earthdok]

Just saw this CL.

Silently falling back to a different symbolizer is not always acceptable, for performance reasons. In Chromium, given the binary size, we never want to fall back to addr2line if llvm-symbolizer cannot be found. Otherwise this happens:
https://code.google.com/p/chromium/issues/detail?id=390552

So something like a flag to disable this behavior would be nice.

lib/sanitizer_common/sanitizer_symbolizer.h

 #ifndef SANITIZER_SYMBOLIZER_H
 #define SANITIZER_SYMBOLIZER_H
 
 #include "sanitizer_common.h"
 #include "sanitizer_mutex.h"
 
 namespace __sanitizer {
 
+// Parsing helpers, 'str' is searched for delimiter(s) and a string or uptr
+// is extracted. When extracting a string, a newly allocated (using
+// InternalAlloc) and null-terminataed buffer is returned. They return a pointer
+// to the next characted after the found delimiter.
+const char *ExtractToken(const char *str, const char *delims, char **result);
+const char *ExtractUptr(const char *str, const char *delims, uptr *result);
+const char *ExtractTokenUpToDelimiter(const char *str, const char *delimiter,
+                                      char **result);
+
 struct AddressInfo {
   // Owns all the string members. Storage for them is
   // (de)allocated using sanitizer internal allocator.
   uptr address;
 
   char *module;
   uptr module_offset;
 
protected:
    public:
     explicit SymbolizerScope(const Symbolizer *sym);
     ~SymbolizerScope();
    private:
     const Symbolizer *sym_;
   };
 };
 
+class SymbolizerInterface {
+ public:
+  // Can't declare pure virtual functions in sanitizer runtimes:
+  // __cxa_pure_virtual might be unavailable.
+
+  // The |stack| parameter is inout. It is pre-filled with the address,
+  // module base and module offset values and is to be used to construct
+  // other stack frames.
+  virtual bool SymbolizePC(uptr addr, SymbolizedStack *stack) {
+    UNIMPLEMENTED();
+  }
+
+  virtual bool SymbolizeData(uptr addr, DataInfo *info) {
+    UNIMPLEMENTED();
+  }
+
+  virtual void Flush() {}
+
+  virtual const char *Demangle(const char *name) {
+    return name;
+  }
+
+  SymbolizerInterface *next_symbolizer_;
+};
+
+// SymbolizerProcess encapsulates communication between the tool and
+// external symbolizer program, running in a different subprocess.
+// SymbolizerProcess may not be used from two threads simultaneously.
+class SymbolizerProcess {
+ public:
+  explicit SymbolizerProcess(const char *path);
+  char *SendCommand(const char *command);
+
+ protected:
+  bool Restart();
+  char *SendCommandImpl(const char *command);
+  bool ReadFromSymbolizer(char *buffer, uptr max_length);
+  bool WriteToSymbolizer(const char *buffer, uptr length);
+  bool StartSymbolizerSubprocess();
+
+  virtual bool ReachedEndOfOutput(const char *buffer, uptr length) const {
+    UNIMPLEMENTED();
+  }
+
+  virtual void ExecuteWithDefaultArgs(const char *path_to_binary) const {
+    UNIMPLEMENTED();
+  }
+
+  const char *path_;
+  int input_fd_;
+  int output_fd_;
+
+  static const uptr kBufferSize = 16 * 1024;
+  char buffer_[kBufferSize];
+
+  static const uptr kMaxTimesRestarted = 5;
+  static const int kSymbolizerStartupTimeMillis = 10;
+  uptr times_restarted_;
+  bool failed_to_start_;
+  bool reported_invalid_path_;
+};
+
 }  // namespace __sanitizer
 
 #endif  // SANITIZER_SYMBOLIZER_H

lib/sanitizer_common/sanitizer_symbolizer_libbacktrace.h

 #endif
 
 #ifndef SANITIZER_CP_DEMANGLE
 # define SANITIZER_CP_DEMANGLE 0
 #endif
 
 namespace __sanitizer {
 
-class LibbacktraceSymbolizer {
+class LibbacktraceSymbolizer : public SymbolizerInterface {
  public:
   static LibbacktraceSymbolizer *get(LowLevelAllocator *alloc);
 
-  SymbolizedStack *SymbolizeCode(uptr addr, const char *module_name,
-                                 uptr module_offset);
+  bool SymbolizePC(uptr addr, SymbolizedStack *stack) override;
 
-  bool SymbolizeData(uptr addr, DataInfo *info);
+  bool SymbolizeData(uptr addr, DataInfo *info) override;
 
   // May return NULL if demangling failed.
-  static char *Demangle(const char *name, bool always_alloc = false);
+  const char *Demangle(const char *name) override;
 
  private:
   explicit LibbacktraceSymbolizer(void *state) : state_(state) {}
 
   void *state_;  // Leaked.
 };
 
 }  // namespace __sanitizer
 #endif  // SANITIZER_SYMBOLIZER_LIBBACKTRACE_H

lib/sanitizer_common/sanitizer_symbolizer_libbacktrace.cc

 #  endif
 # else
 #  define SANITIZER_LIBBACKTRACE 0
 # endif
 #endif
 
 namespace __sanitizer {
 
+static char *DemangleAlloc(const char *name, bool always_alloc) {
+#if SANITIZER_LIBBACKTRACE && SANITIZER_CP_DEMANGLE
+  if (char *demangled = CplusV3Demangle(name))
+    return demangled;
+#endif
+  if (always_alloc)
+    return internal_strdup(name);
+  return 0;
+}
+
 #if SANITIZER_LIBBACKTRACE
 
 namespace {
 
 # if SANITIZER_CP_DEMANGLE
 struct CplusV3DemangleData {
   char *buf;
   uptr size, allocated;
if (data.buf)
     InternalFree(data.buf);
   return 0;
 }
 # endif  // SANITIZER_CP_DEMANGLE
 
 struct SymbolizeCodeCallbackArg {
   SymbolizedStack *first;
   SymbolizedStack *last;
-  const char *module_name;
-  uptr module_offset;
+  uptr frames_symbolized;
 
   void append(SymbolizedStack *f) {
     if (last != nullptr) {
       last->next = f;
       last = f;
     } else {
       first = f;
       last = f;
     }
   }
+
+  AddressInfo *get_new_frame(uintptr_t addr) {
+    SymbolizedStack *cur = last;
+    if (frames_symbolized > 0) {
+      cur = SymbolizedStack::New(addr);
+      AddressInfo *info = &cur->info;
+      info->FillAddressAndModuleInfo(addr, first->info.module,
+                                     first->info.module_offset);
+      append(cur);
+    } else {
+      CHECK_EQ(addr, cur->info.address);
+    }
+    return &cur->info;
+  }
 };
 
 extern "C" {
 static int SymbolizeCodePCInfoCallback(void *vdata, uintptr_t addr,
                                        const char *filename, int lineno,
                                        const char *function) {
   SymbolizeCodeCallbackArg *cdata = (SymbolizeCodeCallbackArg *)vdata;
   if (function) {
-    SymbolizedStack *cur = SymbolizedStack::New(addr);
-    cdata->append(cur);
-    AddressInfo *info = &cur->info;
-    info->FillAddressAndModuleInfo(addr, cdata->module_name,
-                                   cdata->module_offset);
-    info->function = LibbacktraceSymbolizer::Demangle(function, true);
+    AddressInfo *info = cdata->get_new_frame(addr);
+    info->function = DemangleAlloc(function, true);
     if (filename)
       info->file = internal_strdup(filename);
     info->line = lineno;
+    cdata->frames_symbolized++;
   }
   return 0;
 }
 
 static void SymbolizeCodeCallback(void *vdata, uintptr_t addr,
                                   const char *symname, uintptr_t, uintptr_t) {
   SymbolizeCodeCallbackArg *cdata = (SymbolizeCodeCallbackArg *)vdata;
   if (symname) {
-    SymbolizedStack *cur = SymbolizedStack::New(addr);
-    cdata->append(cur);
-    AddressInfo *info = &cur->info;
-    info->FillAddressAndModuleInfo(addr, cdata->module_name,
-                                   cdata->module_offset);
-    info->function = LibbacktraceSymbolizer::Demangle(symname, true);
+    AddressInfo *info = cdata->get_new_frame(addr);
+    info->function = DemangleAlloc(symname, true);
+    cdata->frames_symbolized++;
   }
 }
 
 static void SymbolizeDataCallback(void *vdata, uintptr_t, const char *symname,
                                   uintptr_t symval, uintptr_t symsize) {
   DataInfo *info = (DataInfo *)vdata;
   if (symname && symval) {
-    info->name = LibbacktraceSymbolizer::Demangle(symname, true);
+    info->name = DemangleAlloc(symname, true);
     info->start = symval;
     info->size = symsize;
   }
 }
 
 static void ErrorCallback(void *, const char *, int) {}
 }  // extern "C"
 
 }  // namespace
 
 LibbacktraceSymbolizer *LibbacktraceSymbolizer::get(LowLevelAllocator *alloc) {
   // State created in backtrace_create_state is leaked.
   void *state = (void *)(backtrace_create_state("/proc/self/exe", 0,
                                                 ErrorCallback, NULL));
   if (!state)
     return 0;
   return new(*alloc) LibbacktraceSymbolizer(state);
 }
 
-SymbolizedStack *LibbacktraceSymbolizer::SymbolizeCode(uptr addr,
-                                                       const char *module_name,
-                                                       uptr module_offset) {
+bool LibbacktraceSymbolizer::SymbolizePC(uptr addr, SymbolizedStack *stack) {
   SymbolizeCodeCallbackArg data;
-  data.first = nullptr;
-  data.last = nullptr;
-  data.module_name = module_name;
-  data.module_offset = module_offset;
+  data.first = stack;
+  data.last = stack;
+  data.frames_symbolized = 0;
   backtrace_pcinfo((backtrace_state *)state_, addr, SymbolizeCodePCInfoCallback,
                    ErrorCallback, &data);
-  if (data.first)
-    return data.first;
+  if (data.frames_symbolized > 0)
+    return true;
   backtrace_syminfo((backtrace_state *)state_, addr, SymbolizeCodeCallback,
                     ErrorCallback, &data);
-  return data.first;
+  return true;
 }
 
 bool LibbacktraceSymbolizer::SymbolizeData(uptr addr, DataInfo *info) {
   backtrace_syminfo((backtrace_state *)state_, addr, SymbolizeDataCallback,
                     ErrorCallback, info);
   return true;
 }
 
 #else  // SANITIZER_LIBBACKTRACE
 
 LibbacktraceSymbolizer *LibbacktraceSymbolizer::get(LowLevelAllocator *alloc) {
   return 0;
 }
 
-SymbolizedStack *LibbacktraceSymbolizer::SymbolizeCode(uptr addr,
-                                                       const char *module_name,
-                                                       uptr module_offset) {
+bool LibbacktraceSymbolizer::SymbolizePC(uptr addr, SymbolizedStack *stack) {
   (void)state_;
-  return nullptr;
+  return false;
 }
 
 bool LibbacktraceSymbolizer::SymbolizeData(uptr addr, DataInfo *info) {
   return false;
 }
 
 #endif  // SANITIZER_LIBBACKTRACE
 
-char *LibbacktraceSymbolizer::Demangle(const char *name, bool always_alloc) {
-#if SANITIZER_LIBBACKTRACE && SANITIZER_CP_DEMANGLE
-  if (char *demangled = CplusV3Demangle(name))
-    return demangled;
-#endif
-  if (always_alloc)
-    return internal_strdup(name);
-  return 0;
+const char *LibbacktraceSymbolizer::Demangle(const char *name) {
+  return DemangleAlloc(name, false);
 }
 
 }  // namespace __sanitizer

lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc

static const char *DemangleCXXABI(const char *name) {
   return name;
 }
 
 // Extracts the prefix of "str" that consists of any characters not
 // present in "delims" string, and copies this prefix to "result", allocating
 // space for it.
 // Returns a pointer to "str" after skipping extracted prefix and first
 // delimiter char.
-static const char *ExtractToken(const char *str, const char *delims,
-                                char **result) {
+const char *ExtractToken(const char *str, const char *delims, char **result) {
   uptr prefix_len = internal_strcspn(str, delims);
   *result = (char*)InternalAlloc(prefix_len + 1);
   internal_memcpy(*result, str, prefix_len);
   (*result)[prefix_len] = '\0';
   const char *prefix_end = str + prefix_len;
   if (*prefix_end != '\0') prefix_end++;
   return prefix_end;
 }
 
-// Same as ExtractToken, but converts extracted token to integer.
-static const char *ExtractInt(const char *str, const char *delims,
-                              int *result) {
-  char *buff;
-  const char *ret = ExtractToken(str, delims, &buff);
-  if (buff != 0) {
-    *result = (int)internal_atoll(buff);
-  }
-  InternalFree(buff);
-  return ret;
-}
-
-static const char *ExtractUptr(const char *str, const char *delims,
-                               uptr *result) {
+// Same as ExtractToken, but converts extracted token to uptr.
+const char *ExtractUptr(const char *str, const char *delims, uptr *result) {
   char *buff;
   const char *ret = ExtractToken(str, delims, &buff);
   if (buff != 0) {
     *result = (uptr)internal_atoll(buff);
   }
   InternalFree(buff);
   return ret;
 }
 
-class ExternalSymbolizerInterface {
- public:
-  // Can't declare pure virtual functions in sanitizer runtimes:
-  // __cxa_pure_virtual might be unavailable.
-  virtual char *SendCommand(bool is_data, const char *module_name,
-                            uptr module_offset) {
-    UNIMPLEMENTED();
-  }
-};
+// Similar to ExtractToken, but looks only for a single delimiter which can
+// be multiple characters long.
+const char *ExtractTokenUpToDelimiter(const char *str, const char *delimiter,
+                                      char **result) {
+  const char *found_delimiter = internal_strstr(str, delimiter);
+  uptr prefix_len =
+      found_delimiter ? found_delimiter - str : internal_strlen(str);
+  *result = (char *)InternalAlloc(prefix_len + 1);
+  internal_memcpy(*result, str, prefix_len);
+  (*result)[prefix_len] = '\0';
+  const char *prefix_end = str + prefix_len;
+  if (*prefix_end != '\0') prefix_end += internal_strlen(delimiter);
+  return prefix_end;
 
-// SymbolizerProcess encapsulates communication between the tool and
-// external symbolizer program, running in a different subprocess.
-// SymbolizerProcess may not be used from two threads simultaneously.
-class SymbolizerProcess : public ExternalSymbolizerInterface {
- public:
-  explicit SymbolizerProcess(const char *path)
-      : path_(path),
-        input_fd_(kInvalidFd),
-        output_fd_(kInvalidFd),
-        times_restarted_(0),
-        failed_to_start_(false),
-        reported_invalid_path_(false) {
-    CHECK(path_);
-    CHECK_NE(path_[0], '\0');
-  }
+SymbolizerProcess::SymbolizerProcess(const char *path)
 
-  char *SendCommand(bool is_data, const char *module_name, uptr module_offset) {
-    for (; times_restarted_ < kMaxTimesRestarted; times_restarted_++) {
-      // Start or restart symbolizer if we failed to send command to it.
-      if (char *res = SendCommandImpl(is_data, module_name, module_offset))
-        return res;
-      Restart();
-    }
-    if (!failed_to_start_) {
-      Report("WARNING: Failed to use and restart external symbolizer!\n");
-      failed_to_start_ = true;
-    }
-    return 0;
-  }
+char *SymbolizerProcess::SendCommand(const char *command) {
+    if (char *res = SendCommandImpl(command))
 
- private:
-  bool Restart() {
-    if (input_fd_ != kInvalidFd)
-      internal_close(input_fd_);
-    if (output_fd_ != kInvalidFd)
-      internal_close(output_fd_);
-    return StartSymbolizerSubprocess();
-  }
+bool SymbolizerProcess::Restart() {
 
-  char *SendCommandImpl(bool is_data, const char *module_name,
-                        uptr module_offset) {
-    if (input_fd_ == kInvalidFd || output_fd_ == kInvalidFd)
-      return 0;
-    CHECK(module_name);
-    if (!RenderInputCommand(buffer_, kBufferSize, is_data, module_name,
-                            module_offset))
-      return 0;
-    if (!writeToSymbolizer(buffer_, internal_strlen(buffer_)))
-      return 0;
-    if (!readFromSymbolizer(buffer_, kBufferSize))
-      return 0;
-    return buffer_;
-  }
+char *SymbolizerProcess::SendCommandImpl(const char *command) {
+  if (!WriteToSymbolizer(command, internal_strlen(command)))
+  if (!ReadFromSymbolizer(buffer_, kBufferSize))
 
-  bool readFromSymbolizer(char *buffer, uptr max_length) {
-    if (max_length == 0)
-      return true;
-    uptr read_len = 0;
-    while (true) {
-      uptr just_read = internal_read(input_fd_, buffer + read_len,
-                                     max_length - read_len - 1);
-      // We can't read 0 bytes, as we don't expect external symbolizer to close
-      // its stdout.
-      if (just_read == 0 || just_read == (uptr)-1) {
-        Report("WARNING: Can't read from symbolizer at fd %d\n", input_fd_);
-        return false;
-      }
-      read_len += just_read;
-      if (ReachedEndOfOutput(buffer, read_len))
-        break;
-    }
-    buffer[read_len] = '\0';
-    return true;
-  }
+bool SymbolizerProcess::ReadFromSymbolizer(char *buffer, uptr max_length) {
 
-  bool writeToSymbolizer(const char *buffer, uptr length) {
-    if (length == 0)
-      return true;
-    uptr write_len = internal_write(output_fd_, buffer, length);
-    if (write_len == 0 || write_len == (uptr)-1) {
-      Report("WARNING: Can't write to symbolizer at fd %d\n", output_fd_);
-      return false;
-    }
-    return true;
-  }
+bool SymbolizerProcess::WriteToSymbolizer(const char *buffer, uptr length) {
 
-  bool StartSymbolizerSubprocess() {
-    if (!FileExists(path_)) {
-      if (!reported_invalid_path_) {
-        Report("WARNING: invalid path to external symbolizer!\n");
-        reported_invalid_path_ = true;
-      }
-      return false;
-    }
+bool SymbolizerProcess::StartSymbolizerSubprocess() {
 
-    int *infd = NULL;
-    int *outfd = NULL;
-    // The client program may close its stdin and/or stdout and/or stderr
-    // thus allowing socketpair to reuse file descriptors 0, 1 or 2.
-    // In this case the communication between the forked processes may be
-    // broken if either the parent or the child tries to close or duplicate
-    // these descriptors. The loop below produces two pairs of file
-    // descriptors, each greater than 2 (stderr).
-    int sock_pair[5][2];
-    for (int i = 0; i < 5; i++) {
-      if (pipe(sock_pair[i]) == -1) {
-        for (int j = 0; j < i; j++) {
-          internal_close(sock_pair[j][0]);
-          internal_close(sock_pair[j][1]);
-        }
-        Report("WARNING: Can't create a socket pair to start "
-               "external symbolizer (errno: %d)\n", errno);
-        return false;
-      } else if (sock_pair[i][0] > 2 && sock_pair[i][1] > 2) {
-        if (infd == NULL) {
-          infd = sock_pair[i];
-        } else {
-          outfd = sock_pair[i];
-          for (int j = 0; j < i; j++) {
-            if (sock_pair[j] == infd) continue;
-            internal_close(sock_pair[j][0]);
-            internal_close(sock_pair[j][1]);
-          }
-          break;
-        }
-      }
-    }
-    CHECK(infd);
-    CHECK(outfd);
 
-    // Real fork() may call user callbacks registered with pthread_atfork().
-    int pid = internal_fork();
-    if (pid == -1) {
-      // Fork() failed.
-      internal_close(infd[0]);
-      internal_close(infd[1]);
-      internal_close(outfd[0]);
-      internal_close(outfd[1]);
-      Report("WARNING: failed to fork external symbolizer "
-             " (errno: %d)\n", errno);
-      return false;
-    } else if (pid == 0) {
-      // Child subprocess.
-      internal_close(STDOUT_FILENO);
-      internal_close(STDIN_FILENO);
-      internal_dup2(outfd[0], STDIN_FILENO);
-      internal_dup2(infd[1], STDOUT_FILENO);
-      internal_close(outfd[0]);
-      internal_close(outfd[1]);
-      internal_close(infd[0]);
-      internal_close(infd[1]);
-      for (int fd = sysconf(_SC_OPEN_MAX); fd > 2; fd--)
-        internal_close(fd);
-      ExecuteWithDefaultArgs(path_);
-      internal__exit(1);
-    }
 
-    // Continue execution in parent process.
-    internal_close(outfd[0]);
-    internal_close(infd[1]);
-    input_fd_ = infd[0];
-    output_fd_ = outfd[1];
 
-    // Check that symbolizer subprocess started successfully.
-    int pid_status;
-    SleepForMillis(kSymbolizerStartupTimeMillis);
-    int exited_pid = waitpid(pid, &pid_status, WNOHANG);
-    if (exited_pid != 0) {
-      // Either waitpid failed, or child has already exited.
-      Report("WARNING: external symbolizer didn't start up correctly!\n");
-      return false;
-    }
 
-    return true;
-  }
 
-  virtual bool RenderInputCommand(char *buffer, uptr max_length, bool is_data,
-                                  const char *module_name,
-                                  uptr module_offset) const {
-    UNIMPLEMENTED();
-  }
+// Parses a two-line string in the following format:
+//   <function_name>
+//   <file_name>:<line_number>:<column_number>
+// or this format (without the column number):
+//   <function_name>
+//   <file_name>:<line_number>
+// Used by LLVMSymbolizer, Addr2LinePool and InternalSymbolizer, since all of
+// them use the same output format.
+static void ParseSymbolizePCOutput(const char *str, SymbolizedStack *res) {
+  uptr orig_addr = res->info.address;
+  const char *orig_module_name = res->info.module;
+  uptr orig_module_offset = res->info.module_offset;
 
-  virtual bool ReachedEndOfOutput(const char *buffer, uptr length) const {
-    UNIMPLEMENTED();
+  bool top_frame = true;
+  SymbolizedStack *last = res;
+  while (true) {
+    char *function_name = 0;
+    str = ExtractToken(str, "\n", &function_name);
+    CHECK(function_name);
+    if (function_name[0] == '\0') {
+      // There are no more frames.
+      break;
     }
-
-  virtual void ExecuteWithDefaultArgs(const char *path_to_binary) const {
-    UNIMPLEMENTED();
+    SymbolizedStack *cur;
+    if (top_frame) {
+      cur = res;
+      top_frame = false;
+    } else {
+      cur = SymbolizedStack::New(orig_addr);
+      cur->info.FillAddressAndModuleInfo(orig_addr, orig_module_name,
+                                         orig_module_offset);
+      last->next = cur;
+      last = cur;
     }
 
-  const char *path_;
-  int input_fd_;
-  int output_fd_;
+    AddressInfo *info = &cur->info;
+    info->function = function_name;
+    // Parse <file>:<line>(:<column>) buffer.
+    char *file_line_info = 0;
+    str = ExtractToken(str, "\n", &file_line_info);
+    CHECK(file_line_info);
+    const char *line_info = ExtractToken(file_line_info, ":", &info->file);
+    uptr line_number;
+    line_info = ExtractUptr(line_info, ":", &line_number);
+    info->line = line_number;
+    if (*line_info != '\0') {
+      uptr column_number;
+      line_info = ExtractUptr(line_info, "", &column_number);
+      info->column = column_number;
+    }
+    InternalFree(file_line_info);
 
-  static const uptr kBufferSize = 16 * 1024;
-  char buffer_[kBufferSize];
+    // Functions and filenames can be "??", in which case we write 0
+    // to address info to mark that names are unknown.
+    if (0 == internal_strcmp(info->function, "??")) {
+      InternalFree(info->function);
+      info->function = 0;
+    }
+    if (0 == internal_strcmp(info->file, "??")) {
+      InternalFree(info->file);
+      info->file = 0;
+    }
+  }
+}
 
-  static const uptr kMaxTimesRestarted = 5;
-  static const int kSymbolizerStartupTimeMillis = 10;
-  uptr times_restarted_;
-  bool failed_to_start_;
-  bool reported_invalid_path_;
-};
+// Parses a two-line string in the following format:
+//   <symbol_name>
+//   <start_address> <size>
+// Used by LLVMSymbolizer and InternalSymbolizer.
+static void ParseSymbolizeDataOutput(const char *str, uptr addr,
+                                     DataInfo *info) {
+  str = ExtractToken(str, "\n", &info->name);
+  str = ExtractUptr(str, " ", &info->start);
+  str = ExtractUptr(str, "\n", &info->size);
+  info->start += (addr - info->module_offset);  // Add the module base address.
+}
 
 // For now we assume the following protocol:
 // For each request of the form
 //   <module_name> <module_offset>
 // passed to STDIN, external symbolizer prints to STDOUT response:
 //   <function_name>
 //   <file_name>:<line_number>:<column_number>
 //   <function_name>
 //   <file_name>:<line_number>:<column_number>
 //   ...
 //   <empty line>
 class LLVMSymbolizerProcess : public SymbolizerProcess {
  public:
   explicit LLVMSymbolizerProcess(const char *path) : SymbolizerProcess(path) {}
 
- private:
-  bool RenderInputCommand(char *buffer, uptr max_length, bool is_data,
-                          const char *module_name, uptr module_offset) const {
-    internal_snprintf(buffer, max_length, "%s\"%s\" 0x%zx\n",
-                      is_data ? "DATA " : "", module_name, module_offset);
-    return true;
-  }
-
-  bool ReachedEndOfOutput(const char *buffer, uptr length) const {
+ protected:
+  bool ReachedEndOfOutput(const char *buffer, uptr length) const override {
     // Empty line marks the end of llvm-symbolizer output.
     return length >= 2 && buffer[length - 1] == '\n' &&
            buffer[length - 2] == '\n';
   }
 
-  void ExecuteWithDefaultArgs(const char *path_to_binary) const {
+  void ExecuteWithDefaultArgs(const char *path_to_binary) const override {
 #if defined(__x86_64__)
     const char* const kSymbolizerArch = "--default-arch=x86_64";
 #elif defined(__i386__)
     const char* const kSymbolizerArch = "--default-arch=i386";
 #elif defined(__powerpc64__) && defined(__BIG_ENDIAN__)
     const char* const kSymbolizerArch = "--default-arch=powerpc64";
 #elif defined(__powerpc64__) && defined(__LITTLE_ENDIAN__)
     const char* const kSymbolizerArch = "--default-arch=powerpc64le";
 #else
     const char* const kSymbolizerArch = "--default-arch=unknown";
 #endif
 
     const char *const inline_flag = common_flags()->symbolize_inline_frames
                                         ? "--inlining=true"
                                         : "--inlining=false";
     execl(path_to_binary, path_to_binary, inline_flag, kSymbolizerArch,
           (char *)0);
   }
 };
 
+class LLVMSymbolizer : public SymbolizerInterface {
+ public:
+  explicit LLVMSymbolizer(const char *path, LowLevelAllocator *allocator)
+      : symbolizer_process_(new(*allocator) LLVMSymbolizerProcess(path)) {}
+
+  bool SymbolizePC(uptr addr, SymbolizedStack *stack) override {
+    const char *module_name = stack->info.module;
+    uptr module_offset = stack->info.module_offset;
+    char *buffer =
+        RenderInputCommand(/*is_data*/ false, module_name, module_offset);
+    char *buf = symbolizer_process_->SendCommand(buffer);
+    if (!buf)
+      return false;
+    ParseSymbolizePCOutput(buf, stack);
+    return true;
+  }
+
+  bool SymbolizeData(uptr addr, DataInfo *info) override {
+    const char *module_name = info->module;
+    uptr module_offset = info->module_offset;
+    char *buffer =
+        RenderInputCommand(/*is_data*/ true, module_name, module_offset);
+    const char *str = symbolizer_process_->SendCommand(buffer);
+    if (str == 0)
+      return true;
+    ParseSymbolizeDataOutput(str, addr, info);
+    return true;
+  }
+
+ private:
+  char *RenderInputCommand(bool is_data, const char *module_name,
+                           uptr module_offset) {
+    internal_snprintf(buffer_, kBufferSize, "%s\"%s\" 0x%zx\n",
+                      is_data ? "DATA " : "", module_name, module_offset);
+    return buffer_;
+  }
+
+  LLVMSymbolizerProcess *symbolizer_process_;
+
+  static const uptr kBufferSize = 16 * 1024;
+  char buffer_[kBufferSize];
+};
+
 class Addr2LineProcess : public SymbolizerProcess {
  public:
   Addr2LineProcess(const char *path, const char *module_name)
       : SymbolizerProcess(path), module_name_(internal_strdup(module_name)) {}
 
   const char *module_name() const { return module_name_; }
 
  private:
-  bool RenderInputCommand(char *buffer, uptr max_length, bool is_data,
-                          const char *module_name, uptr module_offset) const {
-    if (is_data)
-      return false;
-    CHECK_EQ(0, internal_strcmp(module_name, module_name_));
-    internal_snprintf(buffer, max_length, "0x%zx\n", module_offset);
-    return true;
-  }
-
-  bool ReachedEndOfOutput(const char *buffer, uptr length) const {
+  bool ReachedEndOfOutput(const char *buffer, uptr length) const override {
     // Output should consist of two lines.
     int num_lines = 0;
     for (uptr i = 0; i < length; ++i) {
       if (buffer[i] == '\n')
         num_lines++;
       if (num_lines >= 2)
         return true;
     }
     return false;
   }
 
-  void ExecuteWithDefaultArgs(const char *path_to_binary) const {
+  void ExecuteWithDefaultArgs(const char *path_to_binary) const override {
     execl(path_to_binary, path_to_binary, "-Cfe", module_name_, (char *)0);
   }
 
   const char *module_name_;  // Owned, leaked.
 };
 
-class Addr2LinePool : public ExternalSymbolizerInterface {
+class Addr2LinePool : public SymbolizerInterface {
  public:
   explicit Addr2LinePool(const char *addr2line_path,
                          LowLevelAllocator *allocator)
       : addr2line_path_(addr2line_path), allocator_(allocator),
         addr2line_pool_(16) {}
 
-  char *SendCommand(bool is_data, const char *module_name, uptr module_offset) {
-    if (is_data)
-      return 0;
+  bool SymbolizePC(uptr addr, SymbolizedStack *stack) override {
+    const char *module_name = stack->info.module;
+    uptr module_offset = stack->info.module_offset;
+
     Addr2LineProcess *addr2line = 0;
     for (uptr i = 0; i < addr2line_pool_.size(); ++i) {
       if (0 ==
           internal_strcmp(module_name, addr2line_pool_[i]->module_name())) {
         addr2line = addr2line_pool_[i];
         break;
       }
     }
     if (!addr2line) {
       addr2line =
           new(*allocator_) Addr2LineProcess(addr2line_path_, module_name);
       addr2line_pool_.push_back(addr2line);
     }
-    return addr2line->SendCommand(is_data, module_name, module_offset);
+
+    CHECK_EQ(0, internal_strcmp(module_name, addr2line->module_name()));
+    char *req = RenderInputCommand(module_name, module_offset);
+    char *buf = addr2line->SendCommand(req);
+    if (!buf)
+      return false;
+    ParseSymbolizePCOutput(buf, stack);
+    return true;
+  }
+
+  bool SymbolizeData(uptr addr, DataInfo *info) override {
+    return false;
   }
 
  private:
+  char *RenderInputCommand(const char *module_name, uptr module_offset) {
+    internal_snprintf(buffer_, kBufferSize, "0x%zx\n", module_offset);
+    return buffer_;
+  }
+
   const char *addr2line_path_;
   LowLevelAllocator *allocator_;
   InternalMmapVector<Addr2LineProcess*> addr2line_pool_;
+
+  static const uptr kBufferSize = 16 * 1024;
+  char buffer_[kBufferSize];
 };
 
 #if SANITIZER_SUPPORTS_WEAK_HOOKS
 extern "C" {
 SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE
 bool __sanitizer_symbolize_code(const char *ModuleName, u64 ModuleOffset,
                                 char *Buffer, int MaxLength);
 SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE
 bool __sanitizer_symbolize_data(const char *ModuleName, u64 ModuleOffset,
                                 char *Buffer, int MaxLength);
 SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE
 void __sanitizer_symbolize_flush();
 SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE
 int __sanitizer_symbolize_demangle(const char *Name, char *Buffer,
                                    int MaxLength);
 }  // extern "C"
 
-class InternalSymbolizer {
+class InternalSymbolizer : public SymbolizerInterface {
  public:
   typedef bool (*SanitizerSymbolizeFn)(const char*, u64, char*, int);
 
   static InternalSymbolizer *get(LowLevelAllocator *alloc) {
     if (__sanitizer_symbolize_code != 0 &&
         __sanitizer_symbolize_data != 0) {
       return new(*alloc) InternalSymbolizer();
     }
     return 0;
   }
 
-  char *SendCommand(bool is_data, const char *module_name, uptr module_offset) {
-    SanitizerSymbolizeFn symbolize_fn = is_data ? __sanitizer_symbolize_data
-                                                : __sanitizer_symbolize_code;
-    if (symbolize_fn(module_name, module_offset, buffer_, kBufferSize))
-      return buffer_;
-    return 0;
+  bool SymbolizePC(uptr addr, SymbolizedStack *stack) override {
+    char *buf = SymbolizePCOrData(/*is_data*/ false, stack->info.module,
+                                  stack->info.module_offset);
+    if (!buf)
+      return false;
+    ParseSymbolizePCOutput(buf, stack);
+    return true;
   }
 
-  void Flush() {
+  bool SymbolizeData(uptr addr, DataInfo *info) override {
+    char *buf =
+        SymbolizePCOrData(/*is_data*/ true, info->module, info->module_offset);
+    if (!buf)
+      return false;
+    ParseSymbolizeDataOutput(buf, addr, info);
+    return true;
+  }
+
+  void Flush() override {
     if (__sanitizer_symbolize_flush)
       __sanitizer_symbolize_flush();
   }
 
-  const char *Demangle(const char *name) {
+  const char *Demangle(const char *name) override {
     if (__sanitizer_symbolize_demangle) {
       for (uptr res_length = 1024;
            res_length <= InternalSizeClassMap::kMaxSize;) {
         char *res_buff = static_cast<char*>(InternalAlloc(res_length));
         uptr req_length =
             __sanitizer_symbolize_demangle(name, res_buff, res_length);
         if (req_length > res_length) {
           res_length = req_length + 1;
           InternalFree(res_buff);
           continue;
         }
         return res_buff;
       }
     }
     return name;
   }
 
  private:
   InternalSymbolizer() { }
 
+  char *SymbolizePCOrData(bool is_data, const char *module_name,
+                          uptr module_offset) {
+    SanitizerSymbolizeFn symbolize_fn =
+        is_data ? __sanitizer_symbolize_data : __sanitizer_symbolize_code;
+    if (symbolize_fn(module_name, module_offset, buffer_, kBufferSize))
+      return buffer_;
+    return 0;
+  }
+
   static const int kBufferSize = 16 * 1024;
   static const int kMaxDemangledNameSize = 1024;
   char buffer_[kBufferSize];
 };
 #else  // SANITIZER_SUPPORTS_WEAK_HOOKS
 
-class InternalSymbolizer {
+class InternalSymbolizer : public SymbolizerInterface {
  public:
   static InternalSymbolizer *get(LowLevelAllocator *alloc) { return 0; }
-  char *SendCommand(bool is_data, const char *module_name, uptr module_offset) {
-    return 0;
+  bool SymbolizePC(uptr addr, SymbolizedStack *stack) override {
+    return false;
+  }
+  bool SymbolizeData(uptr addr, DataInfo *info) override {
+    return false;
   }
-  void Flush() { }
-  const char *Demangle(const char *name) { return name; }
 };
 
 #endif  // SANITIZER_SUPPORTS_WEAK_HOOKS
 
 class POSIXSymbolizer : public Symbolizer {
  public:
-  POSIXSymbolizer(ExternalSymbolizerInterface *external_symbolizer,
-                  InternalSymbolizer *internal_symbolizer,
-                  LibbacktraceSymbolizer *libbacktrace_symbolizer)
-      : Symbolizer(),
-        external_symbolizer_(external_symbolizer),
-        internal_symbolizer_(internal_symbolizer),
-        libbacktrace_symbolizer_(libbacktrace_symbolizer) {}
+  explicit POSIXSymbolizer(SymbolizerInterface *first_symbolizer)
+      : Symbolizer(), first_symbolizer_(first_symbolizer) {}
 
   SymbolizedStack *SymbolizePC(uptr addr) override {
     BlockingMutexLock l(&mu_);
+
+    // Always fill data about module name and offset.
     const char *module_name;
     uptr module_offset;
+    SymbolizedStack *res = SymbolizedStack::New(addr);
     if (!FindModuleNameAndOffsetForAddress(addr, &module_name, &module_offset))
-      return SymbolizedStack::New(addr);
-    // First, try to use libbacktrace symbolizer (if it's available).
-    if (libbacktrace_symbolizer_ != 0) {
-      mu_.CheckLocked();
-      if (SymbolizedStack *res = libbacktrace_symbolizer_->SymbolizeCode(
-              addr, module_name, module_offset))
-        return res;
-    }
-    // Always fill data about module name and offset.
-    SymbolizedStack *res = SymbolizedStack::New(addr);
     res->info.FillAddressAndModuleInfo(addr, module_name, module_offset);
 
-    const char *str = SendCommand(false, module_name, module_offset);
-    if (str == 0) {
-      // Symbolizer was not initialized or failed.
+    SymbolizerScope sym_scope(this);
+    SymbolizerInterface *symbolizer = first_symbolizer_;
+    while (symbolizer) {
+      if (symbolizer->SymbolizePC(addr, res))
         return res;
+      symbolizer = symbolizer->next_symbolizer_;
     }
 
-    bool top_frame = true;
-    SymbolizedStack *last = res;
-    while (true) {
-      char *function_name = 0;
-      str = ExtractToken(str, "\n", &function_name);
-      CHECK(function_name);
-      if (function_name[0] == '\0') {
-        // There are no more frames.
-        break;
-      }
-      SymbolizedStack *cur;
-      if (top_frame) {
-        cur = res;
-        top_frame = false;
-      } else {
-        cur = SymbolizedStack::New(addr);
-        cur->info.FillAddressAndModuleInfo(addr, module_name, module_offset);
-        last->next = cur;
-        last = cur;
-      }
-
-      AddressInfo *info = &cur->info;
-      info->function = function_name;
-      // Parse <file>:<line>:<column> buffer.
-      char *file_line_info = 0;
-      str = ExtractToken(str, "\n", &file_line_info);
-      CHECK(file_line_info);
-      const char *line_info = ExtractToken(file_line_info, ":", &info->file);
-      line_info = ExtractInt(line_info, ":", &info->line);
-      line_info = ExtractInt(line_info, "", &info->column);
-      InternalFree(file_line_info);
-
-      // Functions and filenames can be "??", in which case we write 0
-      // to address info to mark that names are unknown.
-      if (0 == internal_strcmp(info->function, "??")) {
-        InternalFree(info->function);
-        info->function = 0;
-      }
-      if (0 == internal_strcmp(info->file, "??")) {
-        InternalFree(info->file);
-        info->file = 0;
-      }
-    }
     return res;
   }
 
   bool SymbolizeData(uptr addr, DataInfo *info) override {
     BlockingMutexLock l(&mu_);
     LoadedModule *module = FindModuleForAddress(addr);
     if (module == 0)
       return false;
     const char *module_name = module->full_name();
     uptr module_offset = addr - module->base_address();
     info->Clear();
     info->module = internal_strdup(module_name);
     info->module_offset = module_offset;
-    // First, try to use libbacktrace symbolizer (if it's available).
-    if (libbacktrace_symbolizer_ != 0) {
-      mu_.CheckLocked();
-      if (libbacktrace_symbolizer_->SymbolizeData(addr, info))
+
+    SymbolizerScope sym_scope(this);
+    SymbolizerInterface *symbolizer = first_symbolizer_;
+    while (symbolizer) {
+      if (symbolizer->SymbolizeData(addr, info))
         return true;
+      symbolizer = symbolizer->next_symbolizer_;
     }
-    const char *str = SendCommand(true, module_name, module_offset);
-    if (str == 0)
-      return true;
-    str = ExtractToken(str, "\n", &info->name);
-    str = ExtractUptr(str, " ", &info->start);
-    str = ExtractUptr(str, "\n", &info->size);
-    info->start += module->base_address();
-    return true;
+
+    return false;
   }
 
   bool GetModuleNameAndOffsetForPC(uptr pc, const char **module_name,
                                    uptr *module_address) override {
     BlockingMutexLock l(&mu_);
     return FindModuleNameAndOffsetForAddress(pc, module_name, module_address);
   }
 
   bool CanReturnFileLineInfo() override {
-    return internal_symbolizer_ != 0 || external_symbolizer_ != 0 ||
-           libbacktrace_symbolizer_ != 0;
+    return first_symbolizer_ != nullptr;
   }
 
   void Flush() override {
     BlockingMutexLock l(&mu_);
-    if (internal_symbolizer_ != 0) {
-      SymbolizerScope sym_scope(this);
-      internal_symbolizer_->Flush();
+    SymbolizerInterface *symbolizer = first_symbolizer_;
+    while (symbolizer) {
+      symbolizer->Flush();
+      symbolizer = symbolizer->next_symbolizer_;
     }
   }
 
   const char *Demangle(const char *name) override {
     BlockingMutexLock l(&mu_);
     // Run hooks even if we don't use internal symbolizer, as cxxabi
     // demangle may call system functions.
     SymbolizerScope sym_scope(this);
-    // Try to use libbacktrace demangler (if available).
-    if (libbacktrace_symbolizer_ != 0) {
-      if (const char *demangled = libbacktrace_symbolizer_->Demangle(name))
+    SymbolizerInterface *symbolizer = first_symbolizer_;
+    while (symbolizer) {
+      if (const char *demangled = symbolizer->Demangle(name))
         return demangled;
+      symbolizer = symbolizer->next_symbolizer_;
     }
-    if (internal_symbolizer_ != 0)
-      return internal_symbolizer_->Demangle(name);
     return DemangleCXXABI(name);
   }
 
   void PrepareForSandboxing() override {
 #if SANITIZER_LINUX && !SANITIZER_ANDROID
     BlockingMutexLock l(&mu_);
     // Cache /proc/self/exe on Linux.
     CacheBinaryName();
 #endif
   }
 
  private:
-  char *SendCommand(bool is_data, const char *module_name, uptr module_offset) {
-    mu_.CheckLocked();
-    // First, try to use internal symbolizer.
-    if (internal_symbolizer_) {
-      SymbolizerScope sym_scope(this);
-      return internal_symbolizer_->SendCommand(is_data, module_name,
-                                               module_offset);
-    }
-    // Otherwise, fall back to external symbolizer.
-    if (external_symbolizer_) {
-      SymbolizerScope sym_scope(this);
-      return external_symbolizer_->SendCommand(is_data, module_name,
-                                               module_offset);
-    }
-    return 0;
-  }
-
   LoadedModule *FindModuleForAddress(uptr address) {
     mu_.CheckLocked();
     bool modules_were_reloaded = false;
     if (modules_ == 0 || !modules_fresh_) {
       modules_ = (LoadedModule*)(symbolizer_allocator_.Allocate(
           kMaxNumberOfModuleContexts * sizeof(LoadedModule)));
       CHECK(modules_);
       n_modules_ = GetListOfModules(modules_, kMaxNumberOfModuleContexts,
private:
   // 16K loaded modules should be enough for everyone.
   static const uptr kMaxNumberOfModuleContexts = 1 << 14;
   LoadedModule *modules_;  // Array of module descriptions is leaked.
   uptr n_modules_;
   // If stale, need to reload the modules before looking up addresses.
   bool modules_fresh_;
   BlockingMutex mu_;
 
-  ExternalSymbolizerInterface *external_symbolizer_;  // Leaked.
-  InternalSymbolizer *const internal_symbolizer_;     // Leaked.
-  LibbacktraceSymbolizer *libbacktrace_symbolizer_;   // Leaked.
+  SymbolizerInterface *first_symbolizer_;          // Leaked.
 };
 
+static void AddSymbolizerIntoChain(SymbolizerInterface **first,
+                                   SymbolizerInterface **last,
+                                   SymbolizerInterface *to_add) {
+  if (*first == nullptr) {
+    *first = to_add;
+    *last = to_add;
+    return;
+  }
+
+  (*last)->next_symbolizer_ = to_add;
+  *last = to_add;
+}
+
 Symbolizer *Symbolizer::PlatformInit() {
   if (!common_flags()->symbolize) {
-    return new(symbolizer_allocator_) POSIXSymbolizer(0, 0, 0);
+    VReport(2, "Symbolizing is disabled.\n");
+    return new(symbolizer_allocator_) POSIXSymbolizer(nullptr);
   }
-  InternalSymbolizer* internal_symbolizer =
+
+  SymbolizerInterface *sym_first = nullptr;
+  SymbolizerInterface *sym_last = nullptr;
+
+  InternalSymbolizer *internal_symbolizer =
       InternalSymbolizer::get(&symbolizer_allocator_);
-  ExternalSymbolizerInterface *external_symbolizer = 0;
-  LibbacktraceSymbolizer *libbacktrace_symbolizer = 0;
+  if (internal_symbolizer) {
+    VReport(2, "Using internal symbolizer.\n");
+    AddSymbolizerIntoChain(&sym_first, &sym_last, internal_symbolizer);
+  }
 
-  if (!internal_symbolizer) {
-    libbacktrace_symbolizer =
-        LibbacktraceSymbolizer::get(&symbolizer_allocator_);
-    if (!libbacktrace_symbolizer) {
-      const char *path_to_external = common_flags()->external_symbolizer_path;
-      if (path_to_external && path_to_external[0] == '\0') {
-        // External symbolizer is explicitly disabled. Do nothing.
-      } else {
-        // Find path to llvm-symbolizer if it's not provided.
-        if (!path_to_external)
-          path_to_external = FindPathToBinary("llvm-symbolizer");
-        if (path_to_external) {
-          external_symbolizer = new(symbolizer_allocator_)
-              LLVMSymbolizerProcess(path_to_external);
-        } else if (common_flags()->allow_addr2line) {
-          // If llvm-symbolizer is not found, try to use addr2line.
-          if (const char *addr2line_path = FindPathToBinary("addr2line")) {
-            external_symbolizer = new(symbolizer_allocator_)
-                Addr2LinePool(addr2line_path, &symbolizer_allocator_);
-          }
-        }
-      }
-    }
+  LibbacktraceSymbolizer *libbacktrace_symbolizer =
+  if (libbacktrace_symbolizer) {
+    VReport(2, "Using libbacktrace symbolizer.\n");
+    AddSymbolizerIntoChain(&sym_first, &sym_last, libbacktrace_symbolizer);
+  }
+
+  SymbolizerInterface *external_symbolizer = nullptr;
+  if (!(path_to_external && path_to_external[0] == '\0')) {
+      VReport(2, "Using llvm-symbolizer at path: %s\n", path_to_external);
+          LLVMSymbolizer(path_to_external, &symbolizer_allocator_);
+        VReport(2, "Using addr2line at path: %s\n", addr2line_path);
+
+  if (external_symbolizer) {
+    AddSymbolizerIntoChain(&sym_first, &sym_last, external_symbolizer);
   }
 
-  return new(symbolizer_allocator_) POSIXSymbolizer(
-      external_symbolizer, internal_symbolizer, libbacktrace_symbolizer);
+  return new(symbolizer_allocator_) POSIXSymbolizer(sym_first);
 }
 
 }  // namespace __sanitizer
 
 #endif  // SANITIZER_POSIX