[scudo] Adding a public Scudo interface

[cryptoad]

Summary

The first and only function to start with allows to set the soft or hard RSS
limit at runtime. Add associated tests.

[cryptoad]

I should add here that this is required for programs wanting to set their RSS limit based on computations done at runtime due to hardware constraints or other factors.
From a security perspective, this could have an impact if called with parameters controlled by a malicious user, or if set too low by a program (memory exhaustion exploitation is on the rise).

[alekseyshl]

To address your security concern, maybe have a define to compile Scudo without public API?

[cryptoad]

Added some words about security in the interface comment.
Added a define to allow the disabling of the function.

[alekseyshl]

Maybe add a test case for !SCUDO_CAN_USE_PUBLIC_INTERFACE?

[cryptoad]

Adding a TODO to add tests for the various defines.
Currently the tests we run leverage a static library built with a default
configuration. We should add gtests like the other sanitizers to ensure all
the compile time configuration options work as expected.

[Hahnfeld]

The added test has been disabled on armhf in rCRT320665 because it fails a bot: http://lab.llvm.org:8011/builders/clang-cmake-thumbv7-a15-full-sh/builds/2952/steps/ninja%20check%202/logs/FAIL%3A%20Scudo-armhf%3A%3A%20interface.cpp

I see the exact same error on x86_64 CentOS 7, did you find out what the problem is? I'm happy to assist if you have problems to reproduce the problem...

[cryptoad]

In D41128#964426, @Hahnfeld wrote:

The added test has been disabled on armhf in rCRT320665 because it fails a bot: http://lab.llvm.org:8011/builders/clang-cmake-thumbv7-a15-full-sh/builds/2952/steps/ninja%20check%202/logs/FAIL%3A%20Scudo-armhf%3A%3A%20interface.cpp

I see the exact same error on x86_64 CentOS 7, did you find out what the problem is? I'm happy to assist if you have problems to reproduce the problem...

I haven't been able to repro the issue. If you have a configuration or some insight as to what the failure is, please let me know!

[Hahnfeld]

In D41128#965450, @cryptoad wrote:

In D41128#964426, @Hahnfeld wrote:

The added test has been disabled on armhf in rCRT320665 because it fails a bot: http://lab.llvm.org:8011/builders/clang-cmake-thumbv7-a15-full-sh/builds/2952/steps/ninja%20check%202/logs/FAIL%3A%20Scudo-armhf%3A%3A%20interface.cpp

I see the exact same error on x86_64 CentOS 7, did you find out what the problem is? I'm happy to assist if you have problems to reproduce the problem...

I haven't been able to repro the issue. If you have a configuration or some insight as to what the failure is, please let me know!

Any normal build on CentOS 7 failed reliably. I've submitted D41649 which solves the problem for me, I'll see if this also fixes the failure on ARM...

Side note: Do I understand this correctly that scudo will only get active on an intercepted malloc and exceeded limits without a malloc have no effect? That's a huge restriction, probably most applications in the HPC world first allocate all structures and only start touching the memory afterwards...

[cryptoad]

In D41128#965467, @Hahnfeld wrote:

Any normal build on CentOS 7 failed reliably. I've submitted D41649 which solves the problem for me, I'll see if this also fixes the failure on ARM...

Cool thanks! Will have look.

Side note: Do I understand this correctly that scudo will only get active on an intercepted malloc and exceeded limits without a malloc have no effect? That's a huge restriction, probably most applications in the HPC world first allocate all structures and only start touching the memory afterwards...

I am not sure I understand correctly the question. So I will attempt an answer, and if it doesn't apply, let me know.

If compiled with Scudo, all allocation functions are handled by the allocator (malloc, realloc, calloc, memalign, etc).
The soft/hard RSS limit feature comes from a need from some applications to enforce an upper limit to the memory used.
This is mostly needed due to the fact that we reserve a large portion of the address space, and as such, setting an rlimit is not viable.
The RSS limit comes as an opportunistic, best-effort, check to allow for some type of upper bound check.
It is not enabled by default (eg: 0), and it is expansive (a few syscalls to read from /proc).
If an application uses directly mmap (or any OS backed memory allocation function), we would only catch a limit "overflow" on the next use of a heap allocation function.
Also if we allocate/free some memory backed by the Secondary allocator quickly (before the check interval elapses), we wouldn't catch it as well.

I hope this answers your question.

[cryptoad]

Another point:
For ASan, an identical check exists as a background thread. We decided not to do that due to some requirements on Android.
It could potentially come back as a background thread check on supported platforms if that would address the concerns you are raising.

[Hahnfeld]

In D41128#965469, @cryptoad wrote:

I am not sure I understand correctly the question. So I will attempt an answer, and if it doesn't apply, let me know.

If compiled with Scudo, all allocation functions are handled by the allocator (malloc, realloc, calloc, memalign, etc).
The soft/hard RSS limit feature comes from a need from some applications to enforce an upper limit to the memory used.
This is mostly needed due to the fact that we reserve a large portion of the address space, and as such, setting an rlimit is not viable.
The RSS limit comes as an opportunistic, best-effort, check to allow for some type of upper bound check.
It is not enabled by default (eg: 0), and it is expansive (a few syscalls to read from /proc).
If an application uses directly mmap (or any OS backed memory allocation function), we would only catch a limit "overflow" on the next use of a heap allocation function.
Also if we allocate/free some memory backed by the Secondary allocator quickly (before the check interval elapses), we wouldn't catch it as well.

I hope this answers your question.

Yes, that confirms my understanding that the following logic wouldn't be catched:

1. Call malloc and allocate all arrays that will be needed.
2. Initialize the arrays - this commits the memory to count as RSS.
3. (Computation on arrays)
4. free the arrays.

In D41128#965471, @cryptoad wrote:

Another point:
For ASan, an identical check exists as a background thread. We decided not to do that due to some requirements on Android.
It could potentially come back as a background thread check on supported platforms if that would address the concerns you are raising.

This could work although I'm only trying to understand what this sanitizer is able to detect.

include/CMakeLists.txt

 if (COMPILER_RT_BUILD_SANITIZERS)
   set(SANITIZER_HEADERS
     sanitizer/allocator_interface.h
     sanitizer/asan_interface.h
     sanitizer/common_interface_defs.h
     sanitizer/coverage_interface.h
     sanitizer/dfsan_interface.h
     sanitizer/esan_interface.h
     sanitizer/hwasan_interface.h
     sanitizer/linux_syscall_hooks.h
     sanitizer/lsan_interface.h
     sanitizer/msan_interface.h
+    sanitizer/scudo_interface.h
     sanitizer/tsan_interface.h
     sanitizer/tsan_interface_atomic.h)
 endif(COMPILER_RT_BUILD_SANITIZERS)
 
 if (COMPILER_RT_BUILD_XRAY)
   set(XRAY_HEADERS
     xray/xray_interface.h
     xray/xray_log_interface.h)

include/sanitizer/scudo_interface.h

+//===-- sanitizer/scudo_interface.h -----------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+/// Public Scudo interface header.
+//
+//===----------------------------------------------------------------------===//
+#ifndef SANITIZER_SCUDO_INTERFACE_H_
+#define SANITIZER_SCUDO_INTERFACE_H_
+
+#include <sanitizer/common_interface_defs.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+  // This function may be optionally provided by a user and should return
+  // a string containing Scudo runtime options. See scudo_flags.h for details.
+  const char* __scudo_default_options();
+
+  // This function allows to set the RSS limit at runtime. This can be either
+  // the hard limit (HardLimit=1) or the soft limit (HardLimit=0). The limit
+  // can be removed by setting LimitMb to 0. This function's parameters should
+  // be fully trusted to avoid security mishaps.
+  void __scudo_set_rss_limit(unsigned long LimitMb, int HardLimit);
+#ifdef __cplusplus
+}  // extern "C"
+#endif
+
+#endif  // SANITIZER_SCUDO_INTERFACE_H_

lib/scudo/scudo_allocator.cpp

uptr getStats(AllocatorStat StatType) {
     BackendAllocator.getStats(stats);
     return stats[StatType];
   }
 
   void *handleBadRequest() {
     initThreadMaybe();
     return FailureHandler::OnBadRequest();
   }
+
+  void setRssLimit(uptr LimitMb, bool HardLimit) {
+    if (HardLimit)
+      HardRssLimitMb = LimitMb;
+    else
+      SoftRssLimitMb = LimitMb;
+    CheckRssLimit = HardRssLimitMb || SoftRssLimitMb;
+  }
 };
 
 static ScudoAllocator Instance(LINKER_INITIALIZED);
 
 static ScudoBackendAllocator &getBackendAllocator() {
   return Instance.BackendAllocator;
 }
 

 
 int __sanitizer_get_ownership(const void *Ptr) {
   return Instance.isValidPointer(Ptr);
 }
 
 uptr __sanitizer_get_allocated_size(const void *Ptr) {
   return Instance.getUsableSize(Ptr);
 }
+
+// Interface functions
+
+extern "C" {
+void __scudo_set_rss_limit(unsigned long LimitMb, int HardLimit) {  // NOLINT
+  if (!SCUDO_CAN_USE_PUBLIC_INTERFACE)
+    return;
+  Instance.setRssLimit(LimitMb, !!HardLimit);
+}
+}  // extern "C"

lib/scudo/scudo_interface_internal.h

+//===-- scudo_interface_internal.h ------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// Private Scudo interface header.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef SCUDO_INTERFACE_INTERNAL_H_
+#define SCUDO_INTERFACE_INTERNAL_H_
+
+extern "C" {
+SANITIZER_INTERFACE_ATTRIBUTE
+void __scudo_set_rss_limit(unsigned long LimitMb, int HardLimit);  // NOLINT
+}  // extern "C"
+
+#endif  // SCUDO_INTERFACE_INTERNAL_H_

lib/scudo/scudo_platform.h

 //===-- scudo_platform.h ----------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 ///
 /// Scudo platform specific definitions.
+/// TODO(kostyak): add tests for the compile time defines.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef SCUDO_PLATFORM_H_
 #define SCUDO_PLATFORM_H_
 
 #include "sanitizer_common/sanitizer_allocator.h"
 

 # error "The exclusive TSD model is not supported on this platform."
 #endif
 
 // Maximum number of TSDs that can be created for the Shared model.
 #ifndef SCUDO_SHARED_TSD_POOL_SIZE
 # define SCUDO_SHARED_TSD_POOL_SIZE 32U
 #endif  // SCUDO_SHARED_TSD_POOL_SIZE
 
+// The following allows the public interface functions to be disabled.
+#ifndef SCUDO_CAN_USE_PUBLIC_INTERFACE
+# define SCUDO_CAN_USE_PUBLIC_INTERFACE 1
+#endif
+
 namespace __scudo {
 
 #if SANITIZER_CAN_USE_ALLOCATOR64
 # if defined(__aarch64__) && SANITIZER_ANDROID
 const uptr AllocatorSize = 0x4000000000ULL;  // 256G.
 # elif defined(__aarch64__)
 const uptr AllocatorSize = 0x10000000000ULL;  // 1T.
 # else

test/scudo/interface.cpp

 // RUN: %clangxx_scudo %s -lstdc++ -o %t
-// RUN: %run %t ownership          2>&1
-// RUN: %run %t ownership-and-size 2>&1
-// RUN: %run %t heap-size          2>&1
+// RUN:                                                   %run %t ownership          2>&1
+// RUN:                                                   %run %t ownership-and-size 2>&1
+// RUN:                                                   %run %t heap-size          2>&1
+// RUN: %env_scudo_opts="allocator_may_return_null=1"     %run %t soft-limit         2>&1
+// RUN: %env_scudo_opts="allocator_may_return_null=1" not %run %t hard-limit         2>&1
 
 // Tests that the sanitizer interface functions behave appropriately.
 
 #include <stdlib.h>
 #include <assert.h>
 #include <string.h>
+#include <unistd.h>
 
 #include <vector>
 
 #include <sanitizer/allocator_interface.h>
+#include <sanitizer/scudo_interface.h>
 
 int main(int argc, char **argv)
 {
   assert(argc == 2);
 
   if (!strcmp(argv[1], "ownership")) {
     // Ensures that __sanitizer_get_ownership can be called before any other
     // allocator function, and that it behaves properly on a pointer not owned
for (size_t size : sizes) {
       free(p);
     }
   }
   if (!strcmp(argv[1], "heap-size")) {
     // Ensures that __sanitizer_get_heap_size can be called before any other
     // allocator function.
     assert(__sanitizer_get_heap_size() >= 0);
   }
+  if (!strcmp(argv[1], "soft-limit")) {
+    // Verifies that setting the soft RSS limit at runtime works as expected.
+    std::vector<void *> pointers;
+    size_t size = 1 << 19;  // 512Kb
+    for (int i = 0; i < 5; i++)
+      pointers.push_back(malloc(size));
+    // Set the soft RSS limit to 1Mb.
+    __scudo_set_rss_limit(1, 0);
+    usleep(20000);
+    // The following allocation should return NULL.
+    void *p = malloc(size);
+    assert(!p);
+    // Remove the soft RSS limit.
+    __scudo_set_rss_limit(0, 0);
+    // The following allocation should succeed.
+    p = malloc(size);
+    assert(p);
+    free(p);
+    while (!pointers.empty()) {
+      free(pointers.back());
+      pointers.pop_back();
+    }
+  }
+  if (!strcmp(argv[1], "hard-limit")) {
+    // Verifies that setting the hard RSS limit at runtime works as expected.
+    std::vector<void *> pointers;
+    size_t size = 1 << 19;  // 512Kb
+    for (int i = 0; i < 5; i++)
+      pointers.push_back(malloc(size));
+    // Set the hard RSS limit to 1Mb
+    __scudo_set_rss_limit(1, 1);
+    usleep(20000);
+    // The following should trigger our death.
+    void *p = malloc(size);
+  }
 
   return 0;
 }