[libcxx] Improve shared_ptr dtor performance

[bcraig]

Summary

Improve shared_ptr dtor performance

If the last destruction is uncontended, skip the atomic store on
__shared_weak_owners_. This shifts some costs from normal
shared_ptr usage to weak_ptr uses.

For x86_64, this results in an 8% improvement in shared_ptr ctor+dtor
performance.
Old benchmarks/shared_ptr_create_destroy.cpp: 26.8638 seconds
New benchmarks/shared_ptr_create_destroy.cpp: 24.6019 seconds

Weak_ptr increment / decrement is now slower.
Old benchmarks/weak_ptr_inc_dec_ref.cpp: 11.2892 seconds
New benchmarks/weak_ptr_inc_dec_ref.cpp: 14.5522 seconds

The increment / decrement code path did not degrade on X86_64.
Old benchmarks/shared_ptr_inc_dec_ref.cpp: 13.0896 seconds
New benchmarks/shared_ptr_inc_dec_ref.cpp: 13.0463 seconds

[bcraig]

Added weak_ptr benchmark, as that's where the cost shifted.

[EricWF]

LGTM. Thank you for the thorough doc. It sure made the review easy on my end.

I took the liberty of rewriting your 3 benchmarks to use Google Benchmark: https://gist.github.com/EricWF/6ab9d3ca9315f2dcf8b0b8a7e47a9ac8
I'm hoping to have a singular benchmark format so it's easier to write tooling for. Here are the docs for building libc++'s benchmarks.

[bcraig]

I am going to submit the code changes and the tests independently. I'm having trouble getting cmake to use the right compiler for the libcxx-benchmarks target.

[bcraig]

committed rL277357: Improve shared_ptr dtor performance.

benchmarks/shared_ptr_create_destroy.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include <memory>
+#include <iostream>
+#include <chrono>
+#include <atomic>
+
+void clobber()
+{
+    asm volatile("" : : : "memory");
+}
+
+std::atomic<int> g_int;
+std::atomic<int> g_other;
+
+int main() {
+  auto a = std::chrono::high_resolution_clock::now();
+  {
+    clobber();
+    for (int i = 0; i < 1000000000; ++i)
+    {
+      auto sp = std::make_shared<int>(g_int.load(std::memory_order_relaxed));
+      g_other.store(*sp, std::memory_order_relaxed);
+    }
+    clobber();
+  }
+  auto b = std::chrono::high_resolution_clock::now();
+  std::cout<<std::chrono::duration_cast<std::chrono::nanoseconds>(b - a).count()/1000000000.0<<" seconds"<<std::endl;
+  return 0;
+}

benchmarks/shared_ptr_inc_dec_ref.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include <memory>
+#include <iostream>
+#include <chrono>
+#include <atomic>
+
+void clobber()
+{
+    asm volatile("" : : : "memory");
+}
+
+std::atomic<int> g_int;
+std::atomic<int> g_other;
+
+int main() {
+  auto a = std::chrono::high_resolution_clock::now();
+  auto sp = std::make_shared<int>(g_int.load(std::memory_order_relaxed));
+  {
+    clobber();
+    for (int i = 0; i < 1000000000; ++i)
+    {
+      std::shared_ptr<int> sp2(sp);
+      g_other.store(*sp2, std::memory_order_relaxed);
+    }
+    clobber();
+  }
+  auto b = std::chrono::high_resolution_clock::now();
+  std::cout<<std::chrono::duration_cast<std::chrono::nanoseconds>(b - a).count()/1000000000.0<<" seconds"<<std::endl;
+  return 0;
+}

benchmarks/weak_ptr_inc_dec_ref.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include <memory>
+#include <iostream>
+#include <chrono>
+#include <atomic>
+
+void clobber()
+{
+    asm volatile("" : : : "memory");
+}
+
+std::atomic<int> g_int;
+std::atomic<int> g_other;
+
+int main() {
+  auto a = std::chrono::high_resolution_clock::now();
+  auto sp = std::make_shared<int>(g_int.load(std::memory_order_relaxed));
+  {
+    clobber();
+    for (int i = 0; i < 1000000000; ++i)
+    {
+      std::weak_ptr<int> wp(sp);
+    }
+    clobber();
+  }
+  auto b = std::chrono::high_resolution_clock::now();
+  std::cout<<std::chrono::duration_cast<std::chrono::nanoseconds>(b - a).count()/1000000000.0<<" seconds"<<std::endl;
+  return 0;
+}

src/memory.cpp

 {
     if (__shared_count::__release_shared())
         __release_weak();
 }
 
 void
 __shared_weak_count::__release_weak() _NOEXCEPT
 {
-    if (decrement(__shared_weak_owners_) == -1)
+    // NOTE: The acquire load here is an optimization of the very
+    // common case where a shared pointer is being destructed while
+    // having no other contended references.
+    //
+    // BENEFIT: We avoid expensive atomic stores like XADD and STREX
+    // in a common case.  Those instructions are slow and do nasty
+    // things to caches.
+    //
+    // IS THIS SAFE?  Yes.  During weak destruction, if we see that we
+    // are the last reference, we know that no-one else is accessing
+    // us. If someone were accessing us, then they would be doing so
+    // while the last shared / weak_ptr was being destructed, and
+    // that's undefined anyway.
+    //
+    // If we see anything other than a 0, then we have possible
+    // contention, and need to use an atomicrmw primitive.
+    // The same arguments don't apply for increment, where it is legal
+    // (though inadvisable) to share shared_ptr references between
+    // threads, and have them all get copied at once.  The argument
+    // also doesn't apply for __release_shared, because an outstanding
+    // weak_ptr::lock() could read / modify the shared count.
+    if (__libcpp_atomic_load(&__shared_weak_owners_, _AO_Aquire) == 0)
+    {
+        // no need to do this store, because we are about
+        // to destroy everything.
+        //__libcpp_atomic_store(&__shared_weak_owners_, -1, _AO_Release);
+        __on_zero_shared_weak();
+    }
+    else if (decrement(__shared_weak_owners_) == -1)
         __on_zero_shared_weak();
 }
 
 __shared_weak_count*
 __shared_weak_count::lock() _NOEXCEPT
 {
     long object_owners = __libcpp_atomic_load(&__shared_owners_);
     while (object_owners != -1)