Implement P1007R3 `std::assume_aligned`

[ldionne]

Summary

https://wg21.link/P1007R3

This is a first cut.

• The constexpr stuff is not tested; I don't think it's going to work. Still investigating.
• Only GCC and clang are supported at the moment. I need to add other compilers.

[lebedev.ri]

Looks good in general, like i suspected "deceivingly simple" :)
Related: D54589

include/memory

5591

Drop noexcept?
— R2, 2018-10-08: Revised wording and removed noexcept following LWG guidance.

[mclow.lists]

include/memory

5591

Nope. The standard has to leave off noexcept. Implementers can add it.

Hence the comment I left about LIBCPP_ASSERT_NOEXCEPT(std::assume_aligned<1, T>(p))
which is a libc++-specific check.

test/std/utilities/memory/ptr.align/assume_aligned.pass.cpp

29

Needs a LIBCPP_ASSERT_NOEXCEPT(std::assume_aligned<1, T>(p))

[ldionne]

Subscribed Chandler and emailed Timur in case they want to take a look, since it's their paper.

include/memory

5590

_LIBCPP_INLINE_VISIBILITY?

test/std/utilities/memory/ptr.align/assume_aligned.pass.cpp

29

This check needs to go into test/libcxx/<...>.

[mclow.lists]

Chandler has said to me (via IRC) that when called at constexpr time, this should just be a return __p. I'll add that after we have std:: is_constant_evaluated

include/memory

5590

Right.

test/std/utilities/memory/ptr.align/assume_aligned.pass.cpp

29

I don't think we need an entire test just for one line.
We have that macro specifically for libc++-specific assertions.

[ldionne]

test/std/utilities/memory/ptr.align/assume_aligned.pass.cpp

29

SGTM

[mclow.lists]

Updated the tests; added constexpr tests (which are currently disabled).
Added _LIBCPP_INLINE_VISIBILITY per Louis' (correct) request.
Added LIBCPP_ASSERT_NOEXCEPT

[mclow.lists]

test/std/utilities/memory/ptr.align/assume_aligned.pass.cpp

29

A tab snuck in here. I'll detab before I commit.

[ldionne]

Can you update the cxx2a_status page to say that we're missing constexpr support for the feature?

[lebedev.ri]

include/memory

5596–5598

@mclow.lists were you planning on finishing adding support for other spellings?

[ldionne]

include/memory

5594

Also, this should probably go.

[EricWF]

We don't need to use __builtin_assume_aligned to implement this. We can use __attribute__((assume_aligned(N))) instead. That solves our constexpr problem.

[rsmith]

In D54966#1314038, @EricWF wrote:

We don't need to use __builtin_assume_aligned to implement this. We can use __attribute__((assume_aligned(N))) instead. That solves our constexpr problem.

More than that, we can't use __builtin_assume_aligned, as currently implemented in Clang and GCC, to implement this -- it is non-constant if it cannot prove that the pointer is properly-aligned. The library specification for std::assume_aligned doesn't permit that behavior, and instead appears to require that a call is a constant subexpression whenever p is properly aligned (even if that can't be determined until runtime), meaning that it must also be a constant subexpression in cases where you cannot prove alignment one way or the other. Example:

extern char p; // might be defined with `alignas(16)`
extern char *q;
char *r = q; // required to be initialized to `&p` if `p` is properly aligned, may be `&p` or `nullptr` otherwise
char *q = std::assume_aligned<16>(&p); // required to be statically initialized if `p` is properly aligned

int main() { assert((uintptr_t)&p & 15 || r == &q); }

If you use __builtin_assume_aligned to implement std::assume_aligned, the above assertion can fail, because &p can't be proven to be properly-aligned. But if you use the attribute, then no checking will be done during constant evaluation, and the assertion will always pass. (Try it out yourself: https://godbolt.org/z/YKIdWF -- try changing the alignment from 1 to 2 and back, and observe that q switches between static and dynamic initialization, and r switches from being initialized to &p to being initialized to nullptr, even though the compiler does not know that p will not be 2-byte aligned.)

It's OK that assume_aligned is a constant subexpression even when such a call would be UB; the fact that we might have undefined behavior within a constant expression evaluation of the library function std::assume_aligned is explicitly permitted by [expr.const]p4 (after the long list of bullets):

If e satisfies the constraints of a core constant expression, but evaluation of e would evaluate an operation that has undefined behavior as specified in Clause 15 through Clause 31 of this document[...], it is unspecified whether e is a core constant expression.

@chandlerc, @hfinkel: does an attribute-only implementation (with no constant evaluation enforcement) materially hurt the ability for the optimizer to use this annotation? Eg, in:

extern char k[16];
void f() {
  // the initializer of p is a constant expression and might get constant-folded to &k by the frontend
  char *p = std::assume_aligned<16>(&k);
  // ...do stuff...
}

the alignment assumption may well never be emitted as IR. Is that acceptable?

[hfinkel]

In D54966#1317428, @rsmith wrote:

In D54966#1314038, @EricWF wrote:

We don't need to use __builtin_assume_aligned to implement this. We can use __attribute__((assume_aligned(N))) instead. That solves our constexpr problem.

...

@chandlerc, @hfinkel: does an attribute-only implementation (with no constant evaluation enforcement) materially hurt the ability for the optimizer to use this annotation? Eg, in:

extern char k[16];
void f() {
  // the initializer of p is a constant expression and might get constant-folded to &k by the frontend
  char *p = std::assume_aligned<16>(&k);
  // ...do stuff...
}

the alignment assumption may well never be emitted as IR. Is that acceptable?

__attribute__((assume_aligned(N))) and __builtin_assume_aligned are, at the IR level, implemented in a very-similar way. For functions with that attribute on the return type, we essentially emit an alignment assumption on the return value at every call site (in CodeGenFunction::EmitCall). Thus, from the optimizer's perspective, I don't think that it makes a big difference.

[rsmith]

In D54966#1317750, @hfinkel wrote:

In D54966#1317428, @rsmith wrote:

@chandlerc, @hfinkel: does an attribute-only implementation (with no constant evaluation enforcement) materially hurt the ability for the optimizer to use this annotation? Eg, in:

extern char k[16];
void f() {
  // the initializer of p is a constant expression and might get constant-folded to &k by the frontend
  char *p = std::assume_aligned<16>(&k);
  // ...do stuff...
}

the alignment assumption may well never be emitted as IR. Is that acceptable?

__attribute__((assume_aligned(N))) and __builtin_assume_aligned are, at the IR level, implemented in a very-similar way. For functions with that attribute on the return type, we essentially emit an alignment assumption on the return value at every call site (in CodeGenFunction::EmitCall). Thus, from the optimizer's perspective, I don't think that it makes a big difference.

The point here is that you may well get no IR annotation whatsoever for the above call, because the frontend might constant-evaluate the initializer of p down to just &k, and then emit IR that just initializes p to &k with no alignment assumption. Whereas if we treated assume_aligned<N>(p) as non-constant in the cases where we cannot prove that p is suitably aligned (as __builtin_assume_aligned does), then we would emit IR for the alignment assumption, but the downside is that the initializer of p would no longer be a constant expression.

Essentially, what I'm trying to gauge here is, is it OK that you probably don't actually get an alignment assumption in a function like the f() above, because it will probably be constant-evaluated away to nothing? Or do we need the constant evaluator to have some kind of side-channel by which it can communicate back to the code generator that an alignment assumption should be applied to k? Or, indeed, should assume_aligned<N>(p) not be treated as a constant expression unless we can prove during constant expression evaluation that p is in fact suitably aligned -- as GCC and Clang currently do for __builtin_assume_aligned?

[hfinkel]

In D54966#1319368, @rsmith wrote:

In D54966#1317750, @hfinkel wrote:

In D54966#1317428, @rsmith wrote:

@chandlerc, @hfinkel: does an attribute-only implementation (with no constant evaluation enforcement) materially hurt the ability for the optimizer to use this annotation? Eg, in:

extern char k[16];
void f() {
  // the initializer of p is a constant expression and might get constant-folded to &k by the frontend
  char *p = std::assume_aligned<16>(&k);
  // ...do stuff...
}

the alignment assumption may well never be emitted as IR. Is that acceptable?

__attribute__((assume_aligned(N))) and __builtin_assume_aligned are, at the IR level, implemented in a very-similar way. For functions with that attribute on the return type, we essentially emit an alignment assumption on the return value at every call site (in CodeGenFunction::EmitCall). Thus, from the optimizer's perspective, I don't think that it makes a big difference.

The point here is that you may well get no IR annotation whatsoever for the above call, because the frontend might constant-evaluate the initializer of p down to just &k, and then emit IR that just initializes p to &k with no alignment assumption. Whereas if we treated assume_aligned<N>(p) as non-constant in the cases where we cannot prove that p is suitably aligned (as __builtin_assume_aligned does), then we would emit IR for the alignment assumption, but the downside is that the initializer of p would no longer be a constant expression.

Essentially, what I'm trying to gauge here is, is it OK that you probably don't actually get an alignment assumption in a function like the f() above, because it will probably be constant-evaluated away to nothing? Or do we need the constant evaluator to have some kind of side-channel by which it can communicate back to the code generator that an alignment assumption should be applied to k?

I thought about this side channel option for k, but I don't think that we can because we'd need to prove that f() function was always executed, and that's likely not generally possible. I think that the side channel would apply only to p.

Or, indeed, should assume_aligned<N>(p) not be treated as a constant expression unless we can prove during constant expression evaluation that p is in fact suitably aligned -- as GCC and Clang currently do for __builtin_assume_aligned?

From a C++ perspective, this seems suboptimal. I don't want people to duplicate code, some with assume_aligned, some without, if I want the same code with work both in a constexpr and not. A side channel would be better. It is a trade off, however, and I'd need to think more about it.

[krytarowski]

Could you please test thread_local too?

[chandlerc]

In D54966#1319390, @hfinkel wrote:

In D54966#1319368, @rsmith wrote:

In D54966#1317750, @hfinkel wrote:

In D54966#1317428, @rsmith wrote:

@chandlerc, @hfinkel: does an attribute-only implementation (with no constant evaluation enforcement) materially hurt the ability for the optimizer to use this annotation? Eg, in:

extern char k[16];
void f() {
  // the initializer of p is a constant expression and might get constant-folded to &k by the frontend
  char *p = std::assume_aligned<16>(&k);
  // ...do stuff...
}

the alignment assumption may well never be emitted as IR. Is that acceptable?

__attribute__((assume_aligned(N))) and __builtin_assume_aligned are, at the IR level, implemented in a very-similar way. For functions with that attribute on the return type, we essentially emit an alignment assumption on the return value at every call site (in CodeGenFunction::EmitCall). Thus, from the optimizer's perspective, I don't think that it makes a big difference.

The point here is that you may well get no IR annotation whatsoever for the above call, because the frontend might constant-evaluate the initializer of p down to just &k, and then emit IR that just initializes p to &k with no alignment assumption. Whereas if we treated assume_aligned<N>(p) as non-constant in the cases where we cannot prove that p is suitably aligned (as __builtin_assume_aligned does), then we would emit IR for the alignment assumption, but the downside is that the initializer of p would no longer be a constant expression.

Essentially, what I'm trying to gauge here is, is it OK that you probably don't actually get an alignment assumption in a function like the f() above, because it will probably be constant-evaluated away to nothing? Or do we need the constant evaluator to have some kind of side-channel by which it can communicate back to the code generator that an alignment assumption should be applied to k?

I thought about this side channel option for k, but I don't think that we can because we'd need to prove that f() function was always executed, and that's likely not generally possible. I think that the side channel would apply only to p.

After thinking more about it, I think users of p in the function f really should be able to assume the alignment, and whether they succeed at that should not be determined by whether the initialize of p happens to fail to be a constant expression for some reason.

Let me lay out my reasoning, it comes from considering a few examples.

1a) The fact that the address happens to fold to a constant and thus the initializer is a constant as well is not going to be enough to reliably optimize the uses. Just because we know the address of some very hot vector data is a global and thus a constant "relocation" that we can fold does *not* mean that we will be able to reconstruct the alignment guarantees if we need to do so. This means that we would be missing real optimization opportunities here, and indeed, the exact opportunities that std::assume_aligned was intended to open up.

1b) Indeed, I could imagine a collection of routines which all use the same constant initialized address but which make different alignment assumptions. And I could imagine code dispatching (potentially dynamically) to the correctly aligned routine. That seems like reasonable code to expect to be able to write given this facility, and yet it would be directly undermined by what you describe.

2. Imagine well tuned code using std::assume_aligned that gets refactored slightly such that the pointer happens to become suitable for constant initialization when previously it wasn't. This would in turn cause significant regressions in generated code quality which I think would be an unacceptable surprise to users. We would never be able to explain reasonably why a particular refactoring or change would *regress* performance of some code.

When considering these kinds of situations, it really seems like this needs to be propagated. And not just locally, but as far as the constant evaluation proceeds, walking past as many constant evaluated wrappers as needed. =/

I don't really think of this as needing a side-channel so much as address in the constant evaluation needing to track alignment in some way such that these get propagated as one would expect.

Or, indeed, should assume_aligned<N>(p) not be treated as a constant expression unless we can prove during constant expression evaluation that p is in fact suitably aligned -- as GCC and Clang currently do for __builtin_assume_aligned?

From a C++ perspective, this seems suboptimal. I don't want people to duplicate code, some with assume_aligned, some without, if I want the same code with work both in a constexpr and not. A side channel would be better. It is a trade off, however, and I'd need to think more about it.

Yeah, I really don't think we want to kill constant evaluation just to preserve alignment assumptions. Instead, we want to model those assumptions in the evaluator IMO (even if we don't allow them to fold away in core constant expressions or parts of the ABI).

[zoecarver]

test/std/utilities/memory/ptr.align/assume_aligned.fail.cpp

24

This should be std::assume_aligned<4>(p);.

[ldionne]

Commandeering to abandon since this is superseded by D118938.

[ldionne]

(Thanks for the patch, I did keep some content in D118938 and I credit you for it)

include/__config

 #define _LIBCPP_HAS_NO_CXX20_CHRONO_LITERALS
 #endif
 
+#if __has_builtin(__builtin_assume_aligned)
+#define _LIBCPP_HAS_BUILTIN_ASSUME_ALIGNED
+#endif
+
 #elif defined(_LIBCPP_COMPILER_GCC)
 
 #define _ALIGNAS(x) __attribute__((__aligned__(x)))

 #define _LIBCPP_HAS_UNIQUE_OBJECT_REPRESENTATIONS
 #endif
 
+#define _LIBCPP_HAS_BUILTIN_ASSUME_ALIGNED
+
 #define _LIBCPP_ALWAYS_INLINE __attribute__ ((__always_inline__))
 
 #elif defined(_LIBCPP_COMPILER_MSVC)

include/memory

 
 void* align(size_t alignment, size_t size, void*& ptr, size_t& space);
 
+// [ptr.aligned], pointer alignment hint
+template<size_t N, class T>
+  [[nodiscard]] constexpr T* assume_aligned(T* ptr);  // C++20
+
 }  // std
 
 */

 
 _LIBCPP_FUNC_VIS void* align(size_t __align, size_t __sz, void*& __ptr, size_t& __space);
 
+#if _LIBCPP_STD_VER > 17
+template<size_t _Sz, class _Tp>
+_LIBCPP_NODISCARD_AFTER_CXX17 _LIBCPP_INLINE_VISIBILITY
+constexpr _Tp* assume_aligned(_Tp * __p) noexcept
+{
+#ifdef _LIBCPP_HAS_BUILTIN_ASSUME_ALIGNED
+//    return __builtin_assume_aligned(__p, _Sz);
+    return reinterpret_cast<_Tp*>(__builtin_assume_aligned(__p, _Sz));
+#else
+    return __p;
+#endif
+}
+#endif  // _LIBCPP_STD_VER > 17
+
 // --- Helper for container swap --
 template <typename _Alloc>
 inline _LIBCPP_INLINE_VISIBILITY

test/std/utilities/memory/ptr.align/assume_aligned.fail.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+// UNSUPPORTED: c++98, c++03, c++11, c++14, c++17
+
+// #include <memory>
+
+// template<size_t N, class T>
+//     [[nodiscard]] constexpr T* assume_aligned(T* ptr);
+
+#include <memory>
+#include <cassert>
+
+#include "test_macros.h"
+
+int main ()
+{
+    int *p = nullptr;
+    std::assume_aligned(p, 4);  // expected-error {{ignoring return value of function declared with 'nodiscard' attribute}}
+}

test/std/utilities/memory/ptr.align/assume_aligned.pass.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+// UNSUPPORTED: c++98, c++03, c++11, c++14, c++17
+
+// #include <memory>
+
+// template<size_t N, class T>
+//     [[nodiscard]] constexpr T* assume_aligned(T* ptr);
+
+#include <memory>
+#include <cassert>
+
+#include "test_macros.h"
+
+template <typename T>
+constexpr bool is_aligned(T* p, size_t n)
+{ return (reinterpret_cast<std::uintptr_t>(p) & ((1 << n) - 1)) == 0; }
+
+template <typename T>
+void test(T *p)
+{
+  ASSERT_SAME_TYPE(T*, decltype(std::assume_aligned<1, T>(p)));
+    LIBCPP_ASSERT_NOEXCEPT(       std::assume_aligned<1, T>(p));
+  assert((p == std::assume_aligned<1>(p)));
+
+  if (is_aligned(p,   2))
+    assert((p == std::assume_aligned<  2>(p)));
+  if (is_aligned(p,   4))
+    assert((p == std::assume_aligned<  4>(p)));
+  if (is_aligned(p,   8))
+    assert((p == std::assume_aligned<  8>(p)));
+  if (is_aligned(p,  16))
+    assert((p == std::assume_aligned< 16>(p)));
+  if (is_aligned(p,  32))
+    assert((p == std::assume_aligned< 32>(p)));
+  if (is_aligned(p,  64))
+    assert((p == std::assume_aligned< 64>(p)));
+  if (is_aligned(p, 128))
+    assert((p == std::assume_aligned<128>(p)));
+}
+
+template <typename T>
+constexpr bool constexpr_test(T *p)
+{
+  ASSERT_SAME_TYPE(T*, decltype(std::assume_aligned<1, T>(p)));
+  size_t al = alignof(T);
+  if (p != std::assume_aligned<1>(p)) return false;
+  if (al >= 2)
+    if (p != std::assume_aligned<2>(p)) return false;
+  if (al >= 2)
+    if (p != std::assume_aligned<2>(p)) return false;
+  if (al >= 4)
+    if (p != std::assume_aligned<4>(p)) return false;
+  if (al >= 8)
+    if (p != std::assume_aligned<8>(p)) return false;
+  if (al >= 16)
+    if (p != std::assume_aligned<16>(p)) return false;
+  if (al >= 32)
+    if (p != std::assume_aligned<32>(p)) return false;
+  if (al >= 64)
+    if (p != std::assume_aligned<64>(p)) return false;
+  if (al >= 128)
+    if (p != std::assume_aligned<128>(p)) return false;
+  return true;
+}
+
+int i;
+long l;
+double d;
+long double ld;
+
+struct              S    {};
+struct alignas(  4) S4   {};
+struct alignas(  8) S8   {};
+struct alignas( 16) S16  {};
+struct alignas( 32) S32  {};
+struct alignas( 64) S64  {};
+struct alignas(128) S128 {};
+
+S       s;
+S4     s4;
+S8     s8;
+S16   s16;
+S32   s32;
+S64   s64;
+S128 s128;
+
+
+int main ()
+{
+
+  test( &i);
+  test( &l);
+  test( &d);
+  test(&ld);
+  
+  test(&s);
+  test(&s4);
+  test(&s8);
+  test(&s16);
+  test(&s32);
+  test(&s64);
+  test(&s128);
+
+//  static_assert(constexpr_test( &i), "");
+//  static_assert(constexpr_test( &l), ""); 
+//  static_assert(constexpr_test( &d), ""); 
+//  static_assert(constexpr_test(&ld), ""); 
+// 
+//  static_assert(constexpr_test(   &s), "");
+//  static_assert(constexpr_test(  &s4), "");
+//  static_assert(constexpr_test(  &s8), "");
+//  static_assert(constexpr_test( &s16), "");
+//  static_assert(constexpr_test( &s32), "");
+//  static_assert(constexpr_test( &s64), "");
+//  static_assert(constexpr_test(&s128), "");
+}