[Clang][AVX512][Intrinsics]Convert AVX non-temporal store builtins to LLVM-native IR.

[craig.topper]

LGTM

[RKSimon]

Is there any reason why we can't just get rid of all the SSE movnt builtins and use __builtin_nontemporal_store instead (D12313)?

[ab]

In D20358#446210, @RKSimon wrote:

Is there any reason why we can't just get rid of all the SSE movnt builtins and use __builtin_nontemporal_store instead (D12313)?

I wanted to suggest that too, but I think you'd have problems with the (natural?) alignment requirement of __builtin_nontemporal_store (whereas IIRC, movnti & friends accept unaligned pointers).

[ab]

In D20358#446218, @ab wrote:

In D20358#446210, @RKSimon wrote:

Is there any reason why we can't just get rid of all the SSE movnt builtins and use __builtin_nontemporal_store instead (D12313)?

I wanted to suggest that too, but I think you'd have problems with the (natural?) alignment requirement of __builtin_nontemporal_store (whereas IIRC, movnti & friends accept unaligned pointers).

But now that I look at this again, I suppose we could have some attribute((aligned(1))), or something like r271214.

[RKSimon]

In D20358#446220, @ab wrote:

In D20358#446218, @ab wrote:

In D20358#446210, @RKSimon wrote:

Is there any reason why we can't just get rid of all the SSE movnt builtins and use __builtin_nontemporal_store instead (D12313)?

I wanted to suggest that too, but I think you'd have problems with the (natural?) alignment requirement of __builtin_nontemporal_store (whereas IIRC, movnti & friends accept unaligned pointers).

But now that I look at this again, I suppose we could have some attribute((aligned(1))), or something like r271214.

True, luckily that only affects _mm_stream_si32 and _mm_stream_si64 - the 'real' vector movnt stores all require type alignment. The _mm_stream_load_* (movntdqa) loads cases should be trivial as well.

[RKSimon]

In D20358#446241, @RKSimon wrote:

In D20358#446220, @ab wrote:

In D20358#446218, @ab wrote:

In D20358#446210, @RKSimon wrote:

Is there any reason why we can't just get rid of all the SSE movnt builtins and use __builtin_nontemporal_store instead (D12313)?

I wanted to suggest that too, but I think you'd have problems with the (natural?) alignment requirement of __builtin_nontemporal_store (whereas IIRC, movnti & friends accept unaligned pointers).

But now that I look at this again, I suppose we could have some attribute((aligned(1))), or something like r271214.

True, luckily that only affects _mm_stream_si32 and _mm_stream_si64 - the 'real' vector movnt stores all require type alignment. The _mm_stream_load_* (movntdqa) loads cases should be trivial as well.

I've created D21272 that covers the conversion of SSE/SSE2/AVX/AVX512 non-temporal aligned vector stores to use __builtin_nontemporal_store in headers

[RKSimon]

D21272 has now been committed, which I think removes the need for this patch. D20359 is still needed (with the additional tests requested by Craig).

[RKSimon]

Abandon this patch? We replaced the x86 vector non-temporal store builtins with __builtin_nontemporal_store directly in the headers.

lib/CodeGen/CGBuiltin.cpp

case X86::BI__builtin_ia32_psrldqi256: {
 
     Value *SV = llvm::ConstantDataVector::get(getLLVMContext(), Indices);
     SV = Builder.CreateShuffleVector(Ops[0], Zero, SV, "psrldq");
     llvm::Type *ResultType = ConvertType(E->getType());
     return Builder.CreateBitCast(SV, ResultType, "cast");
   }
   case X86::BI__builtin_ia32_movntps:
   case X86::BI__builtin_ia32_movntps256:
+  case X86::BI__builtin_ia32_movntps512:
   case X86::BI__builtin_ia32_movntpd:
   case X86::BI__builtin_ia32_movntpd256:
+  case X86::BI__builtin_ia32_movntpd512:
   case X86::BI__builtin_ia32_movntdq:
   case X86::BI__builtin_ia32_movntdq256:
+  case X86::BI__builtin_ia32_movntdq512:
   case X86::BI__builtin_ia32_movnti:
   case X86::BI__builtin_ia32_movnti64: {
     llvm::MDNode *Node = llvm::MDNode::get(
         getLLVMContext(), llvm::ConstantAsMetadata::get(Builder.getInt32(1)));
 
     // Convert the type of the pointer to a pointer to the stored type.
     Value *BC = Builder.CreateBitCast(Ops[0],
                                 llvm::PointerType::getUnqual(Ops[1]->getType()),

test/CodeGen/avx512f-builtins.c

 __mmask16 test_mm512_kxor(__mmask16 __A, __mmask16 __B) {
   // CHECK-LABEL: @test_mm512_kxor
   // CHECK: @llvm.x86.avx512.kxor.w
   return _mm512_kxor(__A, __B); 
 }
 
 void test_mm512_stream_si512(__m512i * __P, __m512i __A) {
   // CHECK-LABEL: @test_mm512_stream_si512
-  // CHECK: @llvm.x86.avx512.storent.q.512
+  // CHECK-NOT: call
+  // CHECK: store <8 x i64> %3, <8 x i64>* %2, align 64, !nontemporal !1
   _mm512_stream_si512(__P, __A); 
 }
 
 __m512i test_mm512_stream_load_si512(void *__P) {
   // CHECK-LABEL: @test_mm512_stream_load_si512
   // CHECK: @llvm.x86.avx512.movntdqa
   return _mm512_stream_load_si512(__P); 
 }
 
 void test_mm512_stream_pd(double *__P, __m512d __A) {
   // CHECK-LABEL: @test_mm512_stream_pd
-  // CHECK: @llvm.x86.avx512.storent.pd.512
+  // CHECK-NOT: call
+  // CHECK: store <8 x double> %3, <8 x double>* %cast.i, align 64, !nontemporal !1
   return _mm512_stream_pd(__P, __A); 
 }
 
 void test_mm512_stream_ps(float *__P, __m512 __A) {
   // CHECK-LABEL: @test_mm512_stream_ps
-  // CHECK: @llvm.x86.avx512.storent.ps.512
+  // CHECK-NOT: call
+  // CHECK: store <16 x float> %3, <16 x float>* %cast.i, align 64, !nontemporal !1
   _mm512_stream_ps(__P, __A); 
 }
 
 __m512d test_mm512_mask_compress_pd(__m512d __W, __mmask8 __U, __m512d __A) {
   // CHECK-LABEL: @test_mm512_mask_compress_pd
   // CHECK: @llvm.x86.avx512.mask.compress.pd.512
   return _mm512_mask_compress_pd(__W, __U, __A); 
 }