Introducing llvm.invariant.group.barrier intrinsic

[Prazek]

Summary

For more info for what reason it was invented, goto:
http://lists.llvm.org/pipermail/cfe-dev/2015-July/044227.html

[nlewycky]

include/llvm/IR/IRBuilder.h

1638–1639

Suggested wording: Create a call to invariant.group.barrier which stops the optimizer from propagating equality with invariant.group metadata.

lib/CodeGen/CodeGenPrepare.cpp

1414–1417

Do we also discard the !invariant.group metadata on instructions?

The backend has pointers to the Value*'s, it would be bad if they used the !invariant.group markers but all the invariant.group.barrier calls had been removed.

lib/Transforms/IPO/GlobalOpt.cpp

2507–2508 ↗

(On Diff #33043)

Why is this safe? We have

%p2 = call @invariant.group.barrier(%p1)

and we set that %p2 just plain *is* %p1. Then nothing happens until evaluation completes and we commit the changes, which means replacing %p2 with %p1. At the time, why won't that operation miscompile?

I can think of two possible reasons, one is that %p1 may be constrained (like, it may be a ConstantExpr) and the other is that we may know that there are no remaining loads of either %p1 or %p2 with !invariant.group on them.

In any event, please answer in the form of a comment in this code. :)

[Prazek]

lib/CodeGen/CodeGenPrepare.cpp

1414–1417

The question is, does it make any difference? What I understand, is that CodeGenPrepare is done just before machine code generation, which means that additional metadata in load/stores doesn't make any difference.

lib/Transforms/IPO/GlobalOpt.cpp

2507–2508 ↗

(On Diff #33043)

What do you mean about miscompile?
I think this is safe because global-opt doesn't care about !invariant.group metadata, which means that getting rid of invariant.group.barrier will not break anything.

[Prazek]

deleted globalOpt handling because it was invalid, and I want to push this one first.

[Prazek]

lib/CodeGen/CodeGenPrepare.cpp

1414–1417

http://reviews.llvm.org/D12875

[nlewycky]

Is there a LangRef change that introduces the barrier? If that's simply in a different review, then LGTM. Usually I'd ask for the feature and the LangRef change to land together, but it also makes sense to put the LangRef update for !invariant.group and @llvm.invariant.group.barrier together.

[rsmith]

The LangRef updates are in http://reviews.llvm.org/D11399; approving based on Nick's conditional LGTM.

include/llvm/IR/IRBuilder.h

 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/ConstantFolder.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Support/CBindingWrapping.h"
 
 namespace llvm {
 class MDNode;
 
Value *CreatePtrDiff(Value *LHS, Value *RHS, const Twine &Name = "") {
     Value *LHS_int = CreatePtrToInt(LHS, Type::getInt64Ty(Context));
     Value *RHS_int = CreatePtrToInt(RHS, Type::getInt64Ty(Context));
     Value *Difference = CreateSub(LHS_int, RHS_int);
     return CreateExactSDiv(Difference,
                            ConstantExpr::getSizeOf(ArgType->getElementType()),
                            Name);
   }
 
+  /// \brief Create an invariant.group.barrier intrinsic call, that stops
+  /// optimizer to propagate equality using invariant.group metadata.
+  /// If Ptr type is different from i8*, it's casted to i8* before call
+  /// and casted back to Ptr type after call.
+  Value *CreateInvariantGroupBarrier(Value *Ptr) {
+    Module *M = BB->getParent()->getParent();
+    Function *FnInvariantGroupBarrier = Intrinsic::getDeclaration(M,
+            Intrinsic::invariant_group_barrier);
+
+    Type *ArgumentAndReturnType = FnInvariantGroupBarrier->getReturnType();
+    assert(ArgumentAndReturnType ==
+        FnInvariantGroupBarrier->getFunctionType()->getParamType(0) &&
+        "InvariantGroupBarrier should take and return the same type");
+    Type *PtrType = Ptr->getType();
+
+    bool PtrTypeConversionNeeded = PtrType != ArgumentAndReturnType;
+    if (PtrTypeConversionNeeded)
+      Ptr = CreateBitCast(Ptr, ArgumentAndReturnType);
+
+    CallInst *Fn = CreateCall(FnInvariantGroupBarrier, {Ptr});
+
+    if (PtrTypeConversionNeeded)
+      return CreateBitCast(Fn, PtrType);
+    return Fn;
+  }
+
   /// \brief Return a vector value that contains \arg V broadcasted to \p
   /// NumElts elements.
   Value *CreateVectorSplat(unsigned NumElts, Value *V, const Twine &Name = "") {
     assert(NumElts > 0 && "Cannot splat to an empty vector!");
 
     // First insert it into an undef vector so we can shuffle it.
     Type *I32Ty = getInt32Ty();
     Value *Undef = UndefValue::get(VectorType::get(V->getType(), NumElts));

include/llvm/IR/Intrinsics.td

 def int_stackprotectorcheck : Intrinsic<[], [llvm_ptrptr_ty],
                                         [IntrReadWriteArgMem]>;
 
 // A counter increment for instrumentation based profiling.
 def int_instrprof_increment : Intrinsic<[],
                                         [llvm_ptr_ty, llvm_i64_ty,
                                          llvm_i32_ty, llvm_i32_ty],
                                         []>;
                                         
 //===------------------- Standard C Library Intrinsics --------------------===//
 //
 
 def int_memcpy  : Intrinsic<[],
                              [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
                               llvm_i32_ty, llvm_i1_ty],
                             [IntrReadWriteArgMem, NoCapture<0>, NoCapture<1>,
                              ReadOnly<1>]>;

 def int_invariant_start : Intrinsic<[llvm_descriptor_ty],
                                     [llvm_i64_ty, llvm_ptr_ty],
                                     [IntrReadWriteArgMem, NoCapture<1>]>;
 def int_invariant_end   : Intrinsic<[],
                                     [llvm_descriptor_ty, llvm_i64_ty,
                                      llvm_ptr_ty],
                                     [IntrReadWriteArgMem, NoCapture<2>]>;
 
+def int_invariant_group_barrier : Intrinsic<[llvm_ptr_ty], 
+                                            [llvm_ptr_ty], 
+                                            [IntrNoMem]>;
+
 //===------------------------ Stackmap Intrinsics -------------------------===//
 //
 def int_experimental_stackmap : Intrinsic<[],
                                   [llvm_i64_ty, llvm_i32_ty, llvm_vararg_ty],
                                   [Throws]>;
 def int_experimental_patchpoint_void : Intrinsic<[],
                                                  [llvm_i64_ty, llvm_i32_ty,
                                                   llvm_ptr_ty, llvm_i32_ty,

lib/CodeGen/CodeGenPrepare.cpp

case Intrinsic::aarch64_stxr: {
         return false;
       // Sink a zext feeding stlxr/stxr before it, so it can be folded into it.
       ExtVal->moveBefore(CI);
       // Mark this instruction as "inserted by CGP", so that other
       // optimizations don't touch it.
       InsertedInsts.insert(ExtVal);
       return true;
     }
+    case Intrinsic::invariant_group_barrier:
+      II->replaceAllUsesWith(II->getArgOperand(0));
+      II->eraseFromParent();
+      return true;
     }
 
     if (TLI) {
       // Unknown address space.
       // TODO: Target hook to pick which address space the intrinsic cares
       // about?
       unsigned AddrSpace = ~0u;
       SmallVector<Value*, 2> PtrOps;

test/Transforms/GlobalOpt/invariant.group.barrier.ll

+; RUN: opt -S -globalopt < %s | FileCheck %s
+
+; This test is hint, what could globalOpt optimize and what it can't
+; FIXME: @tmp and @tmp2 can be safely set to 42
+; CHECK: @tmp = global i32 0
+; CHECK: @tmp2 = global i32 0
+; CHECK: @tmp3 = global i32 0
+
+@tmp = global i32 0
+@tmp2 = global i32 0
+@tmp3 = global i32 0
+@ptrToTmp3 = global i32* null
+
[e-mail]_ctors = appending global [1 x { i32, void ()* }] [{ i32, void ()* } { i32 65535, void ()* @_GLOBAL__I_a }]
+
+define i32 @TheAnswerToLifeTheUniverseAndEverything() {
+  ret i32 42
+}
+
+define void @_GLOBAL__I_a() {
+enter:
+  call void @_optimizable()
+  call void @_not_optimizable()
+  ret void
+}
+
+define void @_optimizable() {
+enter:
+  %valptr = alloca i32
+  
+  %val = call i32 @TheAnswerToLifeTheUniverseAndEverything()
+  store i32 %val, i32* @tmp
+  store i32 %val, i32* %valptr
+  
+  %0 = bitcast i32* %valptr to i8*
+  %barr = call i8* @llvm.invariant.group.barrier(i8* %0)
+  %1 = bitcast i8* %barr to i32*
+  
+  %val2 = load i32, i32* %1
+  store i32 %val2, i32* @tmp2
+  ret void
+}
+
+; We can't step through invariant.group.barrier here, because that would change
+; this load in @usage_of_globals()
+; val = load i32, i32* %ptrVal, !invariant.group !0 
+; into 
+; %val = load i32, i32* @tmp3, !invariant.group !0
+; and then we could assume that %val and %val2 to be the same, which coud be 
+; false, because @changeTmp3ValAndCallBarrierInside() may change the value
+; of @tmp3.
+define void @_not_optimizable() {
+enter:
+  store i32 13, i32* @tmp3, !invariant.group !0
+  
+  %0 = bitcast i32* @tmp3 to i8*
+  %barr = call i8* @llvm.invariant.group.barrier(i8* %0)
+  %1 = bitcast i8* %barr to i32*
+  
+  store i32* %1, i32** @ptrToTmp3
+  store i32 42, i32* %1, !invariant.group !0
+  
+  ret void
+}
+define void @usage_of_globals() {
+entry:
+  %ptrVal = load i32*, i32** @ptrToTmp3
+  %val = load i32, i32* %ptrVal, !invariant.group !0
+  
+  call void @changeTmp3ValAndCallBarrierInside()
+  %val2 = load i32, i32* @tmp3, !invariant.group !0
+  ret void;
+}
+
+declare void @changeTmp3ValAndCallBarrierInside()
+
+declare i8* @llvm.invariant.group.barrier(i8*)
+
+!0 = !{!"something"}