Introducing llvm.invariant.group.barrier intrinsic

[Prazek]

Summary

For more info for what reason it was invented, goto:
http://lists.llvm.org/pipermail/cfe-dev/2015-July/044227.html

[nlewycky]

include/llvm/IR/IRBuilder.h

1626–1627

Suggested wording: Create a call to invariant.group.barrier which stops the optimizer from propagating equality with invariant.group metadata.

lib/CodeGen/CodeGenPrepare.cpp

1414–1417

Do we also discard the !invariant.group metadata on instructions?

The backend has pointers to the Value*'s, it would be bad if they used the !invariant.group markers but all the invariant.group.barrier calls had been removed.

lib/Transforms/IPO/GlobalOpt.cpp

2507–2508

Why is this safe? We have

%p2 = call @invariant.group.barrier(%p1)

and we set that %p2 just plain *is* %p1. Then nothing happens until evaluation completes and we commit the changes, which means replacing %p2 with %p1. At the time, why won't that operation miscompile?

I can think of two possible reasons, one is that %p1 may be constrained (like, it may be a ConstantExpr) and the other is that we may know that there are no remaining loads of either %p1 or %p2 with !invariant.group on them.

In any event, please answer in the form of a comment in this code. :)

[Prazek]

lib/CodeGen/CodeGenPrepare.cpp

1414–1417

The question is, does it make any difference? What I understand, is that CodeGenPrepare is done just before machine code generation, which means that additional metadata in load/stores doesn't make any difference.

lib/Transforms/IPO/GlobalOpt.cpp

2507–2508

What do you mean about miscompile?
I think this is safe because global-opt doesn't care about !invariant.group metadata, which means that getting rid of invariant.group.barrier will not break anything.

[Prazek]

deleted globalOpt handling because it was invalid, and I want to push this one first.

[Prazek]

lib/CodeGen/CodeGenPrepare.cpp

1414–1417

http://reviews.llvm.org/D12875

[nlewycky]

Is there a LangRef change that introduces the barrier? If that's simply in a different review, then LGTM. Usually I'd ask for the feature and the LangRef change to land together, but it also makes sense to put the LangRef update for !invariant.group and @llvm.invariant.group.barrier together.

[rsmith]

The LangRef updates are in http://reviews.llvm.org/D11399; approving based on Nick's conditional LGTM.

include/llvm/IR/IRBuilder.h

 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/ConstantFolder.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalVariable.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/Intrinsics.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Support/CBindingWrapping.h"
 
 namespace llvm {
 class MDNode;
 
Value *CreatePtrDiff(Value *LHS, Value *RHS, const Twine &Name = "") {
     Value *LHS_int = CreatePtrToInt(LHS, Type::getInt64Ty(Context));
     Value *RHS_int = CreatePtrToInt(RHS, Type::getInt64Ty(Context));
     Value *Difference = CreateSub(LHS_int, RHS_int);
     return CreateExactSDiv(Difference,
                            ConstantExpr::getSizeOf(ArgType->getElementType()),
                            Name);
   }
 
+  /// \brief Create an invariant.group.barrier intrinsic call, that stops
+  /// optimizer to propagate equality using invariant.group metadata.
+  /// If Ptr type is different from i8*, it's casted to i8* before call
+  /// and casted back to Ptr type after call.
+  Value *CreateInvariantGroupBarrier(Value *Ptr) {
+    Module *M = BB->getParent()->getParent();
+    Function *FnInvariantGroupBarrier = Intrinsic::getDeclaration(M,
+            Intrinsic::invariant_group_barrier);
+
+    Type *ArgumentAndReturnType = FnInvariantGroupBarrier->getReturnType();
+    assert(ArgumentAndReturnType ==
+        FnInvariantGroupBarrier->getFunctionType()->getParamType(0) &&
+        "InvariantGroupBarrier should take and return the same type");
+    Type *PtrType = Ptr->getType();
+
+    bool PtrTypeConversionNeeded = PtrType != ArgumentAndReturnType;
+    if (PtrTypeConversionNeeded)
+      Ptr = CreateBitCast(Ptr, ArgumentAndReturnType);
+
+    CallInst *Fn = CreateCall(FnInvariantGroupBarrier, {Ptr});
+
+    if (PtrTypeConversionNeeded)
+      return CreateBitCast(Fn, PtrType);
+    return Fn;
+  }
+
   /// \brief Return a vector value that contains \arg V broadcasted to \p
   /// NumElts elements.
   Value *CreateVectorSplat(unsigned NumElts, Value *V, const Twine &Name = "") {
     assert(NumElts > 0 && "Cannot splat to an empty vector!");
 
     // First insert it into an undef vector so we can shuffle it.
     Type *I32Ty = getInt32Ty();
     Value *Undef = UndefValue::get(VectorType::get(V->getType(), NumElts));

include/llvm/IR/Intrinsics.td

 def int_stackprotectorcheck : Intrinsic<[], [llvm_ptrptr_ty],
                                         [IntrReadWriteArgMem]>;
 
 // A counter increment for instrumentation based profiling.
 def int_instrprof_increment : Intrinsic<[],
                                         [llvm_ptr_ty, llvm_i64_ty,
                                          llvm_i32_ty, llvm_i32_ty],
                                         []>;
                                         
 //===------------------- Standard C Library Intrinsics --------------------===//
 //
 
 def int_memcpy  : Intrinsic<[],
                              [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
                               llvm_i32_ty, llvm_i1_ty],
                             [IntrReadWriteArgMem, NoCapture<0>, NoCapture<1>,
                              ReadOnly<1>]>;

 def int_invariant_start : Intrinsic<[llvm_descriptor_ty],
                                     [llvm_i64_ty, llvm_ptr_ty],
                                     [IntrReadWriteArgMem, NoCapture<1>]>;
 def int_invariant_end   : Intrinsic<[],
                                     [llvm_descriptor_ty, llvm_i64_ty,
                                      llvm_ptr_ty],
                                     [IntrReadWriteArgMem, NoCapture<2>]>;
 
+def int_invariant_group_barrier : Intrinsic<[llvm_ptr_ty], 
+                                            [llvm_ptr_ty], 
+                                            [IntrNoMem]>;
+
 //===------------------------ Stackmap Intrinsics -------------------------===//
 //
 def int_experimental_stackmap : Intrinsic<[],
                                   [llvm_i64_ty, llvm_i32_ty, llvm_vararg_ty],
                                   [Throws]>;
 def int_experimental_patchpoint_void : Intrinsic<[],
                                                  [llvm_i64_ty, llvm_i32_ty,
                                                   llvm_ptr_ty, llvm_i32_ty,

lib/CodeGen/CodeGenPrepare.cpp

case Intrinsic::aarch64_stxr: {
         return false;
       // Sink a zext feeding stlxr/stxr before it, so it can be folded into it.
       ExtVal->moveBefore(CI);
       // Mark this instruction as "inserted by CGP", so that other
       // optimizations don't touch it.
       InsertedInsts.insert(ExtVal);
       return true;
     }
+    case Intrinsic::invariant_group_barrier:
+      II->replaceAllUsesWith(II->getArgOperand(0));
+      II->eraseFromParent();
+      return true;
     }
 
     if (TLI) {
       // Unknown address space.
       // TODO: Target hook to pick which address space the intrinsic cares
       // about?
       unsigned AddrSpace = ~0u;
       SmallVector<Value*, 2> PtrOps;

lib/Transforms/IPO/GlobalOpt.cpp

if (StoreInst *SI = dyn_cast<StoreInst>(CurInst)) {
             } else {
               DEBUG(dbgs() << "Found a global var, but can not treat it as an "
                     "invariant.\n");
             }
           }
           // Continue even if we do nothing.
           ++CurInst;
           continue;
+        } else if (II->getIntrinsicID() == Intrinsic::invariant_group_barrier) {
+          setVal(II, getVal(II->getOperand(0)));
+          DEBUG(dbgs() << "Passing through invariant.group.barrier intrinsic.\n");
+          ++CurInst;
+          continue;
         } else if (II->getIntrinsicID() == Intrinsic::assume) {
           DEBUG(dbgs() << "Skipping assume intrinsic.\n");
           ++CurInst;
           continue;
         }
 
         DEBUG(dbgs() << "Unknown intrinsic. Can not evaluate.\n");
         return false;

test/Transforms/GlobalOpt/invariant.group.barrier.ll

+; RUN: opt -S -globalopt < %s | FileCheck %s
+
+; CHECK: @tmp = global i32 42
+
+@tmp = global i32 0
[e-mail]_ctors = appending global [1 x { i32, void ()* }] [{ i32, void ()* } { i32 65535, void ()* @_GLOBAL__I_a }]
+
+define i32 @TheAnswerToLifeTheUniverseAndEverything() {
+  ret i32 42
+}
+
+define void @_GLOBAL__I_a() {
+enter:
+  %valptr = alloca i32
+  
+  %val = call i32 @TheAnswerToLifeTheUniverseAndEverything()
+  store i32 %val, i32* %valptr
+  
+  %0 = bitcast i32* %valptr to i8*
+  %barr = call i8* @llvm.invariant.group.barrier(i8* %0)
+  %1 = bitcast i8* %barr to i32*
+  
+  %val2 = load i32, i32* %1
+  store i32 %val2, i32* @tmp
+  ret void
+}
+
+declare i8* @llvm.invariant.group.barrier(i8*)