[Red Flag?] Generating invariant_start/end intrinsics.

[lvoufo]

Summary

[Continued from D11826; sans 'writeonce' annotations.]

Ready to submit for review upstream, based on design doc review.

Please let me know if you see any red flag so far?

[lvoufo]

Add test cases...

[lvoufo]

This is pretty much it.
Next Steps:

• update design doc, and
• submit this for revision upstream.

[lvoufo]

Add test cases for mutable fields, const references and pointers and (basic) inheritance.

[lvoufo]

Update diff formatting.

[lvoufo]

Allow objects with mutable fields to be candidates for writeonce semantics; and specify writeonce candidates as trivial objects with no non-trivial method that may write to memory.

[hans]

Some drive-by comments.

include/clang/AST/Type.h

712

This will need a comment.

lib/AST/DeclCXX.cpp

1348 ↗

(On Diff #36429)

Maybe use an assert instead? IsWriteOnceCandidate is already initialized to false.

1367 ↗

(On Diff #36429)

Doesn't not defined just mean it can be defined out-of-line? Do you want M->isPure() instead?

1399 ↗

(On Diff #36429)

Left-overs from an old patch version?

lib/AST/Type.cpp

1942

No const-object optimization in C? :-(

1946

Would it be safer to use a white-list approach? I.e. return false by default, and explicitly check for a whitelist of things (isConstant, CXXRecord::isWriteOnceCandidate).

lib/CodeGen/CGDecl.cpp

947

ultra nit: } else {

[lvoufo]

Good catches. I'll make sure to incorporate these observations in the initial patch.

lib/AST/DeclCXX.cpp

1367 ↗

(On Diff #36429)

"not defined" here is implemented using "isDefined()", which also checks for out-of-line definitions. I think I probably mean M->isPure() afterall. I will have to double check that.

1399 ↗

(On Diff #36429)

Looks I I need some clean up.

lib/AST/Type.cpp

1942

Hmm... This is following what we are currently with global constants. (cf. QualType::isConstant())

[majnemer]

include/clang/AST/DeclCXX.h

777 ↗

(On Diff #36429)

This would need a comment.

lib/AST/DeclCXX.cpp

1337 ↗

(On Diff #36429)

isWriteOnceCandidate + the const qualifier on the method would indicate to me that this method is just a getter for IsWriteOnceCandidate but it actually computes a result and caches it. Perhaps it should be called something like computeWriteOnceCandidacy.

1350 ↗

(On Diff #36429)

Why can we skip ctors and dtors?

1362 ↗

(On Diff #36429)

Pointers lean right.

1371 ↗

(On Diff #36429)

You are checking if the method is virtual but I think you also need to check that the method is pure.

lib/CodeGen/CGDecl.cpp

861–864

Can you clang-format this?

862

References should lean right.

865

Is list-initialization needed here?

944

Pointers lean right. Also, consider using auto * for the declaration of CAddr.

949

Is this code supposed to be dead?

lib/CodeGen/CGDeclCXX.cpp

158–159

Please use braces around the else side.

lib/CodeGen/CodeGenFunction.h

3222

Perhaps this should have RAII in it's name to make it clear that it's destructor is stateful?

3229

References lean right.

lib/CodeGen/CodeGenModule.h

266

Please start variables with upper-case letters.

267

Please clang-format this.

[lvoufo]

Cool. Roger. Thanks!

lib/AST/DeclCXX.cpp

1350 ↗

(On Diff #36429)

The invariant intrinsic calls are generated right after the construction of a given const object (which must be initialized) and right before its destruction at the end of its lifetime.
Both constructors and destructors modify the allocated memory and will not be called in the middle of an invariant_start/end pair, and thus may not affect the reduction of loads. So, I don't think it helps to handle constructors here as well.

[lvoufo]

Thanks for all the comments. All should be addressed with D13603.

lib/AST/DeclCXX.cpp

1350 ↗

(On Diff #36429)

Added this information as comment.

lib/AST/Type.cpp

1942

I've added a TODO for you here. :)

1946

I am not exactly sure how this would be safer, but I can see how it would be more expressive. Thanks!

lib/CodeGen/CGDecl.cpp

865

I think so, for "uniform initialization" purposes.

include/clang/AST/Type.h

     return QualType::isConstant(*this, Ctx);
   }
 
+  bool isWriteOnce(ASTContext &Ctx);
+
   /// \brief Determine whether this is a Plain Old Data (POD) type (C++ 3.9p10).
   bool isPODType(ASTContext &Context) const;
 

lib/AST/Type.cpp

   }
 }
 
+bool QualType::isWriteOnce(ASTContext &Context) {
+
+  if (!Context.getLangOpts().CPlusPlus ||
+      !isConstant(Context) || getTypePtr()->isReferenceType())
+    return false;
+
+  if (const CXXRecordDecl *Record
+        = Context.getBaseElementType(*this)->getAsCXXRecordDecl())
+    return !Record->hasMutableFields();
+
+  return true;
+}
+
 bool QualType::isPODType(ASTContext &Context) const {
   // C++11 has a more relaxed definition of POD.
   if (Context.getLangOpts().CPlusPlus11)

lib/CodeGen/CGDecl.cpp

   if (D.getInit())
     var = AddInitializerToStaticVarDecl(D, var);
 
+  MarkWriteOnceWritten MWO(*this, D, var);
+
   var->setAlignment(alignment.getQuantity());
 
   if (D.hasAttr<AnnotateAttr>())

       CGF.EmitLifetimeEnd(Size, Addr);
     }
   };
+
+  /// A cleanup to call @llvm.invariant.end.
+  class CallInvariantEnd final : public EHScopeStack::Cleanup {
+    llvm::CallInst *StartInst;
+    llvm::Value *Addr;
+    llvm::Value *Size;    // TODO: Is this even necessary?
+  public:
+    CallInvariantEnd(llvm::CallInst *C, llvm::Value *addr, llvm::Value *size)
+      : StartInst(C), Addr(addr), Size(size) {}
+
+    void Emit(CodeGenFunction &CGF, Flags flags) override {
+      CGF.EmitInvariantEnd(StartInst, Size, Addr);
+    }
+  };
 }
 
 /// EmitAutoVarWithLifetime - Does the setup required for an automatic

          canEmitInitWithFewStoresAfterMemset(Init, StoreBudget);
 }
 
+/// Emit the code necessary to initialize the given global variable.
+/// TODO: Use Address for Addr (to match lifetime intrisics).
+MarkWriteOnceWritten::MarkWriteOnceWritten(CodeGenFunction &CGF,
+                                           const VarDecl& D, llvm::Value *Addr)
+: GV(dyn_cast_or_null<llvm::GlobalVariable>(Addr)),
+  AI(dyn_cast_or_null<llvm::AllocaInst>(Addr)),
+  Args{nullptr, 0, nullptr},
+  CGF(CGF) {
+
+  // Only GlobalVariable's and AllocaInst's can be writeonce.
+  // Exit if the given address is none of these.
+  if (!GV && !AI) return;
+
+  // If the address is writeonce, then emit @llvm.invariant.start() intrinsic.
+  // Then, for non-global variables, push the emission of the
+  // @llvm.invariant.end() intrinsic onto the cleanup stack.
+  if ((AI || !GV->isConstant()) && D.getType().isWriteOnce(CGF.getContext()))
+    Args = CGF.EmitInvariantStart(D, Addr);
+}
+
+MarkWriteOnceWritten::~MarkWriteOnceWritten() {
+  if (AI && Args.StartInst) {
+    assert(!GV && "Can't have it both ways!");
+    CGF.EHStack.pushCleanup<CallInvariantEnd>(NormalCleanup, Args.StartInst,
+                                              Args.Addr, Args.Size);
+  }
+}
+
 /// EmitAutoVarDecl - Emit code and set up an entry in LocalDeclMap for a
 /// variable declaration with auto, register, or no storage class specifier.
 /// These turn into simple stack objects, or GlobalValues depending on target.
 void CodeGenFunction::EmitAutoVarDecl(const VarDecl &D) {
   AutoVarEmission emission = EmitAutoVarAlloca(D);
   EmitAutoVarInit(emission);
+  MarkWriteOnceWritten MWO(*this, D, emission.Addr.isValid() ?
+                           emission.Addr.getPointer() : nullptr);
   EmitAutoVarCleanups(emission);
 }
 

   C->setDoesNotThrow();
 }
 
+/// Emit code to cause the variable at the given address to be considered as
+/// constant from this point onwards.
+CodeGenModule::InvariantArgs
+CodeGenFunction::EmitInvariantStart(const VarDecl &D, llvm::Value *Addr) {
+  // Don't emit the intrinsic if we're not optimizing.
+  if (!CGM.getCodeGenOpts().OptimizationLevel)
+    return CodeGenModule::InvariantArgs{nullptr, 0, nullptr};
+
+  // Grab the llvm.invariant.start intrinsic.
+  llvm::Intrinsic::ID InvStartID = llvm::Intrinsic::invariant_start;
+  llvm::Constant *InvariantStart = CGM.getIntrinsic(InvStartID);
+
+  // Emit a call with the size in bytes of the object.
+  CharUnits WidthChars = getContext().getTypeSizeInChars(D.getType());
+  uint64_t Width = WidthChars.getQuantity();
+
+  llvm::Value *Size;
+  if (llvm::Constant* CAddr = dyn_cast<llvm::Constant>(Addr)) {
+    Size = llvm::ConstantInt::getSigned(Int64Ty, Width);
+    Addr = llvm::ConstantExpr::getBitCast(CAddr, Int8PtrTy);
+  }
+  else {
+    //Width = CGM.getDataLayout().getTypeAllocSize(LTy);
+    Size = llvm::ConstantInt::get(Int64Ty, Width);
+    Addr = Builder.CreateBitCast(Addr, Int8PtrTy);
+  }
+  llvm::CallInst *C = Builder.CreateCall(InvariantStart, {Size, Addr});
+  C->setDoesNotThrow();
+
+  return CodeGenModule::InvariantArgs{C, Size, Addr};
+}
+
+void CodeGenFunction::EmitInvariantEnd(llvm::CallInst *Start,
+                                       llvm::Value *Size, llvm::Value *Addr) {
+  // Grab the llvm.invariant.end intrinsic.
+  llvm::Intrinsic::ID InvEndID = llvm::Intrinsic::invariant_end;
+  llvm::Constant *InvariantEnd = CGM.getIntrinsic(InvEndID);
+
+  // Emit a call with the size in bytes of the object.
+  llvm::CallInst *C = Builder.CreateCall(InvariantEnd, {Start, Size, Addr});
+  C->setDoesNotThrow();
+}
+
 /// EmitAutoVarAlloca - Emit the alloca and debug information for a
 /// local variable.  Does not emit initialization or destruction.
 CodeGenFunction::AutoVarEmission

lib/CodeGen/CGDeclCXX.cpp

   CGM.getCXXABI().registerGlobalDtor(CGF, D, function, argument);
 }
 
-/// Emit code to cause the variable at the given address to be considered as
-/// constant from this point onwards.
-static void EmitDeclInvariant(CodeGenFunction &CGF, const VarDecl &D,
-                              llvm::Constant *Addr) {
-  // Don't emit the intrinsic if we're not optimizing.
-  if (!CGF.CGM.getCodeGenOpts().OptimizationLevel)
-    return;
-
-  // Grab the llvm.invariant.start intrinsic.
-  llvm::Intrinsic::ID InvStartID = llvm::Intrinsic::invariant_start;
-  llvm::Constant *InvariantStart = CGF.CGM.getIntrinsic(InvStartID);
-
-  // Emit a call with the size in bytes of the object.
-  CharUnits WidthChars = CGF.getContext().getTypeSizeInChars(D.getType());
-  uint64_t Width = WidthChars.getQuantity();
-  llvm::Value *Args[2] = { llvm::ConstantInt::getSigned(CGF.Int64Ty, Width),
-                           llvm::ConstantExpr::getBitCast(Addr, CGF.Int8PtrTy)};
-  CGF.Builder.CreateCall(InvariantStart, Args);
-}
-
 void CodeGenFunction::EmitCXXGlobalVarDeclInit(const VarDecl &D,
                                                llvm::Constant *DeclPtr,
                                                bool PerformInit) {

           PerformInit, this);
     if (PerformInit)
       EmitDeclInit(*this, D, DeclAddr);
-    if (CGM.isTypeConstant(D.getType(), true))
-      EmitDeclInvariant(*this, D, DeclPtr);
-    else
+    if (CGM.isTypeConstant(D.getType(), true)) {
+      // Generate the @llvm.invariant.start intrinsic call if DeclPtr is
+      // not 'writeonce'. We'll do the same for 'writeonce' addresses later.
+      llvm::GlobalVariable* GV = dyn_cast<llvm::GlobalVariable>(DeclPtr);
+      if (GV && (GV->isConstant() || !D.getType().isWriteOnce(getContext())))
+        (void) EmitInvariantStart(D, DeclPtr);
+    } else
       EmitDeclDestroy(*this, D, DeclAddr);
     return;
   }

   } else {
     EmitCXXGlobalVarDeclInit(*D, Addr, PerformInit);
   }
+  MarkWriteOnceWritten MWO(*this, *D, Addr);
 
   FinishFunction();
 }

lib/CodeGen/CGExprAgg.cpp

 
   // Optimize the slot if possible.
   CheckAggExprForMemSetUse(Slot, E, *this);
- 
   AggExprEmitter(*this, Slot, Slot.isIgnored()).Visit(const_cast<Expr*>(E));
 }
 

lib/CodeGen/CodeGenFunction.h

 
   llvm::Value *EmitLifetimeStart(uint64_t Size, llvm::Value *Addr);
   void EmitLifetimeEnd(llvm::Value *Size, llvm::Value *Addr);
+  CodeGenModule::InvariantArgs EmitInvariantStart(const VarDecl &D,
+                                                  llvm::Value *Addr);
+  void EmitInvariantEnd(llvm::CallInst *C,
+                        llvm::Value *Size, llvm::Value *Addr);
 
   llvm::Value *EmitCXXNewExpr(const CXXNewExpr *E);
   void EmitCXXDeleteExpr(const CXXDeleteExpr *E);

   llvm::Value *GetValueForARMHint(unsigned BuiltinID);
 };
 
+class MarkWriteOnceWritten {
+  llvm::GlobalVariable *GV;
+  llvm::AllocaInst *AI;
+  CodeGenModule::InvariantArgs Args;
+  CodeGenFunction &CGF;
+public:
+  MarkWriteOnceWritten(CodeGenFunction &CGF,
+                       const VarDecl& D, llvm::Value *Addr);
+  ~MarkWriteOnceWritten();
+};
+
 /// Helper class with most of the code for saving a value for a
 /// conditional expression cleanup.
 struct DominatingLLVMValue {

lib/CodeGen/CodeGenModule.h

 
   typedef std::vector<Structor> CtorList;
 
+  struct InvariantArgs {
+    llvm::CallInst *StartInst;
+    llvm::Value *Size;
+    llvm::Value *Addr;
+    InvariantArgs(llvm::CallInst *C, llvm::Value *size, llvm::Value *addr)
+    : StartInst(C), Size(size), Addr(addr) {}
+  };
+
 private:
   ASTContext &Context;
   const LangOptions &LangOpts;

test/CodeGenCXX/const-invariant.cpp

+// *** const objects.
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O0 -o - -DCONST -DOBJ | FileCheck %s --check-prefix=CHECK-O0
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DCONST -DOBJ | FileCheck %s --check-prefix=CHECK-L-CO --check-prefix=CHECK-L-CO-OBJ --check-prefix=CHECK-NG-CO
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DPART_GLOBAL -DCONST -DOBJ | FileCheck %s --check-prefix=CHECK-GP-CO --check-prefix=CHECK-NL-CO --check-prefix=CHECK-NL-CO-OBJ --check-prefix=CHECK-NG-CO
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DGLOBAL -DCONST -DOBJ | FileCheck %s --check-prefix=CHECK-G-CO --check-prefix=CHECK-G-CO-OBJ --check-prefix=CHECK-NL-CO
+
+// *** const variables of builtin type.
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O0 -o - -DCONST -DINT | FileCheck %s --check-prefix=CHECK-O0
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DCONST -DINT | FileCheck %s --check-prefix=CHECK-L-CO --check-prefix=CHECK-L-CO-INT --check-prefix=CHECK-NG-CO
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DPART_GLOBAL -DCONST -DINT | FileCheck %s --check-prefix=CHECK-GP-CO --check-prefix=CHECK-NL-CO --check-prefix=CHECK-NL-CO-INT --check-prefix=CHECK-NG-CO
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DGLOBAL -DCONST -DINT | FileCheck %s --check-prefix=CHECK-G-CO --check-prefix=CHECK-G-CO-INT --check-prefix=CHECK-NL-CO
+
+// *** non-const objects.
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DOBJ | FileCheck %s --check-prefix=CHECK-NC
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DPART_GLOBAL -DOBJ | FileCheck %s --check-prefix=CHECK-NC
+// RUN: %clang -x c++ -emit-llvm -target x86_64-linux-gnu -S %s -O2 -mllvm -disable-llvm-optzns -o - -DGLOBAL -DOBJ | FileCheck %s --check-prefix=CHECK-NC
+
+// Check that llvm.invariant.start/end are properly generated on C++ const variables.
+
+// Do not produce markers at -O0.
+// CHECK-O0-NOT: llvm.invariant.start
+// CHECK-O0-NOT: llvm.invariant.end
+
+// Do not produce markers from non-const objects
+// CHECK-NC-NOT: llvm.invariant.start
+// CHECK-NC-NOT: llvm.invariant.end
+
+// const or not?
+#ifdef CONST
+ #define Const const
+#else
+ #define Const 
+#endif
+
+// object or builtin?
+#if defined(OBJ)
+  struct A {
+    int a;
+    int b;
+    A(int a) : a(a) { }
+  };
+
+  // A with explicit destructor
+  struct D {
+    int a;
+    int b;
+    D(int a) : a(a) { }
+    ~D();
+  };
+  
+  // A with mutable data field.
+  struct M {
+    int a;
+    mutable int b;
+    M(int a) : a(a) { }
+  };
+  
+  // A non-virtual base class.
+  struct Base {
+    int pa;
+    Base(int a);
+  };
+  struct I : public Base {
+    int a;
+    int b;
+    I(int a);
+  };
+  
+  // A virtual base class.
+  struct VBase {
+    int pa;
+    VBase(int a);
+    virtual void f(); 
+  };
+  struct IV : public VBase {
+    int a;
+    int b;
+    IV(int a);
+  };
+
+  #define Type A    // C++ class
+#elif defined(INT)
+  #define Type int  // Builtin
+#endif
+
+// global, partial global or local?
+#if defined(GLOBAL)
+  #undef LOCAL
+  #undef LOCAL_PARTIAL
+#elif defined(PART_GLOBAL)
+  #undef LOCAL
+  #define LOCAL_PARTIAL
+#else 
+  #define LOCAL
+  #define LOCAL_PARTIAL
+#endif
+
+// --------------- START ---------------
+
+const int one();
+
+#ifndef LOCAL
+Const Type i(one());
+#ifdef CONST
+Const Type& i_r = Type(one());
+#endif
+Const Type* i_p = new Type(one());
+Const Type* Const i_pc = new Type(one());
+#endif
+
+#ifndef LOCAL_PARTIAL
+Const Type j = i;
+Type k = i;
+#endif
+
+#if defined(OBJ) && !defined(LOCAL)
+Const D i_d(one());
+Const M i_m(one());
+Const I i_i(one());
+Const IV i_iv(one());
+#endif
+
+
+// CHECK-NL-CO: @_ZL1i = internal global
+// CHECK-NL-CO: @i_r = global
+// CHECK-NL-CO: @_ZGR3i_r_ = internal global
+// CHECK-NL-CO: @i_p = global
+// CHECK-NL-CO: @_ZL4i_pc = internal global
+// CHECK-G-CO: @k = global
+// CHECK-NL-CO-OBJ: @_ZL3i_d = internal global
+// CHECK-NL-CO-OBJ: @_ZL3i_m = internal global
+// CHECK-NL-CO-OBJ: @_ZL3i_i = internal global
+// CHECK-NL-CO-OBJ: @_ZL4i_iv = internal global
+// CHECK-G-CO: @_ZL1j = internal global
+
+// CHECK-NL-CO-OBJ: call void @_ZN1AC{{[0-9]+}}Ei({{.*}}* @_ZL1i, {{.*}})
+// CHECK-NL-CO-INT: store {{.*}}, {{.*}}* @_ZL1i
+// CHECK-NL-CO: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8* bitcast ({{.*}} @_ZL1i to i8*))
+// CHECK-NL-CO-OBJ: call void @_ZN1DC{{[0-9]+}}Ei({{.*}}* @_ZL3i_d, {{.*}})
+// CHECK-NL-CO-OBJ: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8* bitcast ({{.*}} @_ZL3i_d to i8*))
+// CHECK-NL-CO-OBJ: call void @_ZN1MC{{[0-9]+}}Ei({{.*}}* @_ZL3i_m, {{.*}})
+// CHECK-NL-CO-OBJ-NOT: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8* bitcast ({{.*}} @_ZL3i_m to i8*))
+// CHECK-NL-CO-OBJ: call void @_ZN1IC{{[0-9]+}}Ei({{.*}}* @_ZL3i_i, {{.*}})
+// CHECK-NL-CO-OBJ: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8* bitcast ({{.*}} @_ZL3i_i to i8*))
+// CHECK-NL-CO-OBJ: call void @_ZN2IVC{{[0-9]+}}Ei({{.*}}* @_ZL4i_iv, {{.*}})
+// CHECK-NL-CO-OBJ: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8* bitcast ({{.*}} @_ZL4i_iv to i8*))
+
+void foo(const Type* const);
+void bar(Type);
+#if defined(OBJ)
+void foo_d(const D* const);
+void bar_d(D);
+void foo_m(const M* const);
+void bar_m(M);
+void foo_i(const I* const);
+void bar_i(I);
+void foo_iv(const IV* const);
+void bar_iv(IV);
+#endif
+
+// Example 1: Duplicate loads.
+void ex1() {
+// CHECK: @_Z3ex1v(
+#ifdef LOCAL
+  Const Type i(one());
+#endif
+  // CHECK-L-CO: %i = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN1AC{{[0-9]+}}Ei({{.*}}* %i,
+  // CHECK-L-CO-INT: store {{.*}}, {{.*}}* %i
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8*
+  bar(i);
+  foo(&i);           // Does not change i.
+  bar(i);
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.end({{.*}}, i64 {{[0-9]+}}, i8*
+}
+
+// Example 2: Unnecessary stores and loads.
+void ex2() {
+// CHECK: @_Z3ex2v(
+#ifdef LOCAL
+  Const Type i(one());
+#endif
+#ifdef LOCAL_PARTIAL
+  Const Type j = i;
+#endif
+  // CHECK-L-CO: %i = alloca {{.*}}
+  // CHECK-NG-CO: %j = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN1AC{{[0-9]+}}Ei({{.*}}* %i,
+  // CHECK-L-CO-INT: store {{.*}}, {{.*}}* %i
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8*
+  // CHECK-NG-CO: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8*
+  bar(i);
+  foo(&i);           // Does not change i, nor j.
+  bar(j);
+  // CHECK-NG-CO: call {{.*}}@llvm.invariant.end({{.*}}, i64 {{[0-9]+}}, i8*
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.end({{.*}}, i64 {{[0-9]+}}, i8*
+}
+
+// Example 3: Necessary stores and loads.
+void ex3() {
+// CHECK: @_Z3ex3v(
+#ifdef LOCAL
+  Const Type i(one());
+#endif
+#ifdef LOCAL_PARTIAL
+  Type k = i;
+#endif
+  // CHECK-L-CO: %i = alloca {{.*}}
+  // CHECK-NG-CO: %k = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN1AC{{[0-9]+}}Ei({{.*}}* %i,
+  // CHECK-L-CO-INT: store {{.*}}, {{.*}}* %i
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8*
+  bar(i);
+  foo(&k);           // Does not change i; May change k.
+  bar(k);
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.end({{.*}}, i64 {{[0-9]+}}, i8*
+}
+
+#if defined(OBJ)
+
+// Example 1 with explicit destructor:
+//   Call to invariant_end intrinsic must be generated before
+//   call to destructor.
+void ex1_d() {
+// CHECK: @_Z5ex1_dv(
+#ifdef LOCAL
+  Const D i_d(one());
+#endif
+  // CHECK-L-CO-OBJ: %i_d = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN1DC{{[0-9]+}}Ei({{.*}}* %i_d,
+  // CHECK-L-CO-OBJ: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8*
+  bar_d(i_d);
+  foo_d(&i_d);           // Does not change i_d.
+  bar_d(i_d);
+  // CHECK-L-CO-OBJ: call {{.*}}@llvm.invariant.end({{.*}}, i64 {{[0-9]+}}, i8*
+  // CHECK-L-CO-OBJ: call void @_ZN1DD1Ev(
+}
+
+// Example 1 with mutable data field:
+//   Expect no invariant intrinsic call generated.
+void ex1_m() {
+// CHECK: @_Z5ex1_mv(
+#ifdef LOCAL
+  Const M i_m(one());
+#endif
+  // CHECK-L-CO-OBJ: %i_m = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN1MC{{[0-9]+}}Ei({{.*}}* %i_m,
+  // CHECK-L-CO-OBJ-NOT: call {{.*}}@llvm.invariant.start(
+  bar_m(i_m);
+  foo_m(&i_m);           // May change i_m.
+  bar_m(i_m);
+  // CHECK-L-CO-OBJ-NOT: call {{.*}}@llvm.invariant.end(
+}
+
+// Example 1 with type inheriting non-virtual base:
+//   Expect no change from Example 1.
+void ex1_i() {
+// CHECK: @_Z5ex1_iv(
+#ifdef LOCAL
+  Const I i_i(one());
+#endif
+  // CHECK-L-CO-OBJ: %i_i = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN1IC{{[0-9]+}}Ei({{.*}}* %i_i,
+  // CHECK-L-CO-OBJ: call {{.*}}@llvm.invariant.start(
+  bar_i(i_i);
+  foo_i(&i_i);           // Does not change i_i.
+  bar_i(i_i);
+  // CHECK-L-CO-OBJ: call {{.*}}@llvm.invariant.end(
+}
+
+// Example 1 with type inheriting non-virtual base:
+//   Expect no change from Example 1.
+void ex1_iv() {
+// CHECK: @_Z6ex1_ivv(
+#ifdef LOCAL
+  Const IV i_iv(one());
+#endif
+  // CHECK-L-CO-OBJ: %i_iv = alloca {{.*}}
+
+  // CHECK-L-CO-OBJ: call void @_ZN2IVC{{[0-9]+}}Ei({{.*}}* %i_iv,
+  // CHECK-L-CO-OBJ: call {{.*}}@llvm.invariant.start(
+  bar_iv(i_iv);
+  foo_iv(&i_iv);          // Does not change i_iv.
+  bar_iv(i_iv);
+  // CHECK-L-CO-OBJ: call {{.*}}@llvm.invariant.end(
+}
+
+#endif // #if defined(OBJ)
+
+// Example 1 with references and pointers:
+//   Expect no invariant intrinsic call generated on non const pointers.
+void ex1_p() {
+// CHECK: @_Z5ex1_pv(
+#ifdef LOCAL
+#ifdef CONST
+  Const Type& i_r = Type(one());
+#endif
+  Const Type* i_p = new Type(one());
+  Const Type* Const i_pc = new Type(one());
+#endif
+
+#ifdef CONST
+  // CHECK-L-CO-OBJ: call void @_ZN1AC{{[0-9]+}}Ei({{.*}}* %{{.*.tmp}},
+  // CHECK-L-CO-INT: store {{.*}}, {{.*}}* %{{.*.tmp}}
+  // CHECK-L-CO-NOT: call {{.*}}@llvm.invariant.start(
+  bar(i_r);
+  foo(&i_r);           // May change i_r.
+  bar(i_r);
+  // CHECK-L-CO-NOT: call {{.*}}@llvm.invariant.end(
+#endif
+
+  // CHECK-L-CO: call {{.*}} i8* @_Znwm(
+  // CHECK-L-CO-NOT: call {{.*}}@llvm.invariant.start(
+  bar(*i_p);
+  foo(i_p);           // May change *i_p.
+  bar(*i_p);
+  // CHECK-L-CO-NOT: call {{.*}}@llvm.invariant.end(
+
+  // CHECK-L-CO: call {{.*}} i8* @_Znwm(
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8*
+  bar(*i_pc);
+  foo(i_pc);           // May change *i_p.
+  bar(*i_pc);
+  // CHECK-L-CO: call {{.*}}@llvm.invariant.end({{.*}}, i64 {{[0-9]+}}, i8*
+}
+
+
+// CHECK-G-CO-OBJ: call {{.*}}@llvm.invariant.start(i64 {{[0-9]+}}, i8* bitcast ({{.*}} @_ZL1j to i8*))
+// CHECK-G-CO-INT: store {{.*}}, {{.*}}* @_ZL1j
+
+

test/CodeGenCXX/init-invariant.cpp

 // CHECK-NOT: call {{.*}}@llvm.invariant.start(i64 4, i8* bitcast ({{.*}} @b to i8*))
 
 // CHECK: call void @_ZN1CC1Ev({{.*}}* nonnull @c)
-// CHECK-NOT: call {{.*}}@llvm.invariant.start(i64 4, i8* bitcast ({{.*}} @c to i8*))
+// CHECK: call {{.*}}@llvm.invariant.start(i64 4, i8* bitcast ({{.*}} @c to i8*))
 
 // CHECK: call i32 @_Z1fv(
 // CHECK: store {{.*}}, i32* @d