Allow invokable sub-classes of IntrinsicInst

[reames]

Summary

It used to be that all of our intrinsics were call instructions, but over time, we've added more and more invokable intrinsics. According to the verifier, we're up to 8 right now. As IntrinsicInst is a sub-class of CallInst, this puts us in an awkward spot where the idiomatic means to check for intrinsic has a false negative if the intrinsic is invoked.

This change switches IntrinsicInst from being a sub-class of CallInst to being a subclass of CallBase. This allows invoked intrinsics to be instances of IntrinsicInst, at the cost of requiring a few more casts to CallInst in places where the intrinsic really is known to be a call, not an invoke.

This is admittedly a debatable stylistic issue. To me, allowing invokeable intrinsics seems cleaner, but I also spent a lot of my time dealing with those intrinsics. Do others agree?

At the moment, this patch only builds on x86. I'm sure there will be analogous changes in the other targets. I'll do them if folks are okay with the overall direction.

After this lands (if it does), planned cleanups:

• Make GCStatepointInst a IntrinsicInst subclass.
• Merge intrinsic handling in InstCombine and use idiomatic visitIntrinsicInst entry point for InstVisitor.
• Do the same in SelectionDAG.
• Do the same in FastISEL.

[reames]

Rebase over cleanups - realized several of the things I was going to do to reduce diffs were generally useful code improvements so just went ahead and did them.

[reames]

ping

[nikic]

LGTM. The number of necessary casts seems to be pretty small, so I think this is a reasonable tradeoff.

llvm/include/llvm/IR/InstVisitor.h

313

non-CallBase

[echristo]

I guess my question is "why? what does this make easier/harder/etc?"

I see you added a bunch of people and then committed, but I'd still like to hear the answers.

[reames]

In D99976#2703175, @echristo wrote:

I guess my question is "why? what does this make easier/harder/etc?"

A couple of pieces here.

1. We no longer have the chance of a nasty false negative bug. Before, we could fail to handle some intrinsics because they would not match IntrinsicInst.
2. As listed in the potential cleanups in the commit, we have multiple places in code which handled intrinsics generically, but then had to add extra casing for the invokable ones. We'll be able to clean that up.
3. For intrinsics which really are calls - which, admittedly, is many - this does complicate the pattern matching as you have to both match the intrinsicinst, and then cast to CallInst for interfaces which expect calls. (A quick survey shows many of those interfaces could reasonably take CallBase, but not all.) We could explore having two base classes - one for all intrinsics and one for call only intrinsics - not quite sure if the multiple inheritance works out there. It didn't seem worthwhile to justify exploration to me, if you disagree, let me know.

I see you added a bunch of people and then committed, but I'd still like to hear the answers.

Er, did you intentionally leave out the "waited for LGTM" part? Your wording seems to imply bad faith which I don't think is called for.

If you think this is rushed and needs further discussion, I'm happy to revert while that discussion happens.

[reames]

(For anyone following along, discussion has moved to email on review thread which phabricator isn't catching.)

[aeubanks]

This slightly regressed compile times while it was in
http://llvm-compile-time-tracker.com/compare.php?from=080d48f279e2b70e4e9fc65bc4947a53ffc982f3&to=d87b9b81ccb95217181ce75515c6c68bbb408ca4&stat=instructions

[nikic]

In D99976#2703736, @aeubanks wrote:

This slightly regressed compile times while it was in
http://llvm-compile-time-tracker.com/compare.php?from=080d48f279e2b70e4e9fc65bc4947a53ffc982f3&to=d87b9b81ccb95217181ce75515c6c68bbb408ca4&stat=instructions

Interesting, I didn't expect that. Presumably this is because most places working with IntrinsicInst now have to perform two branches rather than one, and apparently it gets used a lot. Most likely this overestimates actual impact (assuming predicted branches), but still, better to avoid it if we can.

Could it make sense to have both IntrinsicBase and IntrinsicInst, similar to how we have CallBase and CallInst?

[reames]

Tweak the initial approach to optimize the non-intrinsic dispatch and involve fewer checked casts. I'm hoping this reduces the compile time impact.

One behavior change in this patch: unknown intrinsics (e.g. "llvm.unrecognized_name") now get passed to visitIntrinsic whereas there didn't previously. I think this is probably the right behavior anyways, and it simplifies the code a bit.

If this doesn't work, I'll look at optimize the intrinsic dispatch path, but that's a bit more involved.

[echristo]

In D99976#2706447, @nikic wrote:

In D99976#2703736, @aeubanks wrote:

This slightly regressed compile times while it was in
http://llvm-compile-time-tracker.com/compare.php?from=080d48f279e2b70e4e9fc65bc4947a53ffc982f3&to=d87b9b81ccb95217181ce75515c6c68bbb408ca4&stat=instructions

Interesting, I didn't expect that. Presumably this is because most places working with IntrinsicInst now have to perform two branches rather than one, and apparently it gets used a lot. Most likely this overestimates actual impact (assuming predicted branches), but still, better to avoid it if we can.

FWIW we also saw this in a build with predicted branches. :)

Could it make sense to have both IntrinsicBase and IntrinsicInst, similar to how we have CallBase and CallInst?

Perhaps? I think some of the cases you've got like CallInst *findIntrinsicBlah(...) could return CallBase and that might make sense since an intrinsic "found" could be invokeable. Perhaps name it something neither Call or Invoke to encompass that it can be used for both? (Maybe Callable? I don't have any good ideas and the thesaurus is full of hilarious, but unlikely to be useful synonyms :)

Thoughts?

[reames]

Closing review I'm not currently working on, and am unlikely to get back to in near future. Will reopen if priorities change.

llvm/include/llvm/IR/InstVisitor.h

#include "llvm/IR/Instruction.def"
   // not need this generality.
   //
   // These functions can also implement fan-out, when a single opcode and
   // instruction have multiple more specific Instruction subclasses. The Call
   // instruction currently supports this. We implement that by redirecting that
   // instruction to a special delegation helper.
 #define HANDLE_INST(NUM, OPCODE, CLASS) \
     RetTy visit##OPCODE(CLASS &I) { \
-      if (NUM == Instruction::Call) \
-        return delegateCallInst(I); \
+      if (NUM == Instruction::Call || NUM == Instruction::Invoke) \
+        return delegateCallBase(I); \
       else \
         DELEGATE(CLASS); \
     }
 #include "llvm/IR/Instruction.def"
 
   // Specific Instruction type classes... note that all of the casts are
   // necessary because we use the instruction classes as opaque types...
   //
#include "llvm/IR/Instruction.def"
   RetTy visitMemSetInst(MemSetInst &I)            { DELEGATE(MemIntrinsic); }
   RetTy visitMemCpyInst(MemCpyInst &I)            { DELEGATE(MemTransferInst); }
   RetTy visitMemMoveInst(MemMoveInst &I)          { DELEGATE(MemTransferInst); }
   RetTy visitMemTransferInst(MemTransferInst &I)  { DELEGATE(MemIntrinsic); }
   RetTy visitMemIntrinsic(MemIntrinsic &I)        { DELEGATE(IntrinsicInst); }
   RetTy visitVAStartInst(VAStartInst &I)          { DELEGATE(IntrinsicInst); }
   RetTy visitVAEndInst(VAEndInst &I)              { DELEGATE(IntrinsicInst); }
   RetTy visitVACopyInst(VACopyInst &I)            { DELEGATE(IntrinsicInst); }
-  RetTy visitIntrinsicInst(IntrinsicInst &I)      { DELEGATE(CallInst); }
+  RetTy visitIntrinsicInst(IntrinsicInst &I)      {
+    if (isa<CallInst>(I))
+      return static_cast<SubClass*>(this)->visitCallInst(cast<CallInst>(I));
+    else
+      return static_cast<SubClass*>(this)->visitInvokeInst(cast<InvokeInst>(I));
+  }
   RetTy visitCallInst(CallInst &I)                { DELEGATE(CallBase); }
   RetTy visitInvokeInst(InvokeInst &I)            { DELEGATE(CallBase); }
   RetTy visitCallBrInst(CallBrInst &I)            { DELEGATE(CallBase); }
 
   // While terminators don't have a distinct type modeling them, we support
   // intercepting them with dedicated a visitor callback.
   RetTy visitReturnInst(ReturnInst &I) {
     return static_cast<SubClass *>(this)->visitTerminator(I);
#include "llvm/IR/Instruction.def"
   // this instruction just gets ignored.
   //
   // Note that you MUST override this function if your return type is not void.
   //
   void visitInstruction(Instruction &I) {}  // Ignore unhandled instructions
 
 private:
   // Special helper function to delegate to CallInst subclass visitors.
-  RetTy delegateCallInst(CallInst &I) {
+  RetTy delegateCallBase(CallBase &I) {
     if (const Function *F = I.getCalledFunction()) {
       switch (F->getIntrinsicID()) {
       default:                     DELEGATE(IntrinsicInst);
       case Intrinsic::dbg_declare: DELEGATE(DbgDeclareInst);
       case Intrinsic::dbg_value:   DELEGATE(DbgValueInst);
       case Intrinsic::dbg_label:   DELEGATE(DbgLabelInst);
       case Intrinsic::memcpy:      DELEGATE(MemCpyInst);
       case Intrinsic::memmove:     DELEGATE(MemMoveInst);
       case Intrinsic::memset:      DELEGATE(MemSetInst);
       case Intrinsic::vastart:     DELEGATE(VAStartInst);
       case Intrinsic::vaend:       DELEGATE(VAEndInst);
       case Intrinsic::vacopy:      DELEGATE(VACopyInst);
       case Intrinsic::not_intrinsic: break;
       }
     }
-    DELEGATE(CallInst);
+    if (isa<CallInst>(I))
+      return static_cast<SubClass*>(this)->visitCallInst(cast<CallInst>(I));
+    else
+      return static_cast<SubClass*>(this)->visitInvokeInst(cast<InvokeInst>(I));
   }
 
   // An overload that will never actually be called, it is used only from dead
   // code in the dispatching from opcodes to instruction subclasses.
-  RetTy delegateCallInst(Instruction &I) {
-    llvm_unreachable("delegateCallInst called for non-CallInst");
+  RetTy delegateCallBase(Instruction &I) {
+    llvm_unreachable("delegateCallBase called for non-CallBase");
   }
 };
 
 #undef DELEGATE
 
 } // End llvm namespace
 
 #endif

llvm/include/llvm/IR/IntrinsicInst.h

 //===-- llvm/IntrinsicInst.h - Intrinsic Instruction Wrappers ---*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines classes that make it really easy to deal with intrinsic
 // functions with the isa/dyncast family of functions.  In particular, this
 // allows you to do things like:
 //
 //     if (MemCpyInst *MCI = dyn_cast<MemCpyInst>(Inst))
 //        ... MCI->getDest() ... MCI->getSource() ...
 //
-// All intrinsic function calls are instances of the call instruction, so these
-// are all subclasses of the CallInst class.  Note that none of these classes
-// has state or virtual methods, which is an important part of this gross/neat
-// hack working.
+// All intrinsic function calls are instances of the call or invoke instruction,
+// so these are all subclasses of the CallBase class.  Note that none of these
+// classes has state or virtual methods, which is an important part of this
+// gross/neat hack working.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_IR_INTRINSICINST_H
 #define LLVM_IR_INTRINSICINST_H
 
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"

 #include <cassert>
 #include <cstdint>
 
 namespace llvm {
 
 /// A wrapper class for inspecting calls to intrinsic functions.
 /// This allows the standard isa/dyncast/cast functionality to work with calls
 /// to intrinsic functions.
-class IntrinsicInst : public CallInst {
+class IntrinsicInst : public CallBase {
 public:
   IntrinsicInst() = delete;
   IntrinsicInst(const IntrinsicInst &) = delete;
   IntrinsicInst &operator=(const IntrinsicInst &) = delete;
 
   /// Return the intrinsic ID of this intrinsic.
   Intrinsic::ID getIntrinsicID() const {
     return getCalledFunction()->getIntrinsicID();
bool isAssumeLikeIntrinsic() const {
     case Intrinsic::ptr_annotation:
     case Intrinsic::var_annotation:
       return true;
     }
     return false;
   }
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
-  static bool classof(const CallInst *I) {
+  static bool classof(const CallBase *I) {
     if (const Function *CF = I->getCalledFunction())
       return CF->isIntrinsic();
     return false;
   }
   static bool classof(const Value *V) {
-    return isa<CallInst>(V) && classof(cast<CallInst>(V));
+    return isa<CallBase>(V) && classof(cast<CallBase>(V));
   }
 };
 
 /// Check if \p ID corresponds to a debug info intrinsic.
 static inline bool isDbgInfoIntrinsic(Intrinsic::ID ID) {
   switch (ID) {
   case Intrinsic::dbg_declare:
   case Intrinsic::dbg_value:

llvm/lib/Analysis/TypeMetadataUtils.cpp

void llvm::findDevirtualizableCallsForTypeTest(
     DominatorTree &DT) {
   assert(CI->getCalledFunction()->getIntrinsicID() == Intrinsic::type_test);
 
   const Module *M = CI->getParent()->getParent()->getParent();
 
   // Find llvm.assume intrinsics for this llvm.type.test call.
   for (const Use &CIU : CI->uses())
     if (auto *Assume = dyn_cast<AssumeInst>(CIU.getUser()))
-      Assumes.push_back(Assume);
+      Assumes.push_back(cast<CallInst>(Assume));
 
   // If we found any, search for virtual calls based on %p and add them to
   // DevirtCalls.
   if (!Assumes.empty())
     findLoadCallsAtConstantOffset(
         M, DevirtCalls, CI->getArgOperand(0)->stripPointerCasts(), 0, CI, DT);
 }
 

llvm/lib/CodeGen/SelectionDAG/FastISel.cpp

bool FastISel::selectIntrinsicCall(const IntrinsicInst *II) {
   case Intrinsic::expect: {
     Register ResultReg = getRegForValue(II->getArgOperand(0));
     if (!ResultReg)
       return false;
     updateValueMap(II, ResultReg);
     return true;
   }
   case Intrinsic::experimental_stackmap:
-    return selectStackmap(II);
+    return selectStackmap(cast<CallInst>(II));
   case Intrinsic::experimental_patchpoint_void:
   case Intrinsic::experimental_patchpoint_i64:
-    return selectPatchpoint(II);
+    return selectPatchpoint(cast<CallInst>(II));
 
   case Intrinsic::xray_customevent:
-    return selectXRayCustomEvent(II);
+    return selectXRayCustomEvent(cast<CallInst>(II));
   case Intrinsic::xray_typedevent:
-    return selectXRayTypedEvent(II);
+    return selectXRayTypedEvent(cast<CallInst>(II));
   }
 
   return fastLowerIntrinsicCall(II);
 }
 
 bool FastISel::selectCast(const User *I, unsigned Opcode) {
   EVT SrcVT = TLI.getValueType(DL, I->getOperand(0)->getType());
   EVT DstVT = TLI.getValueType(DL, I->getType());

llvm/lib/CodeGen/ShadowStackGCLowering.cpp

void ShadowStackGCLowering::CollectRoots(Function &F) {
   SmallVector<std::pair<CallInst *, AllocaInst *>, 16> MetaRoots;
 
   for (BasicBlock &BB : F)
     for (BasicBlock::iterator II = BB.begin(), E = BB.end(); II != E;)
       if (IntrinsicInst *CI = dyn_cast<IntrinsicInst>(II++))
         if (Function *F = CI->getCalledFunction())
           if (F->getIntrinsicID() == Intrinsic::gcroot) {
             std::pair<CallInst *, AllocaInst *> Pair = std::make_pair(
-                CI,
+                cast<CallInst>(CI),
                 cast<AllocaInst>(CI->getArgOperand(0)->stripPointerCasts()));
             if (IsNullValue(CI->getArgOperand(1)))
               Roots.push_back(Pair);
             else
               MetaRoots.push_back(Pair);
           }
 
   // Number roots with metadata (usually empty) at the beginning, so that the

llvm/lib/CodeGen/StackProtector.cpp

 
 /// Search for the first call to the llvm.stackprotector intrinsic and return it
 /// if present.
 static const CallInst *findStackProtectorIntrinsic(Function &F) {
   for (const BasicBlock &BB : F)
     for (const Instruction &I : BB)
       if (const auto *II = dyn_cast<IntrinsicInst>(&I))
         if (II->getIntrinsicID() == Intrinsic::stackprotector)
-          return II;
+          return cast<CallInst>(II);
   return nullptr;
 }
 
 /// Check whether or not this function needs a stack protector based
 /// upon the stack protector level.
 ///
 /// We use two heuristics: a standard (ssp) and strong (sspstrong).
 /// The standard heuristic which will add a guard variable to functions that

llvm/lib/ExecutionEngine/Interpreter/Execution.cpp

void Interpreter::visitIntrinsicInst(IntrinsicInst &I) {
   // If it is an unknown intrinsic function, use the intrinsic lowering
   // class to transform it into hopefully tasty LLVM code.
   //
   BasicBlock::iterator Me(&I);
   BasicBlock *Parent = I.getParent();
   bool atBegin(Parent->begin() == Me);
   if (!atBegin)
     --Me;
-  IL->LowerIntrinsicCall(&I);
+  IL->LowerIntrinsicCall(cast<CallInst>(&I));
 
   // Restore the CurInst pointer to the first instruction newly inserted, if
   // any.
   if (atBegin) {
     SF.CurInst = Parent->begin();
   } else {
     SF.CurInst = Me;
     ++SF.CurInst;

llvm/lib/Target/AArch64/AArch64FastISel.cpp

if (!MTI->getLength()->getType()->isIntegerTy(64))
       return false;
 
     if (MTI->getSourceAddressSpace() > 255 || MTI->getDestAddressSpace() > 255)
       // Fast instruction selection doesn't support the special
       // address spaces.
       return false;
 
     const char *IntrMemName = isa<MemCpyInst>(II) ? "memcpy" : "memmove";
-    return lowerCallTo(II, IntrMemName, II->getNumArgOperands() - 1);
+    return lowerCallTo(cast<CallInst>(II), IntrMemName,
+                       II->getNumArgOperands() - 1);
   }
   case Intrinsic::memset: {
     const MemSetInst *MSI = cast<MemSetInst>(II);
     // Don't handle volatile.
     if (MSI->isVolatile())
       return false;
 
     if (!MSI->getLength()->getType()->isIntegerTy(64))
       return false;
 
     if (MSI->getDestAddressSpace() > 255)
       // Fast instruction selection doesn't support the special
       // address spaces.
       return false;
 
-    return lowerCallTo(II, "memset", II->getNumArgOperands() - 1);
+    return lowerCallTo(cast<CallInst>(II), "memset",
+                       II->getNumArgOperands() - 1);
   }
   case Intrinsic::sin:
   case Intrinsic::cos:
   case Intrinsic::pow: {
     MVT RetVT;
     if (!isTypeLegal(II->getType(), RetVT))
       return false;
 

llvm/lib/Target/Mips/MipsFastISel.cpp

bool MipsFastISel::fastLowerIntrinsicCall(const IntrinsicInst *II) {
   case Intrinsic::memmove: {
     const auto *MTI = cast<MemTransferInst>(II);
     // Don't handle volatile.
     if (MTI->isVolatile())
       return false;
     if (!MTI->getLength()->getType()->isIntegerTy(32))
       return false;
     const char *IntrMemName = isa<MemCpyInst>(II) ? "memcpy" : "memmove";
-    return lowerCallTo(II, IntrMemName, II->getNumArgOperands() - 1);
+    return lowerCallTo(cast<CallInst>(II), IntrMemName, II->getNumArgOperands() - 1);
   }
   case Intrinsic::memset: {
     const MemSetInst *MSI = cast<MemSetInst>(II);
     // Don't handle volatile.
     if (MSI->isVolatile())
       return false;
     if (!MSI->getLength()->getType()->isIntegerTy(32))
       return false;
-    return lowerCallTo(II, "memset", II->getNumArgOperands() - 1);
+    return lowerCallTo(cast<CallInst>(II), "memset", II->getNumArgOperands() - 1);
   }
   }
   return false;
 }
 
 bool MipsFastISel::selectRet(const Instruction *I) {
   const Function &F = *I->getParent()->getParent();
   const ReturnInst *Ret = cast<ReturnInst>(I);

llvm/lib/Target/X86/X86FastISel.cpp

case Intrinsic::memcpy: {
 
     unsigned SizeWidth = Subtarget->is64Bit() ? 64 : 32;
     if (!MCI->getLength()->getType()->isIntegerTy(SizeWidth))
       return false;
 
     if (MCI->getSourceAddressSpace() > 255 || MCI->getDestAddressSpace() > 255)
       return false;
 
-    return lowerCallTo(II, "memcpy", II->getNumArgOperands() - 1);
+    return lowerCallTo(cast<CallInst>(II), "memcpy", II->getNumArgOperands() - 1);
   }
   case Intrinsic::memset: {
     const MemSetInst *MSI = cast<MemSetInst>(II);
 
     if (MSI->isVolatile())
       return false;
 
     unsigned SizeWidth = Subtarget->is64Bit() ? 64 : 32;
     if (!MSI->getLength()->getType()->isIntegerTy(SizeWidth))
       return false;
 
     if (MSI->getDestAddressSpace() > 255)
       return false;
 
-    return lowerCallTo(II, "memset", II->getNumArgOperands() - 1);
+    return lowerCallTo(cast<CallInst>(II), "memset", II->getNumArgOperands() - 1);
   }
   case Intrinsic::stackprotector: {
     // Emit code to store the stack guard onto the stack.
     EVT PtrTy = TLI.getPointerTy(DL);
 
     const Value *Op1 = II->getArgOperand(0); // The guard's value.
     const AllocaInst *Slot = cast<AllocaInst>(II->getArgOperand(1));
 

llvm/lib/Transforms/InstCombine/InstCombineInternal.h

public:
   Instruction *visitUIToFP(CastInst &CI);
   Instruction *visitSIToFP(CastInst &CI);
   Instruction *visitPtrToInt(PtrToIntInst &CI);
   Instruction *visitIntToPtr(IntToPtrInst &CI);
   Instruction *visitBitCast(BitCastInst &CI);
   Instruction *visitAddrSpaceCast(AddrSpaceCastInst &CI);
   Instruction *foldItoFPtoI(CastInst &FI);
   Instruction *visitSelectInst(SelectInst &SI);
+  Instruction *visitCallBase(CallBase &Call);
   Instruction *visitCallInst(CallInst &CI);
   Instruction *visitInvokeInst(InvokeInst &II);
   Instruction *visitCallBrInst(CallBrInst &CBI);
 
   Instruction *SliceUpIllegalIntegerPHI(PHINode &PN);
   Instruction *visitPHINode(PHINode &PN);
   Instruction *visitGetElementPtrInst(GetElementPtrInst &GEP);
   Instruction *visitAllocaInst(AllocaInst &AI);
private:
   /// operation in OperationResult and result of the overflow check in
   /// OverflowResult, and return true.  If no simplification is possible,
   /// returns false.
   bool OptimizeOverflowCheck(Instruction::BinaryOps BinaryOp, bool IsSigned,
                              Value *LHS, Value *RHS,
                              Instruction &CtxI, Value *&OperationResult,
                              Constant *&OverflowResult);
 
-  Instruction *visitCallBase(CallBase &Call);
   Instruction *tryOptimizeCall(CallInst *CI);
   bool transformConstExprCastCall(CallBase &Call);
   Instruction *transformCallThroughTrampoline(CallBase &Call,
                                               IntrinsicInst &Tramp);
 
   Value *simplifyMaskedLoad(IntrinsicInst &II);
   Instruction *simplifyMaskedStore(IntrinsicInst &II);
   Instruction *simplifyMaskedGather(IntrinsicInst &II);

llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp

struct VarArgAMD64Helper : public VarArgHelper {
   unsigned AMD64FpEndOffset;
   Function &F;
   MemorySanitizer &MS;
   MemorySanitizerVisitor &MSV;
   Value *VAArgTLSCopy = nullptr;
   Value *VAArgTLSOriginCopy = nullptr;
   Value *VAArgOverflowSize = nullptr;
 
-  SmallVector<CallInst*, 16> VAStartInstrumentationList;
+  SmallVector<IntrinsicInst*, 16> VAStartInstrumentationList;
 
   enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
 
   VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
                     MemorySanitizerVisitor &MSV)
       : F(F), MS(MS), MSV(MSV) {
     AMD64FpEndOffset = AMD64FpEndOffsetSSE;
     for (const auto &Attr : F.getAttributes().getFnAttributes()) {
if (!VAStartInstrumentationList.empty()) {
         IRB.CreateMemCpy(VAArgTLSOriginCopy, Align(8), MS.VAArgOriginTLS,
                          Align(8), CopySize);
       }
     }
 
     // Instrument va_start.
     // Copy va_list shadow from the backup copy of the TLS contents.
     for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
-      CallInst *OrigInst = VAStartInstrumentationList[i];
+      auto *OrigInst = VAStartInstrumentationList[i];
       IRBuilder<> IRB(OrigInst->getNextNode());
       Value *VAListTag = OrigInst->getArgOperand(0);
 
       Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
       Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
           IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
                         ConstantInt::get(MS.IntptrTy, 16)),
           PointerType::get(RegSaveAreaPtrTy, 0));

 /// MIPS64-specific implementation of VarArgHelper.
 struct VarArgMIPS64Helper : public VarArgHelper {
   Function &F;
   MemorySanitizer &MS;
   MemorySanitizerVisitor &MSV;
   Value *VAArgTLSCopy = nullptr;
   Value *VAArgSize = nullptr;
 
-  SmallVector<CallInst*, 16> VAStartInstrumentationList;
+  SmallVector<IntrinsicInst*, 16> VAStartInstrumentationList;
 
   VarArgMIPS64Helper(Function &F, MemorySanitizer &MS,
                     MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
 
   void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
     unsigned VAArgOffset = 0;
     const DataLayout &DL = F.getParent()->getDataLayout();
     for (auto ArgIt = CB.arg_begin() + CB.getFunctionType()->getNumParams(),
if (!VAStartInstrumentationList.empty()) {
       // va_arg_tls somewhere in the function entry block.
       VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
       IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
     }
 
     // Instrument va_start.
     // Copy va_list shadow from the backup copy of the TLS contents.
     for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
-      CallInst *OrigInst = VAStartInstrumentationList[i];
+      auto *OrigInst = VAStartInstrumentationList[i];
       IRBuilder<> IRB(OrigInst->getNextNode());
       Value *VAListTag = OrigInst->getArgOperand(0);
       Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
       Value *RegSaveAreaPtrPtr =
           IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
                              PointerType::get(RegSaveAreaPtrTy, 0));
       Value *RegSaveAreaPtr =
           IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
struct VarArgAArch64Helper : public VarArgHelper {
   static const unsigned AArch64VAEndOffset = AArch64VrEndOffset;
 
   Function &F;
   MemorySanitizer &MS;
   MemorySanitizerVisitor &MSV;
   Value *VAArgTLSCopy = nullptr;
   Value *VAArgOverflowSize = nullptr;
 
-  SmallVector<CallInst*, 16> VAStartInstrumentationList;
+  SmallVector<IntrinsicInst*, 16> VAStartInstrumentationList;
 
   enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
 
   VarArgAArch64Helper(Function &F, MemorySanitizer &MS,
                     MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
 
   ArgKind classifyArgument(Value* arg) {
     Type *T = arg->getType();
void finalizeInstrumentation() override {
     }
 
     Value *GrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64GrArgSize);
     Value *VrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64VrArgSize);
 
     // Instrument va_start, copy va_list shadow from the backup copy of
     // the TLS contents.
     for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
-      CallInst *OrigInst = VAStartInstrumentationList[i];
+      auto *OrigInst = VAStartInstrumentationList[i];
       IRBuilder<> IRB(OrigInst->getNextNode());
 
       Value *VAListTag = OrigInst->getArgOperand(0);
 
       // The variadic ABI for AArch64 creates two areas to save the incoming
       // argument registers (one for 64-bit general register xn-x7 and another
       // for 128-bit FP/SIMD vn-v7).
       // We need then to propagate the shadow arguments on both regions

 /// PowerPC64-specific implementation of VarArgHelper.
 struct VarArgPowerPC64Helper : public VarArgHelper {
   Function &F;
   MemorySanitizer &MS;
   MemorySanitizerVisitor &MSV;
   Value *VAArgTLSCopy = nullptr;
   Value *VAArgSize = nullptr;
 
-  SmallVector<CallInst*, 16> VAStartInstrumentationList;
+  SmallVector<IntrinsicInst*, 16> VAStartInstrumentationList;
 
   VarArgPowerPC64Helper(Function &F, MemorySanitizer &MS,
                     MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
 
   void visitCallBase(CallBase &CB, IRBuilder<> &IRB) override {
     // For PowerPC, we need to deal with alignment of stack arguments -
     // they are mostly aligned to 8 bytes, but vectors and i128 arrays
     // are aligned to 16 bytes, byvals can be aligned to 8 or 16 bytes,
if (!VAStartInstrumentationList.empty()) {
       // va_arg_tls somewhere in the function entry block.
       VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
       IRB.CreateMemCpy(VAArgTLSCopy, Align(8), MS.VAArgTLS, Align(8), CopySize);
     }
 
     // Instrument va_start.
     // Copy va_list shadow from the backup copy of the TLS contents.
     for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; i++) {
-      CallInst *OrigInst = VAStartInstrumentationList[i];
+      auto *OrigInst = VAStartInstrumentationList[i];
       IRBuilder<> IRB(OrigInst->getNextNode());
       Value *VAListTag = OrigInst->getArgOperand(0);
       Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
       Value *RegSaveAreaPtrPtr =
           IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
                              PointerType::get(RegSaveAreaPtrTy, 0));
       Value *RegSaveAreaPtr =
           IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
struct VarArgSystemZHelper : public VarArgHelper {
 
   Function &F;
   MemorySanitizer &MS;
   MemorySanitizerVisitor &MSV;
   Value *VAArgTLSCopy = nullptr;
   Value *VAArgTLSOriginCopy = nullptr;
   Value *VAArgOverflowSize = nullptr;
 
-  SmallVector<CallInst *, 16> VAStartInstrumentationList;
+  SmallVector<IntrinsicInst *, 16> VAStartInstrumentationList;
 
   enum class ArgKind {
     GeneralPurpose,
     FloatingPoint,
     Vector,
     Memory,
     Indirect,
   };
if (!VAStartInstrumentationList.empty()) {
                          Align(8), CopySize);
       }
     }
 
     // Instrument va_start.
     // Copy va_list shadow from the backup copy of the TLS contents.
     for (size_t VaStartNo = 0, VaStartNum = VAStartInstrumentationList.size();
          VaStartNo < VaStartNum; VaStartNo++) {
-      CallInst *OrigInst = VAStartInstrumentationList[VaStartNo];
+      auto *OrigInst = VAStartInstrumentationList[VaStartNo];
       IRBuilder<> IRB(OrigInst->getNextNode());
       Value *VAListTag = OrigInst->getArgOperand(0);
       copyRegSaveArea(IRB, VAListTag);
       copyOverflowArea(IRB, VAListTag);
     }
   }
 };