[IR] Add operand bundles to CallInst and InvokeInst.

[sanjoy]

Summary

This change teaches CallInsts and InvokeInsts to maintain a set of
operand bundles as part of its operands. CallInsts and
InvokeInsts with operand bundles co-allocate some space before their
Use array to hold meta information
(OperandBundleUser::BundleOpInfo) about which of its operands are
part of an operand bundle.

The strings corresponding to the bundle tags are interned into
LLVMContextImpl::BundleTagCache. We could reference count these if
needed, but right now I don't see any need to do so.

This change does not include any parsing / bitcode support. That's
the next change.

Depends on D12455.

[JosephTremoulet]

include/llvm/IR/InstrTypes.h

1108

nit: Why explicit, when we have two parameters?

1131

super-nit: I think Back would be a more common name for this than InclusiveEnd

include/llvm/IR/Instructions.h

1430

Should we pad this to pointer-align the StringMapEntry<uint32_t> *Tag field of the BundleOpInfos (and likewise for the other CallInst::Create overload and the two InvokeInst::Creates)?

lib/IR/LLVMContextImpl.h

989–990

nit: worth a comment?

lib/IR/User.cpp

121–122 ↗

(On Diff #33488)

Ok, I should have read ahead :). But I still think it would be good to discuss alignment in the method header comment, and to 8-byte align the CallInst/InvokeInst bundles on 64-bit platforms for the pointer to the string map entry.

[sanjoy]

include/llvm/IR/InstrTypes.h

1108

C++11 allows implicit multi-arg constructors -- like X x = {1, 2}.

1131

Will fix.

lib/IR/LLVMContextImpl.h

989–990

Will add.

lib/IR/User.cpp

121–122 ↗

(On Diff #33488)

I intended this to be part of the first change, I must have messed up rebasing somewhere. I'll move this bit to D12455 and add a comment, as you suggested.

[sanjoy]

I don't know why this hasn't showed up on llvm-commits yet, but I've
replied to the review on phabricator.

[sanjoy]

Address review.

[sanjoy]

include/llvm/IR/Instructions.h

1430

Please do check my math, but after this current round of changes, I think the Tag field should be aligned. Now, the layout of a User (in 64bit platforms) with a BundleOpInfo should be

BundleOpInfo[0]    // addr is 8x
  Tag              // addr is 8x
  uint32_t         // addr is 8(x + 1)
  uint32_t         // addr is 8(x + 1) + 4
BundleOpInfo[1]    // addr is 8(x + 2)
  ...
... BundleOpInfo[1, 2, 3] ...
DescriptorInfo     // addr is 8y
  intptr_t         // addr is 8y
Use[]              // addr is 8(y + 1)
  ...
User               // addr is 8(y + 1 + 3 * # uses)

For 32 bit platforms, things should roughly stay the same, except the uint32_ts will take up one whole word instead.

However, I'm sure if the Begin and End fields should be uint32_ts, instead of being a native int. What do you think?

[MatzeB]

Would be good to document somewhere what an operand bundle is and maybe the motivation or an example use case for it.

[sanjoy]

This is the original discussion:
http://lists.llvm.org/pipermail/llvm-dev/2015-August/089070.html (I
should have included it with the patch to begin with).

I'll update the langref with more information in the next revision.

[JosephTremoulet]

include/llvm/IR/Instructions.h

1430

I agree with your math. I think leaving Begin and End as uint32_t makes sense; I doubt this would be the only weak link if someone tried to put 2^32 operands on a call, and even if we were to run on a platform with an odd pointer size, I'd expect sizeof(BundleOpInfo) to be a multiple of it by virtue of the Tag pointer field.

[JosephTremoulet]

I'll update the langref with more information in the next revision

(in case others get confused looking for it here like I did): the langref update went into D12457.

[sanjoy]

• address @majnemer's off-line review comment -- use an ArrayRef<> instead of a std::pair for getDescriptor()

[sanjoy]

• add more comments to OperandBundleUser
• move Use& population logic to populateBundleOperandInfos to keep everything in one place.

[sanjoy]

• address Duncan's review

[sanjoy]

• Some changes here to support changes in D12457: mainly add accessors around LLVMContextImpl::BundleTagCache.

[sanjoy]

• Supporting changes to support changes to D12457: have OperandBundleDef own the data it contains, by using std::vector and std::string instead of ArrayRef and StringRef
• Unbreak the build with GCC, by correctly using the ArrayRef<> in OperandBundleUse
• Remove 'Set' from OperandBundleSetDef and OperandBundleSetUse

[sanjoy]

Just to make it clear: this was LGTM'ed by Duncan via email (for some reason these emails did not show up here).

[dbabokin]

Sanjoy,

Your comment in include/llvm/IR/InstrTypes.h in line 1137 is undeniably
nice, but it causes a warning (or an error when building with "-Wall"):

“error: multi-line comment [-Werror=comment]”

Could you please fix it? For example you can use | instead of \, or add
some "shielding" (|, ///, or whatever you prefer) in the leftmost position
of the whole pseudo graphics (i.e. lines 1137-1149.

Thanks!

Dmitry.

[sanjoy]

Dmitry Babokin via llvm-commits wrote:
> dbabokin added a subscriber: dbabokin.
> dbabokin added a comment.
>
> Sanjoy,
>
> Your comment in include/llvm/IR/InstrTypes.h in line 1137 is undeniably
> nice, but it causes a warning (or an error when building with "-Wall"):
>
> “error: multi-line comment [-Werror=comment]”
>
> Could you please fix it? For example you can use | instead of , or add
> some "shielding" (|, ///, or whatever you prefer) in the leftmost
position
> of the whole pseudo graphics (i.e. lines 1137-1149.

I'm more than happy to fix this, but I'm not able to reproduce this
issue using clang++ on OSX (my default build environment).

Can you please verify that the change below fixes the issue?

diff --git a/include/llvm/IR/InstrTypes.h b/include/llvm/IR/InstrTypes.h
index 70ab973..0c2ff91 100644

• a/include/llvm/IR/InstrTypes.h

+++ b/include/llvm/IR/InstrTypes.h
@@ -1134,19 +1134,19 @@ typedef OperandBundleDefT<const Value *>
ConstOperandBundleDef;

///
/// The layout of an operand bundle user is
///

-/ +-------uint32_t End---------------------------------+
-/ /
-/// / +------uint32_t Begin------------------+

-///           /  / 
  +///          +-----------uint32_t

End-------------------------------------+
+/ | |
+/ | +--------uint32_t Begin--------------------+ |
+/// | | | |

///          ^  ^                                          v 
   v
///

///  | BOI0 | BOI1 | .. | DU | U0 | U1 | .. | BOI0_U0 | .. | BOI1_U0 |

.. | Un

///

///   v  v                                  ^              ^

-/ / /
-/ +------uint32_t Begin----------+ /
-/ /
-/ +-------uint32_t End-------------------------+
+/ | | | |
+/ | +--------uint32_t Begin------------+ |
+/ | |
+/ +-----------uint32_t End-----------------------------+

///
///
/// BOI0, BOI1 ... are descriptions of operand bundles in this User's

use list.

>
> Thanks!
>
> Dmitry.
>
>
> Repository:
> rL LLVM
>
> http://reviews.llvm.org/D12456
>
>
>
> _______________________________________________
> llvm-commits mailing list
> [e-mail]
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits

[sanjoy]

Sanjoy Das wrote:

sanjoy added a comment.

Dmitry Babokin via llvm-commits wrote:

>  dbabokin added a subscriber: dbabokin.
>  dbabokin added a comment.
>
>  Sanjoy,
>
>  Your comment in include/llvm/IR/InstrTypes.h in line 1137 is undeniably
>  nice, but it causes a warning (or an error when building with "-Wall"):
>
>  “error: multi-line comment [-Werror=comment]”
>
>  Could you please fix it? For example you can use | instead of , or add
>  some "shielding" (|, ///, or whatever you prefer) in the leftmost

position

>  of the whole pseudo graphics (i.e. lines 1137-1149.

I'm more than happy to fix this, but I'm not able to reproduce this
issue using clang++ on OSX (my default build environment).

Can you please verify that the change below fixes the issue?

Looks like my mail client screwed up the diff. Sending it as an
attachment this time.

diff --git a/include/llvm/IR/InstrTypes.h b/include/llvm/IR/InstrTypes.h
index 70ab973..0c2ff91 100644

• a/include/llvm/IR/InstrTypes.h

+++ b/include/llvm/IR/InstrTypes.h
@@ -1134,19 +1134,19 @@ typedef OperandBundleDefT<const Value *>
ConstOperandBundleDef;

///
/// The layout of an operand bundle user is
///

-/ +-------uint32_t End---------------------------------+
-/ /
-/// / +------uint32_t Begin------------------+

-///           /  /
  +///          +-----------uint32_t

End-------------------------------------+
+/ | |
+/ | +--------uint32_t Begin--------------------+ |
+/// | | | |

///          ^  ^                                          v
   v
///

///  | BOI0 | BOI1 | .. | DU | U0 | U1 | .. | BOI0_U0 | .. | BOI1_U0 |

.. | Un

///

///   v  v                                  ^              ^

-/ / /
-/ +------uint32_t Begin----------+ /
-/ /
-/ +-------uint32_t End-------------------------+
+/ | | | |
+/ | +--------uint32_t Begin------------+ |
+/ | |
+/ +-----------uint32_t End-----------------------------+

///
///
/// BOI0, BOI1 ... are descriptions of operand bundles in this User's

use list.

>  Thanks!
>
>  Dmitry.
>
>
>  Repository:
>     http://reviews.llvm.org/diffusion/L/ LLVM
>
>  http://reviews.llvm.org/D12456
>
>
>
>  _______________________________________________
>  llvm-commits mailing list
>  [e-mail]
>  http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits

Repository:

rL LLVM

http://reviews.llvm.org/D12456

• fix.diff1 KBDownload

[dbabokin]

Sanjoy,

Thanks for the fix, it works.

I forgot to mention that the problem shows up when building with gcc (we
use 4.9.1). We have to use gcc on Linux when building our project, which
includes LLVM libraries and hence we include include/llvm/* stuff.

Thanks again for quick response.

[sanjoy]

Dmitry Babokin via llvm-commits wrote:

dbabokin added a comment.

Sanjoy,

Thanks for the fix, it works.

Checked in at r249212.

• Sanjoy

I forgot to mention that the problem shows up when building with gcc (we
use 4.9.1). We have to use gcc on Linux when building our project, which
includes LLVM libraries and hence we include include/llvm/* stuff.

Thanks again for quick response.

Repository:

rL LLVM

http://reviews.llvm.org/D12456

---

llvm-commits mailing list
[e-mail]
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits

include/llvm/IR/CallSite.h

 
 class CallInst;
 class InvokeInst;
 
 template <typename FunTy = const Function,
           typename BBTy = const BasicBlock,
           typename ValTy = const Value,
           typename UserTy = const User,
+          typename UseTy = const Use,
           typename InstrTy = const Instruction,
           typename CallTy = const CallInst,
           typename InvokeTy = const InvokeInst,
           typename IterTy = User::const_op_iterator>
 class CallSiteBase {
 protected:
   PointerIntPair<InstrTy*, 1, bool> I;
 
#define CALLSITE_DELEGATE_SETTER(METHOD) \
   /// @brief Determine if the call cannot unwind.
   bool doesNotThrow() const {
     CALLSITE_DELEGATE_GETTER(doesNotThrow());
   }
   void setDoesNotThrow() {
     CALLSITE_DELEGATE_SETTER(setDoesNotThrow());
   }
 
+  int getNumOperandBundles() const {
+    CALLSITE_DELEGATE_GETTER(getNumOperandBundles());
+  }
+
+  bool hasOperandBundles() const {
+    CALLSITE_DELEGATE_GETTER(hasOperandBundles());
+  }
+
+  int getNumTotalBundleOperands() const {
+    CALLSITE_DELEGATE_GETTER(getNumTotalBundleOperands());
+  }
+
+  OperandBundleSetT<UseTy> getOperandBundle(unsigned Index) const {
+    CALLSITE_DELEGATE_GETTER(getOperandBundle(Index));
+  }
+
 #undef CALLSITE_DELEGATE_GETTER
 #undef CALLSITE_DELEGATE_SETTER
 
   /// @brief Determine whether this argument is not captured.
   bool doesNotCapture(unsigned ArgNo) const {
     return paramHasAttr(ArgNo + 1, Attribute::NoCapture);
   }
 
for (arg_iterator AI = this->arg_begin(), E = this->arg_end(); AI != E;
          ++AI)
       if (AI->get() == Arg)
         return true;
     return false;
   }
 
 private:
   unsigned getArgumentEndOffset() const {
-    if (isCall())
-      return 1; // Skip Callee
-    else
-      return 3; // Skip BB, BB, Callee
+    if (isCall()) {
+      // Skip [ operand bundles ], Callee
+      auto *CI = cast<CallInst>(getInstruction());
+      return 1 + CI->getNumTotalBundleOperands();
+    } else {
+      // Skip [ operand bundles ], BB, BB, Callee
+      auto *II = cast<InvokeInst>(getInstruction());
+      return 3 + II->getNumTotalBundleOperands();
+    }
   }
 
   IterTy getCallee() const {
     if (isCall()) // Skip Callee
       return cast<CallInst>(getInstruction())->op_end() - 1;
     else // Skip BB, BB, Callee
       return cast<InvokeInst>(getInstruction())->op_end() - 3;
   }
 };
 
-class CallSite : public CallSiteBase<Function, BasicBlock, Value, User,
+class CallSite : public CallSiteBase<Function, BasicBlock, Value, User, Use,
                                      Instruction, CallInst, InvokeInst,
                                      User::op_iterator> {
 public:
   CallSite() {}
   CallSite(CallSiteBase B) : CallSiteBase(B) {}
   CallSite(CallInst *CI) : CallSiteBase(CI) {}
   CallSite(InvokeInst *II) : CallSiteBase(II) {}
   explicit CallSite(Instruction *II) : CallSiteBase(II) {}

include/llvm/IR/InstrTypes.h

 
 // FIXME: these are redundant if CmpInst < BinaryOperator
 template <>
 struct OperandTraits<CmpInst> : public FixedNumOperandTraits<CmpInst, 2> {
 };
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CmpInst, Value)
 
+/// \brief A lightweight accessor for a set of operand bundles meant to be
+/// passed around by value.
+///
+/// There are two distinct views of a set of operand bundles: a "use" view
+/// (OperandBundleSetUse) and a "def" view (OperandBundleSetDef).  The former is
+/// more useful when viewing or editing the operand bundle set of an existing
+/// instruction, while the latter is useful when viewing an operand bundle as a
+/// distinct entity.
+template <typename InputTy> struct OperandBundleSetT {
+  StringRef Tag;
+  ArrayRef<InputTy> Inputs;
+
+  OperandBundleSetT() {}
+  explicit OperandBundleSetT(StringRef Tag, ArrayRef<InputTy> Inputs)
+      : Tag(Tag), Inputs(Inputs) {}
+};
+
+typedef OperandBundleSetT<Use> OperandBundleSetUse;
+typedef OperandBundleSetT<const Use> ConstOperandBundleSetUse;
+
+typedef OperandBundleSetT<Value *> OperandBundleSetDef;
+typedef OperandBundleSetT<const Value *> ConstOperandBundleSetDef;
+
+/// \brief A mixin to add operand bundle functionality to llvm instruction
+/// classes.
+///
+/// OperandBundleUser uses the descriptor area co-allocated with the host User
+/// to store some meta information about which operands are "normal" operands,
+/// and which ones belong to some operand bundle.
+///
+/// The layout of an operand bundle user is
+///
+///              +-------uint32_t End---------------------------------+
+///             /                                                      \
+///            /  +------uint32_t Begin------------------+              \
+///           /  /                                        \              \
+///          ^  ^                                          v              v
+///  |------|------|----|----|----|----|----|---------|----|---------|----|-----
+///  | BOI0 | BOI1 | .. | DU | U0 | U1 | .. | BOI0_U0 | .. | BOI1_U0 | .. | User
+///  |------|------|----|----|----|----|----|---------|----|---------|----|-----
+///   v  v                                  ^              ^
+///    \  \                                /              /
+///     \  +------uint32_t Begin----------+              /
+///      \                                              /
+///       +-------uint32_t End-------------------------+
+///
+///
+/// BOI0, BOI1 ... are descriptions of operand bundles in this User's use list.
+/// These descriptions are installed and managed by this class, and they're all
+/// instances of OperandBundleUser<T>::BundleOpInfo.
+///
+/// DU is an additional descriptor installed by User's 'operator new' to keep
+/// track of the additional 'BOI0 ... BOIN' co-allocation.  OperandBundleUser
+/// does not access or modify DU in any way, it's an implementation detail
+/// private to User.
+///
+/// The regular Use& vector for the User starts at U0.  The operand bundle uses
+/// are part of the Use& vector, just like normal uses.  In the diagram above,
+/// these start at OB1_U0.  Each instance of BundleOpInfo has information about
+/// a contiguous set of uses constituting an operand bundle.  All of these uses
+/// corresponding to bundle operands are appended *after* the normal uses.
+///
+/// Currently operand bundle users with hung-off operands are not supported.
+template <typename InstrTy, typename OpIteratorTy> class OperandBundleUser {
+public:
+  /// \brief Return the number of operand bundles associated with this User.
+  unsigned getNumOperandBundles() const {
+    return std::distance(bundle_op_info_begin(), bundle_op_info_end());
+  }
+
+  /// \brief Return true if this User has any operand bundles.
+  bool hasOperandBundles() const { return getNumOperandBundles() != 0; }
+
+  /// \brief Return the total number operands (not operand bundles) used by
+  /// every operand bundle in this OperandBundleUser.
+  unsigned getNumTotalBundleOperands() const {
+    if (!hasOperandBundles())
+      return 0;
+
+    auto *Begin = bundle_op_info_begin();
+    auto *Back = bundle_op_info_end() - 1;
+
+    assert(Begin <= Back && "hasOperandBundles() returned true!");
+
+    return Back->End - Begin->Begin;
+  }
+
+  /// \brief Return the operand bundle at a specific index.
+  ConstOperandBundleSetUse getOperandBundle(unsigned Index) const {
+    assert(Index < getNumOperandBundles() && "Index out of bounds!");
+    auto *BOI = bundle_op_info_begin() + Index;
+    auto op_begin = static_cast<const InstrTy *>(this)->op_begin();
+    ArrayRef<const Use> Inputs(op_begin + BOI->Begin, op_begin + BOI->End);
+    return ConstOperandBundleSetUse(BOI->Tag->getKey(), Inputs);
+  }
+
+  /// \brief Return the operand bundle at a specific index.
+  OperandBundleSetUse getOperandBundle(unsigned Index) {
+    assert(Index < getNumOperandBundles() && "Index out of bounds!");
+    auto *BOI = bundle_op_info_begin() + Index;
+    auto op_begin = static_cast<InstrTy *>(this)->op_begin();
+    ArrayRef<Use> Inputs(op_begin + BOI->Begin, op_begin + BOI->End);
+    return OperandBundleSetUse(BOI->Tag->getKey(), Inputs);
+  }
+
+protected:
+  /// \brief Used to keep track of an operand bundle.  See the main comment on
+  /// OperandBundleUser above.
+  struct BundleOpInfo {
+    /// \brief The operand bundle tag, interned by
+    /// LLVMContextImpl::getOrInsertBundleTag.
+    StringMapEntry<uint32_t> *Tag;
+
+    /// \brief The index in the Use& vector where operands for this operand
+    /// bundle starts.
+    uint32_t Begin;
+
+    /// \brief The index in the Use& vector where operands for this operand
+    /// bundle ends.
+    uint32_t End;
+  };
+
+  typedef BundleOpInfo *bundle_op_iterator;
+  typedef const BundleOpInfo *const_bundle_op_iterator;
+
+  /// \brief Return the start of the list of BundleOpInfo instances associated
+  /// with this OperandBundleUser.
+  bundle_op_iterator bundle_op_info_begin() {
+    if (!static_cast<InstrTy *>(this)->hasDescriptor())
+      return nullptr;
+
+    uint8_t *BytesBegin = static_cast<InstrTy *>(this)->getDescriptor().begin();
+    return reinterpret_cast<bundle_op_iterator>(BytesBegin);
+  }
+
+  /// \brief Return the start of the list of BundleOpInfo instances associated
+  /// with this OperandBundleUser.
+  const_bundle_op_iterator bundle_op_info_begin() const {
+    auto *NonConstThis =
+        const_cast<OperandBundleUser<InstrTy, OpIteratorTy> *>(this);
+    return NonConstThis->bundle_op_info_begin();
+  }
+
+  /// \brief Return the end of the list of BundleOpInfo instances associated
+  /// with this OperandBundleUser.
+  bundle_op_iterator bundle_op_info_end() {
+    if (!static_cast<InstrTy *>(this)->hasDescriptor())
+      return nullptr;
+
+    uint8_t *BytesEnd = static_cast<InstrTy *>(this)->getDescriptor().end();
+    return reinterpret_cast<bundle_op_iterator>(BytesEnd);
+  }
+
+  /// \brief Return the end of the list of BundleOpInfo instances associated
+  /// with this OperandBundleUser.
+  const_bundle_op_iterator bundle_op_info_end() const {
+    auto *NonConstThis =
+        const_cast<OperandBundleUser<InstrTy, OpIteratorTy> *>(this);
+    return NonConstThis->bundle_op_info_end();
+  }
+
+  /// \brief Return the range [\p bundle_op_info_begin, \p bundle_op_info_end).
+  iterator_range<bundle_op_iterator> bundle_op_infos() {
+    return iterator_range<bundle_op_iterator>(bundle_op_info_begin(),
+                                              bundle_op_info_end());
+  }
+
+  /// \brief Return the range [\p bundle_op_info_begin, \p bundle_op_info_end).
+  iterator_range<const_bundle_op_iterator> bundle_op_infos() const {
+    return iterator_range<const_bundle_op_iterator>(bundle_op_info_begin(),
+                                                    bundle_op_info_end());
+  }
+
+  /// \brief Populate the BundleOpInfo instances and the Use& vector from \p
+  /// Bundles.  Return the op_iterator pointing to the Use& one past the last
+  /// last bundle operand use.
+  ///
+  /// Each \p OperandBundleSetDef instance is tracked by a OperandBundleInfo
+  /// instance allocated in this User's descriptor.
+  OpIteratorTy populateBundleOperandInfos(ArrayRef<OperandBundleSetDef> Bundles,
+                                          const unsigned BeginIndex) {
+    auto It = static_cast<InstrTy *>(this)->op_begin() + BeginIndex;
+    for (auto &B : Bundles)
+      It = std::copy(B.Inputs.begin(), B.Inputs.end(), It);
+
+    auto *ContextImpl = static_cast<InstrTy *>(this)->getContext().pImpl;
+    auto BI = Bundles.begin();
+    unsigned CurrentIndex = BeginIndex;
+
+    for (auto &BOI : bundle_op_infos()) {
+      assert(BI != Bundles.end() && "Incorrect allocation?");
+
+      BOI.Tag = ContextImpl->getOrInsertBundleTag(BI->Tag);
+      BOI.Begin = CurrentIndex;
+      BOI.End = CurrentIndex + BI->Inputs.size();
+      CurrentIndex = BOI.End;
+      BI++;
+    }
+
+    assert(BI == Bundles.end() && "Incorrect allocation?");
+
+    return It;
+  }
+
+  /// \brief Return the total number of values used in \p Bundles.
+  static unsigned CountBundleInputs(ArrayRef<OperandBundleSetDef> Bundles) {
+    unsigned Total = 0;
+    for (auto &B : Bundles)
+      Total += B.Inputs.size();
+    return Total;
+  }
+};
+
 } // End llvm namespace
 
 #endif

include/llvm/IR/Instructions.h

 };
 
 //===----------------------------------------------------------------------===//
 /// CallInst - This class represents a function call, abstracting a target
 /// machine's calling convention.  This class uses low bit of the SubClassData
 /// field to indicate whether or not this is a tail call.  The rest of the bits
 /// hold the calling convention of the call.
 ///
-class CallInst : public Instruction {
+class CallInst : public Instruction,
+                 public OperandBundleUser<CallInst, User::op_iterator> {
   AttributeSet AttributeList; ///< parameter attributes for call
   FunctionType *FTy;
   CallInst(const CallInst &CI);
-  void init(Value *Func, ArrayRef<Value *> Args, const Twine &NameStr) {
+  void init(Value *Func, ArrayRef<Value *> Args,
+            ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr) {
     init(cast<FunctionType>(
              cast<PointerType>(Func->getType())->getElementType()),
-         Func, Args, NameStr);
+         Func, Args, Bundles, NameStr);
   }
   void init(FunctionType *FTy, Value *Func, ArrayRef<Value *> Args,
-            const Twine &NameStr);
+            ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr);
   void init(Value *Func, const Twine &NameStr);
 
   /// Construct a CallInst given a range of arguments.
   /// \brief Construct a CallInst from a range of arguments
   inline CallInst(FunctionType *Ty, Value *Func, ArrayRef<Value *> Args,
-                  const Twine &NameStr, Instruction *InsertBefore);
-  inline CallInst(Value *Func, ArrayRef<Value *> Args, const Twine &NameStr,
+                  ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr,
+                  Instruction *InsertBefore);
+  inline CallInst(Value *Func, ArrayRef<Value *> Args,
+                  ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr,
                   Instruction *InsertBefore)
       : CallInst(cast<FunctionType>(
                      cast<PointerType>(Func->getType())->getElementType()),
-                 Func, Args, NameStr, InsertBefore) {}
+                 Func, Args, Bundles, NameStr, InsertBefore) {}
+
+  inline CallInst(Value *Func, ArrayRef<Value *> Args, const Twine &NameStr,
+                  Instruction *InsertBefore)
+      : CallInst(Func, Args, None, NameStr, InsertBefore) {}
 
   /// Construct a CallInst given a range of arguments.
   /// \brief Construct a CallInst from a range of arguments
   inline CallInst(Value *Func, ArrayRef<Value *> Args,
-                  const Twine &NameStr, BasicBlock *InsertAtEnd);
+                  ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr,
+                  BasicBlock *InsertAtEnd);
 
   explicit CallInst(Value *F, const Twine &NameStr,
                     Instruction *InsertBefore);
   CallInst(Value *F, const Twine &NameStr, BasicBlock *InsertAtEnd);
+
+  friend class OperandBundleUser<CallInst, User::op_iterator>;
+  bool hasDescriptor() const { return HasDescriptor; }
+
 protected:
   // Note: Instruction needs to be a friend here to call cloneImpl.
   friend class Instruction;
   CallInst *cloneImpl() const;
 
 public:
-  static CallInst *Create(Value *Func,
-                          ArrayRef<Value *> Args,
+  static CallInst *Create(Value *Func, ArrayRef<Value *> Args,
+                          ArrayRef<OperandBundleSetDef> Bundles = None,
                           const Twine &NameStr = "",
                           Instruction *InsertBefore = nullptr) {
     return Create(cast<FunctionType>(
                       cast<PointerType>(Func->getType())->getElementType()),
-                  Func, Args, NameStr, InsertBefore);
+                  Func, Args, Bundles, NameStr, InsertBefore);
+  }
+  static CallInst *Create(Value *Func, ArrayRef<Value *> Args,
+                          const Twine &NameStr,
+                          Instruction *InsertBefore = nullptr) {
+    return Create(cast<FunctionType>(
+                      cast<PointerType>(Func->getType())->getElementType()),
+                  Func, Args, None, NameStr, InsertBefore);
   }
   static CallInst *Create(FunctionType *Ty, Value *Func, ArrayRef<Value *> Args,
-                          const Twine &NameStr = "",
+                          const Twine &NameStr,
                           Instruction *InsertBefore = nullptr) {
     return new (unsigned(Args.size() + 1))
-        CallInst(Ty, Func, Args, NameStr, InsertBefore);
+        CallInst(Ty, Func, Args, None, NameStr, InsertBefore);
   }
-  static CallInst *Create(Value *Func,
-                          ArrayRef<Value *> Args,
+  static CallInst *Create(FunctionType *Ty, Value *Func, ArrayRef<Value *> Args,
+                          ArrayRef<OperandBundleSetDef> Bundles = None,
+                          const Twine &NameStr = "",
+                          Instruction *InsertBefore = nullptr) {
+    const unsigned TotalOps =
+        unsigned(Args.size()) + CountBundleInputs(Bundles) + 1;
+    const unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (TotalOps, DescriptorBytes)
+        CallInst(Ty, Func, Args, Bundles, NameStr, InsertBefore);
+  }
+  static CallInst *Create(Value *Func, ArrayRef<Value *> Args,
+                          ArrayRef<OperandBundleSetDef> Bundles,
+                          const Twine &NameStr, BasicBlock *InsertAtEnd) {
+    const unsigned TotalOps =
+        unsigned(Args.size()) + CountBundleInputs(Bundles) + 1;
+    const unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (TotalOps, DescriptorBytes)
+        CallInst(Func, Args, Bundles, NameStr, InsertAtEnd);
+  }
+  static CallInst *Create(Value *Func, ArrayRef<Value *> Args,
                           const Twine &NameStr, BasicBlock *InsertAtEnd) {
-    return new(unsigned(Args.size() + 1))
-      CallInst(Func, Args, NameStr, InsertAtEnd);
+    return new (unsigned(Args.size() + 1))
+        CallInst(Func, Args, None, NameStr, InsertAtEnd);
   }
   static CallInst *Create(Value *F, const Twine &NameStr = "",
                           Instruction *InsertBefore = nullptr) {
     return new(1) CallInst(F, NameStr, InsertBefore);
   }
   static CallInst *Create(Value *F, const Twine &NameStr,
                           BasicBlock *InsertAtEnd) {
     return new(1) CallInst(F, NameStr, InsertAtEnd);
setInstructionSubclassData((getSubclassDataFromInstruction() & ~3) |
                                unsigned(TCK));
   }
 
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   /// getNumArgOperands - Return the number of call arguments.
   ///
-  unsigned getNumArgOperands() const { return getNumOperands() - 1; }
+  unsigned getNumArgOperands() const {
+    return getNumOperands() - getNumTotalBundleOperands() - 1;
+  }
 
   /// getArgOperand/setArgOperand - Return/set the i-th call argument.
   ///
   Value *getArgOperand(unsigned i) const {
     assert(i < getNumArgOperands() && "Out of bounds!");
     return getOperand(i);
   }
   void setArgOperand(unsigned i, Value *v) {
     assert(i < getNumArgOperands() && "Out of bounds!");
     setOperand(i, v);
   }
 
   /// arg_operands - iteration adapter for range-for loops.
   iterator_range<op_iterator> arg_operands() {
     // The last operand in the op list is the callee - it's not one of the args
     // so we don't want to iterate over it.
-    return iterator_range<op_iterator>(op_begin(), op_end() - 1);
+    return iterator_range<op_iterator>(
+        op_begin(), op_end() - getNumTotalBundleOperands() - 1);
   }
 
   /// arg_operands - iteration adapter for range-for loops.
   iterator_range<const_op_iterator> arg_operands() const {
-    return iterator_range<const_op_iterator>(op_begin(), op_end() - 1);
+    return iterator_range<const_op_iterator>(
+        op_begin(), op_end() - getNumTotalBundleOperands() - 1);
   }
 
   /// \brief Wrappers for getting the \c Use of a call argument.
   const Use &getArgOperandUse(unsigned i) const {
     assert(i < getNumArgOperands() && "Out of bounds!");
     return getOperandUse(i);
   }
   Use &getArgOperandUse(unsigned i) {
private:
   }
 };
 
 template <>
 struct OperandTraits<CallInst> : public VariadicOperandTraits<CallInst, 1> {
 };
 
 CallInst::CallInst(Value *Func, ArrayRef<Value *> Args,
-                   const Twine &NameStr, BasicBlock *InsertAtEnd)
-  : Instruction(cast<FunctionType>(cast<PointerType>(Func->getType())
-                                   ->getElementType())->getReturnType(),
-                Instruction::Call,
-                OperandTraits<CallInst>::op_end(this) - (Args.size() + 1),
-                unsigned(Args.size() + 1), InsertAtEnd) {
-  init(Func, Args, NameStr);
+                   ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr,
+                   BasicBlock *InsertAtEnd)
+    : Instruction(
+          cast<FunctionType>(cast<PointerType>(Func->getType())
+          Instruction::Call, OperandTraits<CallInst>::op_end(this) -
+                                 (Args.size() + CountBundleInputs(Bundles) + 1),
+          unsigned(Args.size() + CountBundleInputs(Bundles) + 1), InsertAtEnd) {
+  init(Func, Args, Bundles, NameStr);
 }
 
 CallInst::CallInst(FunctionType *Ty, Value *Func, ArrayRef<Value *> Args,
-                   const Twine &NameStr, Instruction *InsertBefore)
+                   ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr,
+                   Instruction *InsertBefore)
     : Instruction(Ty->getReturnType(), Instruction::Call,
-                  OperandTraits<CallInst>::op_end(this) - (Args.size() + 1),
-                  unsigned(Args.size() + 1), InsertBefore) {
-  init(Ty, Func, Args, NameStr);
+                  OperandTraits<CallInst>::op_end(this) -
+                      (Args.size() + CountBundleInputs(Bundles) + 1),
+                  unsigned(Args.size() + CountBundleInputs(Bundles) + 1),
+                  InsertBefore) {
+  init(Ty, Func, Args, Bundles, NameStr);
 }
 
 
 // Note: if you get compile errors about private methods then
 //       please update your code to use the high-level operand
 //       interfaces. See line 943 above.
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CallInst, Value)
 

 
 //===----------------------------------------------------------------------===//
 //                               InvokeInst Class
 //===----------------------------------------------------------------------===//
 
 /// InvokeInst - Invoke instruction.  The SubclassData field is used to hold the
 /// calling convention of the call.
 ///
-class InvokeInst : public TerminatorInst {
+class InvokeInst : public TerminatorInst,
+                   public OperandBundleUser<InvokeInst, User::op_iterator> {
   AttributeSet AttributeList;
   FunctionType *FTy;
   InvokeInst(const InvokeInst &BI);
   void init(Value *Func, BasicBlock *IfNormal, BasicBlock *IfException,
-            ArrayRef<Value *> Args, const Twine &NameStr) {
+            ArrayRef<Value *> Args, ArrayRef<OperandBundleSetDef> Bundles,
+            const Twine &NameStr) {
     init(cast<FunctionType>(
              cast<PointerType>(Func->getType())->getElementType()),
-         Func, IfNormal, IfException, Args, NameStr);
+         Func, IfNormal, IfException, Args, Bundles, NameStr);
   }
   void init(FunctionType *FTy, Value *Func, BasicBlock *IfNormal,
             BasicBlock *IfException, ArrayRef<Value *> Args,
-            const Twine &NameStr);
+            ArrayRef<OperandBundleSetDef> Bundles, const Twine &NameStr);
 
   /// Construct an InvokeInst given a range of arguments.
   ///
   /// \brief Construct an InvokeInst from a range of arguments
   inline InvokeInst(Value *Func, BasicBlock *IfNormal, BasicBlock *IfException,
-                    ArrayRef<Value *> Args, unsigned Values,
+                    ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleSetDef> Bundles, unsigned Values,
                     const Twine &NameStr, Instruction *InsertBefore)
       : InvokeInst(cast<FunctionType>(
                        cast<PointerType>(Func->getType())->getElementType()),
-                   Func, IfNormal, IfException, Args, Values, NameStr,
+                   Func, IfNormal, IfException, Args, Bundles, Values, NameStr,
                    InsertBefore) {}
 
   inline InvokeInst(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
                     BasicBlock *IfException, ArrayRef<Value *> Args,
-                    unsigned Values, const Twine &NameStr,
-                    Instruction *InsertBefore);
+                    ArrayRef<OperandBundleSetDef> Bundles, unsigned Values,
+                    const Twine &NameStr, Instruction *InsertBefore);
   /// Construct an InvokeInst given a range of arguments.
   ///
   /// \brief Construct an InvokeInst from a range of arguments
   inline InvokeInst(Value *Func, BasicBlock *IfNormal, BasicBlock *IfException,
-                    ArrayRef<Value *> Args, unsigned Values,
+                    ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleSetDef> Bundles, unsigned Values,
                     const Twine &NameStr, BasicBlock *InsertAtEnd);
+
+  friend class OperandBundleUser<InvokeInst, User::op_iterator>;
+  bool hasDescriptor() const { return HasDescriptor; }
+
 protected:
   // Note: Instruction needs to be a friend here to call cloneImpl.
   friend class Instruction;
   InvokeInst *cloneImpl() const;
 
 public:
-  static InvokeInst *Create(Value *Func,
-                            BasicBlock *IfNormal, BasicBlock *IfException,
-                            ArrayRef<Value *> Args, const Twine &NameStr = "",
+  static InvokeInst *Create(Value *Func, BasicBlock *IfNormal,
+                            BasicBlock *IfException, ArrayRef<Value *> Args,
+                            const Twine &NameStr,
                             Instruction *InsertBefore = nullptr) {
     return Create(cast<FunctionType>(
                       cast<PointerType>(Func->getType())->getElementType()),
-                  Func, IfNormal, IfException, Args, NameStr, InsertBefore);
+                  Func, IfNormal, IfException, Args, None, NameStr,
+                  InsertBefore);
   }
-  static InvokeInst *Create(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
+  static InvokeInst *Create(Value *Func, BasicBlock *IfNormal,
                             BasicBlock *IfException, ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleSetDef> Bundles = None,
                             const Twine &NameStr = "",
                             Instruction *InsertBefore = nullptr) {
+    return Create(cast<FunctionType>(
+                      cast<PointerType>(Func->getType())->getElementType()),
+                  Func, IfNormal, IfException, Args, Bundles, NameStr,
+                  InsertBefore);
+  }
+  static InvokeInst *Create(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
+                            BasicBlock *IfException, ArrayRef<Value *> Args,
+                            const Twine &NameStr,
+                            Instruction *InsertBefore = nullptr) {
     unsigned Values = unsigned(Args.size()) + 3;
-    return new (Values) InvokeInst(Ty, Func, IfNormal, IfException, Args,
+    return new (Values) InvokeInst(Ty, Func, IfNormal, IfException, Args, None,
                                    Values, NameStr, InsertBefore);
   }
+  static InvokeInst *Create(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
+                            BasicBlock *IfException, ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleSetDef> Bundles = None,
+                            const Twine &NameStr = "",
+                            Instruction *InsertBefore = nullptr) {
+    unsigned Values = unsigned(Args.size()) + CountBundleInputs(Bundles) + 3;
+    unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (Values, DescriptorBytes)
+        InvokeInst(Ty, Func, IfNormal, IfException, Args, Bundles, Values,
+                   NameStr, InsertBefore);
+  }
   static InvokeInst *Create(Value *Func,
                             BasicBlock *IfNormal, BasicBlock *IfException,
                             ArrayRef<Value *> Args, const Twine &NameStr,
                             BasicBlock *InsertAtEnd) {
     unsigned Values = unsigned(Args.size()) + 3;
-    return new(Values) InvokeInst(Func, IfNormal, IfException, Args,
+    return new (Values) InvokeInst(Func, IfNormal, IfException, Args, None,
                                    Values, NameStr, InsertAtEnd);
   }
+  static InvokeInst *Create(Value *Func, BasicBlock *IfNormal,
+                            BasicBlock *IfException, ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleSetDef> Bundles,
+                            const Twine &NameStr, BasicBlock *InsertAtEnd) {
+    unsigned Values = unsigned(Args.size()) + CountBundleInputs(Bundles) + 3;
+    unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (Values, DescriptorBytes)
+        InvokeInst(Func, IfNormal, IfException, Args, Bundles, Values, NameStr,
+                   InsertAtEnd);
+  }
 
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   FunctionType *getFunctionType() const { return FTy; }
 
   void mutateFunctionType(FunctionType *FTy) {
     mutateType(FTy->getReturnType());
     this->FTy = FTy;
   }
 
   /// getNumArgOperands - Return the number of invoke arguments.
   ///
-  unsigned getNumArgOperands() const { return getNumOperands() - 3; }
+  unsigned getNumArgOperands() const {
+    return getNumOperands() - getNumTotalBundleOperands() - 3;
+  }
 
   /// getArgOperand/setArgOperand - Return/set the i-th invoke argument.
   ///
   Value *getArgOperand(unsigned i) const {
     assert(i < getNumArgOperands() && "Out of bounds!");
     return getOperand(i);
   }
   void setArgOperand(unsigned i, Value *v) {
     assert(i < getNumArgOperands() && "Out of bounds!");
     setOperand(i, v);
   }
 
   /// arg_operands - iteration adapter for range-for loops.
   iterator_range<op_iterator> arg_operands() {
-    return iterator_range<op_iterator>(op_begin(), op_end() - 3);
+    return iterator_range<op_iterator>(
+        op_begin(), op_end() - getNumTotalBundleOperands() - 3);
   }
 
   /// arg_operands - iteration adapter for range-for loops.
   iterator_range<const_op_iterator> arg_operands() const {
-    return iterator_range<const_op_iterator>(op_begin(), op_end() - 3);
+    return iterator_range<const_op_iterator>(
+        op_begin(), op_end() - getNumTotalBundleOperands() - 3);
   }
 
   /// \brief Wrappers for getting the \c Use of a invoke argument.
   const Use &getArgOperandUse(unsigned i) const {
     assert(i < getNumArgOperands() && "Out of bounds!");
     return getOperandUse(i);
   }
   Use &getArgOperandUse(unsigned i) {

 };
 
 template <>
 struct OperandTraits<InvokeInst> : public VariadicOperandTraits<InvokeInst, 3> {
 };
 
 InvokeInst::InvokeInst(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
                        BasicBlock *IfException, ArrayRef<Value *> Args,
-                       unsigned Values, const Twine &NameStr,
-                       Instruction *InsertBefore)
+                       ArrayRef<OperandBundleSetDef> Bundles, unsigned Values,
+                       const Twine &NameStr, Instruction *InsertBefore)
     : TerminatorInst(Ty->getReturnType(), Instruction::Invoke,
                      OperandTraits<InvokeInst>::op_end(this) - Values, Values,
                      InsertBefore) {
-  init(Ty, Func, IfNormal, IfException, Args, NameStr);
+  init(Ty, Func, IfNormal, IfException, Args, Bundles, NameStr);
 }
-InvokeInst::InvokeInst(Value *Func,
-                       BasicBlock *IfNormal, BasicBlock *IfException,
-                       ArrayRef<Value *> Args, unsigned Values,
+InvokeInst::InvokeInst(Value *Func, BasicBlock *IfNormal,
+                       BasicBlock *IfException, ArrayRef<Value *> Args,
+                       ArrayRef<OperandBundleSetDef> Bundles, unsigned Values,
                        const Twine &NameStr, BasicBlock *InsertAtEnd)
-  : TerminatorInst(cast<FunctionType>(cast<PointerType>(Func->getType())
-                                      ->getElementType())->getReturnType(),
-                   Instruction::Invoke,
-                   OperandTraits<InvokeInst>::op_end(this) - Values,
-                   Values, InsertAtEnd) {
-  init(Func, IfNormal, IfException, Args, NameStr);
+    : TerminatorInst(
+          cast<FunctionType>(cast<PointerType>(Func->getType())
+          Instruction::Invoke, OperandTraits<InvokeInst>::op_end(this) - Values,
+  init(Func, IfNormal, IfException, Args, Bundles, NameStr);
 }
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(InvokeInst, Value)
 
 //===----------------------------------------------------------------------===//
 //                              ResumeInst Class
 //===----------------------------------------------------------------------===//
 

lib/IR/Instructions.cpp

 //===----------------------------------------------------------------------===//
 //                        CallInst Implementation
 //===----------------------------------------------------------------------===//
 
 CallInst::~CallInst() {
 }
 
 void CallInst::init(FunctionType *FTy, Value *Func, ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleSetDef> Bundles,
                     const Twine &NameStr) {
   this->FTy = FTy;
-  assert(getNumOperands() == Args.size() + 1 && "NumOperands not set up?");
+  assert(getNumOperands() == Args.size() + CountBundleInputs(Bundles) + 1 &&
+         "NumOperands not set up?");
   Op<-1>() = Func;
 
 #ifndef NDEBUG
   assert((Args.size() == FTy->getNumParams() ||
           (FTy->isVarArg() && Args.size() > FTy->getNumParams())) &&
          "Calling a function with bad signature!");
 
   for (unsigned i = 0; i != Args.size(); ++i)
     assert((i >= FTy->getNumParams() || 
             FTy->getParamType(i) == Args[i]->getType()) &&
            "Calling a function with a bad signature!");
 #endif
 
   std::copy(Args.begin(), Args.end(), op_begin());
+
+  auto It = populateBundleOperandInfos(Bundles, Args.size());
+  (void)It;
+  assert(It + 1 == op_end() && "Should add up!");
+
   setName(NameStr);
 }
 
 void CallInst::init(Value *Func, const Twine &NameStr) {
   FTy =
       cast<FunctionType>(cast<PointerType>(Func->getType())->getElementType());
   assert(getNumOperands() == 1 && "NumOperands not set up?");
   Op<-1>() = Func;

 
 CallInst::CallInst(const CallInst &CI)
     : Instruction(CI.getType(), Instruction::Call,
                   OperandTraits<CallInst>::op_end(this) - CI.getNumOperands(),
                   CI.getNumOperands()),
       AttributeList(CI.AttributeList), FTy(CI.FTy) {
   setTailCallKind(CI.getTailCallKind());
   setCallingConv(CI.getCallingConv());
-    
   std::copy(CI.op_begin(), CI.op_end(), op_begin());
+  std::copy(CI.bundle_op_info_begin(), CI.bundle_op_info_end(),
+            bundle_op_info_begin());
   SubclassOptionalData = CI.SubclassOptionalData;
 }
 
 void CallInst::addAttribute(unsigned i, Attribute::AttrKind attr) {
   AttributeSet PAL = getAttributes();
   PAL = PAL.addAttribute(getContext(), i, attr);
   setAttributes(PAL);
 }

 }
 
 //===----------------------------------------------------------------------===//
 //                        InvokeInst Implementation
 //===----------------------------------------------------------------------===//
 
 void InvokeInst::init(FunctionType *FTy, Value *Fn, BasicBlock *IfNormal,
                       BasicBlock *IfException, ArrayRef<Value *> Args,
+                      ArrayRef<OperandBundleSetDef> Bundles,
                       const Twine &NameStr) {
   this->FTy = FTy;
 
-  assert(getNumOperands() == 3 + Args.size() && "NumOperands not set up?");
+  assert(getNumOperands() == 3 + Args.size() + CountBundleInputs(Bundles) &&
+         "NumOperands not set up?");
   Op<-3>() = Fn;
   Op<-2>() = IfNormal;
   Op<-1>() = IfException;
 
 #ifndef NDEBUG
   assert(((Args.size() == FTy->getNumParams()) ||
           (FTy->isVarArg() && Args.size() > FTy->getNumParams())) &&
          "Invoking a function with bad signature");
 
   for (unsigned i = 0, e = Args.size(); i != e; i++)
     assert((i >= FTy->getNumParams() || 
             FTy->getParamType(i) == Args[i]->getType()) &&
            "Invoking a function with a bad signature!");
 #endif
 
   std::copy(Args.begin(), Args.end(), op_begin());
+
+  auto It = populateBundleOperandInfos(Bundles, Args.size());
+  (void)It;
+  assert(It + 3 == op_end() && "Should add up!");
+
   setName(NameStr);
 }
 
 InvokeInst::InvokeInst(const InvokeInst &II)
     : TerminatorInst(II.getType(), Instruction::Invoke,
                      OperandTraits<InvokeInst>::op_end(this) -
                          II.getNumOperands(),
                      II.getNumOperands()),
       AttributeList(II.AttributeList), FTy(II.FTy) {
   setCallingConv(II.getCallingConv());
   std::copy(II.op_begin(), II.op_end(), op_begin());
+  std::copy(II.bundle_op_info_begin(), II.bundle_op_info_end(),
+            bundle_op_info_begin());
   SubclassOptionalData = II.SubclassOptionalData;
 }
 
 BasicBlock *InvokeInst::getSuccessorV(unsigned idx) const {
   return getSuccessor(idx);
 }
 unsigned InvokeInst::getNumSuccessorsV() const {
   return getNumSuccessors();

lib/IR/LLVMContextImpl.h

#include "llvm/IR/Metadata.def"
   /// the operand of an unparented ReturnInst so that the prologue data has a
   /// Use.
   typedef DenseMap<const Function *, ReturnInst *> PrologueDataMapTy;
   PrologueDataMapTy PrologueDataMap;
 
   int getOrAddScopeRecordIdxEntry(MDNode *N, int ExistingIdx);
   int getOrAddScopeInlinedAtIdxEntry(MDNode *Scope, MDNode *IA,int ExistingIdx);
 
+  /// \brief A set of interned tags for operand bundles.  It currently maps all
+  /// tags to 0, but in the future we may consider using the uint32_t to
+  /// refcount the interned strings.
+  StringMap<uint32_t> BundleTagCache;
+  StringMapEntry<uint32_t> *getOrInsertBundleTag(StringRef Tag);
+
   LLVMContextImpl(LLVMContext &C);
   ~LLVMContextImpl();
 
   /// Destroy the ConstantArrays if they are not used.
   void dropTriviallyDeadConstantArrays();
 };
 
 }
 
 #endif

lib/IR/LLVMContextImpl.cpp

 #endif
   return Hash;
 }
 
 unsigned MDNodeOpsKey::calculateHash(ArrayRef<Metadata *> Ops) {
   return hash_combine_range(Ops.begin(), Ops.end());
 }
 
+StringMapEntry<uint32_t> *LLVMContextImpl::getOrInsertBundleTag(StringRef Tag) {
+  return &*(BundleTagCache.insert(std::make_pair(Tag, (uint32_t)0)).first);
+}
+
 // ConstantsContext anchors
 void UnaryConstantExpr::anchor() { }
 
 void BinaryConstantExpr::anchor() { }
 
 void SelectConstantExpr::anchor() { }
 
 void ExtractElementConstantExpr::anchor() { }