[VP] Introducing VectorBuilder, the VP intrinsic builder

[simoll]

Summary

VectorBuilder wraps around an IRBuilder and VectorBuilder::createVectorInstructions emits VP intrinsics as if they were regular instructions.

[simoll]

• Cleanup
• Better test coverage (paths where no evl and/or no mask set).

[craig.topper]

llvm/lib/IR/VectorBuilder.cpp

31

Doesn't BasicBlock have getModule?

[simoll]

BasicBlock::getModule

[craig.topper]

llvm/include/llvm/IR/VectorBuilder.h

93

Aren't Twine's usually passed by const reference?

[simoll]

• const Twine&
• Rebased

[frasercrmck]

LGTM though we should probably add extra tests for vp.select and vp.reduce.* now that they're in.

llvm/lib/IR/VectorBuilder.cpp

68

I'm thinking .hasValue() is a bit clearer than (bool)

70

don't need the parentheses around the equality comparison here or below.

[craig.topper]

llvm/include/llvm/IR/VectorBuilder.h

27

Do we really need this typedef? Mainly asking because it went into the llvm namespace.

29

Can we move this into the class so it becomes VectorBuilder::Behavior?

84

How do you set StaticVectorLength to a scalable value?

llvm/lib/IR/VectorBuilder.cpp

76

Does this assume the Mask and EVL are at the end? If they aren't then ParamIdx isn't the correct index here if the Mask and EVL were earlier. Assuming there aren't empty slots for them in InstOpArray.

If they are always at the end and we're going to assume, that then there's not a lot of point for handling them inside the loop. You can just copy the InstOps, then add mask and evl if they exist.

llvm/unittests/IR/VectorBuilderTest.cpp

22

nit: int->unsigned. That's at least what most of the interfaces this gets passed to use.

32

Use std::make_unique instead of creating a unique_ptr for the return.

[simoll]

Added fast path for trailing mask, evl in parameter list

[simoll]

vp.merge and vp.select do not ignore masked-off lanes. We probably shouldn't treat their mask parameters as "masks" in the VP sense. I suggest we stop reporting their MaskPos in VPintrinsics.def.

llvm/include/llvm/IR/VectorBuilder.h

84

Scalable types aren't supported in this first patch. Planned for a followup.

llvm/lib/IR/VectorBuilder.cpp

76

There is an issue with ParamIdx at the moment. If the loop is past the first mask or evl parameter the ParamIdx will be off by one for the other one. It is okay for the other parameters though (note that there is no else in front of the last if).

[simoll]

Rebased onto vp.merge|select-mask-is-not-a-vp-mask patch.

[simoll]

Where are we with this patch?

[craig.topper]

llvm/lib/IR/VectorBuilder.cpp

86

This loop doesn't work if VL is before mask and they are adjacent to each other. VL will increment VPParamIdx, but don't re-check mask after adding VL.

I think the for statement should be reponsible for incrementing VPParamIdx only. And ParamIdx should only be incremented in the loop body then a parameter from InstOpsArray is pushed. There should be no increments of VPParamIdx in the loop body.

[simoll]

Improved parameter list construction following @craig.topper 's suggestion. Also, fast path (both mask and evl at tail) improved to allow for any ordering of the two.

[craig.topper]

LGTM you can address the comments if you want, but I don't want to hold this up anymore.

llvm/lib/IR/VectorBuilder.cpp

81

What if we just called IntrinParams.resize(NumVPParams); and then assigned the mask and vl into their slots?

llvm/unittests/IR/VectorBuilderTest.cpp

131

Can simplify a little by merging the isa and cast to a single dyn_cast.

static bool isLegalConstEVL(Value *Val, unsigned ExpectedEVL) {
  auto *ConstEVL = dyn_cast<ConstantInt>(Val);
  if (!ConstEVL)
    return false;

  // Value check.
  
  if (ConstEVL->getZExtValue() != ExpectedEVL)
    return false;

  // Type check.
  return ConstEVL->getType()->isIntegerTy(32);
}

Or go really packed and do.

static bool isLegalConstEVL(Value *Val, unsigned ExpectedEVL) {
  auto *ConstEVL = dyn_cast<ConstantInt>(Val);
  return ConstEVL && ConstEVL->getZExtValue() == ExpectedEVL && ConstEVL->getType()->isIntegerTy(32);
}

[simoll]

llvm/lib/IR/VectorBuilder.cpp

81

I like it

llvm/unittests/IR/VectorBuilderTest.cpp

131

The first one. This isn't code golf, sir ;-)

[simoll]

Cleaner code for the parameter list construction

[simoll]

Breaking the clang-cmake-x86_64-avx2-linux bot: https://lab.llvm.org/buildbot/#/builders/110/builds/10912
I suspect Behavior Behavior = Behavior::[..] doesn't go down well with that version of gcc. Waiting to find out which version of gcc is used there to reproduce.

[steven_wu]

Just a note, this also breaks implicit module build as the newly added header indirectly includes llvm/IR/Attributes.inc which needs to be generated first. If you do a module build (-DLLVM_ENABLE_MODULES=ON), you are likely to get error:

09:05:38 While building module 'LLVM_MC' imported from /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/lib/MC/MCAsmBackend.cpp:9:
09:05:38 While building module 'LLVM_IR' imported from /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/include/llvm/MC/MCPseudoProbe.h:49:
09:05:38 While building module 'LLVM_intrinsic_gen' imported from /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/include/llvm/IR/VectorBuilder.h:18:
09:05:38 In file included from <module-includes>:1:
09:05:38 In file included from /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/include/llvm/IR/Argument.h:17:
09:05:38 /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/include/llvm/IR/Attributes.h:75:14: fatal error: 'llvm/IR/Attributes.inc' file not found
09:05:38     #include "llvm/IR/Attributes.inc"
09:05:38              ^~~~~~~~~~~~~~~~~~~~~~~~
09:05:38 While building module 'LLVM_MC' imported from /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/lib/MC/MCAsmBackend.cpp:9:
09:05:38 While building module 'LLVM_IR' imported from /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/include/llvm/MC/MCPseudoProbe.h:49:
09:05:38 In file included from <module-includes>:62:
09:05:38 /System/Volumes/Data/jenkins/workspace/apple-clang-stage2-configure-RA_master/llvm-project/llvm/include/llvm/IR/VectorBuilder.h:18:10: fatal error: could not build module 'LLVM_intrinsic_gen'
09:05:38 #include <llvm/IR/IRBuilder.h>
09:05:38  ~~~~~~~~^

You probably need to do something in the module map to fix this (see: module LLVM_intrinsic_gen in llvm/include/llvm/module.modulemap )

[craig.topper]

llvm/include/llvm/IR/VectorBuilder.h

22

Is anything from PatternMatch being used?

[simoll]

llvm/include/llvm/IR/VectorBuilder.h

22

Definitely not in this header, no

[simoll]

• Add module map entry
• Rename Behavior variable (this may be confusing gcc 5.4; not confirmed)
• Remove PatternMatch include

llvm/include/llvm/IR/VectorBuilder.h

+//===- llvm/VectorBuilder.h - Builder for VP Intrinsics ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the VectorBuilder class, which is used as a convenient way
+// to create VP intrinsics as if they were LLVM instructions with a consistent
+// and simplified interface.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_IR_VECTORBUILDER_H
+#define LLVM_IR_VECTORBUILDER_H
+
+#include <llvm/IR/IRBuilder.h>
+#include <llvm/IR/InstrTypes.h>
+#include <llvm/IR/Instruction.h>
+#include <llvm/IR/PatternMatch.h>
+#include <llvm/IR/Value.h>
+
+namespace llvm {
+
+class VectorBuilder {
+public:
+  enum class Behavior {
+    // Abort if the requested VP intrinsic could not be created.
+    // This is useful for strict consistency.
+    ReportAndAbort = 0,
+
+    // Return a default-initialized value if the requested VP intrinsic could
+    // not be created.
+    // This is useful for a defensive fallback to non-VP code.
+    SilentlyReturnNone = 1,
+  };
+
+private:
+  IRBuilder<> &Builder;
+  Behavior Behavior;
+
+  // Explicit mask parameter.
+  Value *Mask;
+  // Explicit vector length parameter.
+  Value *ExplicitVectorLength;
+  // Compile-time vector length.
+  ElementCount StaticVectorLength;
+
+  // Get mask/evl value handles for the current configuration.
+  Value &requestMask();
+  Value &requestEVL();
+
+  void handleError(const char *ErrorMsg) const;
+  template <typename RetType>
+  RetType returnWithError(const char *ErrorMsg) const {
+    handleError(ErrorMsg);
+    return RetType();
+  }
+
+public:
+  VectorBuilder(IRBuilder<> &Builder,
+                enum Behavior Behavior = Behavior::ReportAndAbort)
+      : Builder(Builder), Behavior(Behavior), Mask(nullptr),
+        ExplicitVectorLength(nullptr),
+        StaticVectorLength(ElementCount::getFixed(0)) {}
+
+  Module &getModule() const;
+  LLVMContext &getContext() const { return Builder.getContext(); }
+
+  // All-true mask for the currently configured explicit vector length.
+  Value *getAllTrueMask();
+
+  VectorBuilder &setMask(Value *NewMask) {
+    Mask = NewMask;
+    return *this;
+  }
+  VectorBuilder &setEVL(Value *NewExplicitVectorLength) {
+    ExplicitVectorLength = NewExplicitVectorLength;
+    return *this;
+  }
+  VectorBuilder &setStaticVL(unsigned NewFixedVL) {
+    StaticVectorLength = ElementCount::getFixed(NewFixedVL);
+    return *this;
+  }
+  // TODO: setStaticVL(ElementCount) for scalable types.
+
+  // Emit a VP intrinsic call that mimics a regular instruction.
+  // This operation behaves according to the VectorBuilderBehavior.
+  // \p Opcode      The functional instruction opcode of the emitted intrinsic.
+  // \p ReturnTy    The return type of the operation.
+  // \p VecOpArray  The operand list.
+  Value *createVectorInstruction(unsigned Opcode, Type *ReturnTy,
+                                 ArrayRef<Value *> VecOpArray,
+                                 const Twine &Name = Twine());
+};
+
+} // namespace llvm
+
+#endif // LLVM_IR_VECTORBUILDER_H

llvm/lib/IR/CMakeLists.txt

add_llvm_component_library(LLVMCore
   Statepoint.cpp
   StructuralHash.cpp
   Type.cpp
   TypeFinder.cpp
   Use.cpp
   User.cpp
   Value.cpp
   ValueSymbolTable.cpp
+  VectorBuilder.cpp
   Verifier.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/IR
 
   LINK_LIBS
   ${LLVM_PTHREAD_LIB}
 
   DEPENDS
   intrinsics_gen
 
   LINK_COMPONENTS
   BinaryFormat
   Remarks
   Support
   )

llvm/lib/IR/VectorBuilder.cpp

+//===- VectorBuilder.cpp - Builder for VP Intrinsics ----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the VectorBuilder class, which is used as a convenient
+// way to create VP intrinsics as if they were LLVM instructions with a
+// consistent and simplified interface.
+//
+//===----------------------------------------------------------------------===//
+
+#include <llvm/ADT/SmallVector.h>
+#include <llvm/IR/FPEnv.h>
+#include <llvm/IR/Instructions.h>
+#include <llvm/IR/Intrinsics.h>
+#include <llvm/IR/VectorBuilder.h>
+
+namespace llvm {
+
+void VectorBuilder::handleError(const char *ErrorMsg) const {
+  if (Behavior == Behavior::SilentlyReturnNone)
+    return;
+  report_fatal_error(ErrorMsg);
+}
+
+Module &VectorBuilder::getModule() const {
+  return *Builder.GetInsertBlock()->getModule();
+}
+
+Value *VectorBuilder::getAllTrueMask() {
+  auto *BoolTy = Builder.getInt1Ty();
+  auto *MaskTy = VectorType::get(BoolTy, StaticVectorLength);
+  return ConstantInt::getAllOnesValue(MaskTy);
+}
+
+Value &VectorBuilder::requestMask() {
+  if (Mask)
+    return *Mask;
+
+  return *getAllTrueMask();
+}
+
+Value &VectorBuilder::requestEVL() {
+  if (ExplicitVectorLength)
+    return *ExplicitVectorLength;
+
+  assert(!StaticVectorLength.isScalable() && "TODO vscale lowering");
+  auto *IntTy = Builder.getInt32Ty();
+  return *ConstantInt::get(IntTy, StaticVectorLength.getFixedValue());
+}
+
+Value *VectorBuilder::createVectorInstruction(unsigned Opcode, Type *ReturnTy,
+                                              ArrayRef<Value *> InstOpArray,
+                                              const Twine &Name) {
+  auto VPID = VPIntrinsic::getForOpcode(Opcode);
+  if (VPID == Intrinsic::not_intrinsic)
+    return returnWithError<Value *>("No VPIntrinsic for this opcode");
+
+  auto MaskPosOpt = VPIntrinsic::getMaskParamPos(VPID);
+  auto VLenPosOpt = VPIntrinsic::getVectorLengthParamPos(VPID);
+  size_t NumInstParams = InstOpArray.size();
+  size_t NumVPParams =
+      NumInstParams + MaskPosOpt.hasValue() + VLenPosOpt.hasValue();
+
+  SmallVector<Value *, 6> IntrinParams;
+
+  // Whether the mask and vlen parameter are at the end of the parameter list.
+  bool TrailingMaskAndVLen =
+      std::min<size_t>(MaskPosOpt.getValueOr(NumInstParams),
+                       VLenPosOpt.getValueOr(NumInstParams)) >= NumInstParams;
+
+  if (TrailingMaskAndVLen) {
+    // Fast path for trailing mask, then vector length.
+    assert(MaskPosOpt.getValueOr(0) <= VLenPosOpt.getValueOr(NumVPParams));
+
+    IntrinParams.append(InstOpArray.begin(), InstOpArray.end());
+    if (MaskPosOpt.hasValue())
+      IntrinParams.push_back(&requestMask());
+    if (VLenPosOpt.hasValue())
+      IntrinParams.push_back(&requestEVL());
+  } else {
+    // Insert mask and evl operands in between the instruction operands.
+    for (size_t ParamIdx = 0, VPParamIdx = 0; VPParamIdx < NumVPParams;
+         ++ParamIdx, ++VPParamIdx) {
+      if (MaskPosOpt && MaskPosOpt.getValueOr(NumVPParams) == VPParamIdx) {
+        IntrinParams.push_back(&requestMask());
+        ++VPParamIdx;
+      }
+      if (VLenPosOpt && VLenPosOpt.getValueOr(NumVPParams) == VPParamIdx) {
+        IntrinParams.push_back(&requestEVL());
+        ++VPParamIdx;
+      }
+      if (ParamIdx < NumInstParams)
+        IntrinParams.push_back(InstOpArray[ParamIdx]);
+    }
+  }
+
+  auto *VPDecl = VPIntrinsic::getDeclarationForParams(&getModule(), VPID,
+                                                      ReturnTy, IntrinParams);
+  return Builder.CreateCall(VPDecl, IntrinParams, Name);
+}
+
+} // namespace llvm

llvm/unittests/IR/CMakeLists.txt

add_llvm_unittest(IRTests
   PatternMatch.cpp
   TimePassesTest.cpp
   TypesTest.cpp
   UseTest.cpp
   UserTest.cpp
   ValueHandleTest.cpp
   ValueMapTest.cpp
   ValueTest.cpp
+  VectorBuilderTest.cpp
   VectorTypesTest.cpp
   VerifierTest.cpp
   VPIntrinsicTest.cpp
   )
 
 target_link_libraries(IRTests PRIVATE LLVMTestingSupport)

llvm/unittests/IR/VectorBuilderTest.cpp

+//===--------- VectorBuilderTest.cpp - VectorBuilder unit tests -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/VectorBuilder.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/IR/Module.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+
+namespace {
+
+static unsigned VectorNumElements = 8;
+
+class VectorBuilderTest : public testing::Test {
+protected:
+  LLVMContext Context;
+
+  VectorBuilderTest() : Context() {}
+
+  std::unique_ptr<Module> createBuilderModule(Function *&Func, BasicBlock *&BB,
+                                              Value *&Mask, Value *&EVL) {
+    auto Mod = std::make_unique<Module>("TestModule", Context);
+    auto *Int32Ty = Type::getInt32Ty(Context);
+    auto *Mask8Ty =
+        FixedVectorType::get(Type::getInt1Ty(Context), VectorNumElements);
+    auto *VoidFuncTy =
+        FunctionType::get(Type::getVoidTy(Context), {Mask8Ty, Int32Ty}, false);
+    Func =
+        Function::Create(VoidFuncTy, GlobalValue::ExternalLinkage, "bla", *Mod);
+    Mask = Func->getArg(0);
+    EVL = Func->getArg(1);
+    BB = BasicBlock::Create(Context, "entry", Func);
+
+    return Mod;
+  }
+};
+
+/// Check that creating binary arithmetic VP intrinsics works.
+TEST_F(VectorBuilderTest, TestCreateBinaryInstructions) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  VectorBuilder VBuild(Builder);
+  VBuild.setMask(Mask).setEVL(EVL);
+
+  auto *FloatVecTy =
+      FixedVectorType::get(Type::getFloatTy(Context), VectorNumElements);
+  auto *IntVecTy =
+      FixedVectorType::get(Type::getInt32Ty(Context), VectorNumElements);
+
+#define HANDLE_BINARY_INST(NUM, OPCODE, INSTCLASS)                             \
+  {                                                                            \
+    auto VPID = VPIntrinsic::getForOpcode(Instruction::OPCODE);                \
+    bool IsFP = (#INSTCLASS)[0] == 'F';                                        \
+    auto *ValueTy = IsFP ? FloatVecTy : IntVecTy;                              \
+    Value *Op = UndefValue::get(ValueTy);                                      \
+    auto *I = VBuild.createVectorInstruction(Instruction::OPCODE, ValueTy,     \
+                                             {Op, Op});                        \
+    ASSERT_TRUE(isa<VPIntrinsic>(I));                                          \
+    auto *VPIntrin = cast<VPIntrinsic>(I);                                     \
+    ASSERT_EQ(VPIntrin->getIntrinsicID(), VPID);                               \
+    ASSERT_EQ(VPIntrin->getMaskParam(), Mask);                                 \
+    ASSERT_EQ(VPIntrin->getVectorLengthParam(), EVL);                          \
+  }
+#include "llvm/IR/Instruction.def"
+}
+
+static bool isAllTrueMask(Value *Val, unsigned NumElements) {
+  auto *ConstMask = dyn_cast<Constant>(Val);
+  if (!ConstMask)
+    return false;
+
+  // Structure check.
+  if (!ConstMask->isAllOnesValue())
+    return false;
+
+  // Type check.
+  auto *MaskVecTy = cast<FixedVectorType>(ConstMask->getType());
+  if (MaskVecTy->getNumElements() != NumElements)
+    return false;
+
+  return MaskVecTy->getElementType()->isIntegerTy(1);
+}
+
+/// Check that creating binary arithmetic VP intrinsics works.
+TEST_F(VectorBuilderTest, TestCreateBinaryInstructions_FixedVector_NoMask) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  VectorBuilder VBuild(Builder);
+  VBuild.setEVL(EVL).setStaticVL(VectorNumElements);
+
+  auto *FloatVecTy =
+      FixedVectorType::get(Type::getFloatTy(Context), VectorNumElements);
+  auto *IntVecTy =
+      FixedVectorType::get(Type::getInt32Ty(Context), VectorNumElements);
+
+#define HANDLE_BINARY_INST(NUM, OPCODE, INSTCLASS)                             \
+  {                                                                            \
+    auto VPID = VPIntrinsic::getForOpcode(Instruction::OPCODE);                \
+    bool IsFP = (#INSTCLASS)[0] == 'F';                                        \
+    Type *ValueTy = IsFP ? FloatVecTy : IntVecTy;                              \
+    Value *Op = UndefValue::get(ValueTy);                                      \
+    auto *I = VBuild.createVectorInstruction(Instruction::OPCODE, ValueTy,     \
+                                             {Op, Op});                        \
+    ASSERT_TRUE(isa<VPIntrinsic>(I));                                          \
+    auto *VPIntrin = cast<VPIntrinsic>(I);                                     \
+    ASSERT_EQ(VPIntrin->getIntrinsicID(), VPID);                               \
+    ASSERT_TRUE(isAllTrueMask(VPIntrin->getMaskParam(), VectorNumElements));   \
+    ASSERT_EQ(VPIntrin->getVectorLengthParam(), EVL);                          \
+  }
+#include "llvm/IR/Instruction.def"
+}
+
+static bool isLegalConstEVL(Value *Val, unsigned ExpectedEVL) {
+  if (!isa<ConstantInt>(Val))
+    return false;
+
+  // Value check.
+  auto *ConstEVL = cast<ConstantInt>(Val);
+  if (ConstEVL->getZExtValue() != ExpectedEVL)
+    return false;
+
+  // Type check.
+  return ConstEVL->getType()->isIntegerTy(32);
+}
+
+/// Check that creating binary arithmetic VP intrinsics works.
+TEST_F(VectorBuilderTest, TestCreateBinaryInstructions_FixedVector_NoEVL) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  VectorBuilder VBuild(Builder);
+  VBuild.setMask(Mask).setStaticVL(VectorNumElements);
+
+  auto *FloatVecTy =
+      FixedVectorType::get(Type::getFloatTy(Context), VectorNumElements);
+  auto *IntVecTy =
+      FixedVectorType::get(Type::getInt32Ty(Context), VectorNumElements);
+
+#define HANDLE_BINARY_INST(NUM, OPCODE, INSTCLASS)                             \
+  {                                                                            \
+    auto VPID = VPIntrinsic::getForOpcode(Instruction::OPCODE);                \
+    bool IsFP = (#INSTCLASS)[0] == 'F';                                        \
+    Type *ValueTy = IsFP ? FloatVecTy : IntVecTy;                              \
+    Value *Op = UndefValue::get(ValueTy);                                      \
+    auto *I = VBuild.createVectorInstruction(Instruction::OPCODE, ValueTy,     \
+                                             {Op, Op});                        \
+    ASSERT_TRUE(isa<VPIntrinsic>(I));                                          \
+    auto *VPIntrin = cast<VPIntrinsic>(I);                                     \
+    ASSERT_EQ(VPIntrin->getIntrinsicID(), VPID);                               \
+    ASSERT_EQ(VPIntrin->getMaskParam(), Mask);                                 \
+    ASSERT_TRUE(                                                               \
+        isLegalConstEVL(VPIntrin->getVectorLengthParam(), VectorNumElements)); \
+  }
+#include "llvm/IR/Instruction.def"
+}
+
+/// Check that creating binary arithmetic VP intrinsics works.
+TEST_F(VectorBuilderTest,
+       TestCreateBinaryInstructions_FixedVector_NoMask_NoEVL) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  VectorBuilder VBuild(Builder);
+  VBuild.setStaticVL(VectorNumElements);
+
+  auto *FloatVecTy =
+      FixedVectorType::get(Type::getFloatTy(Context), VectorNumElements);
+  auto *IntVecTy =
+      FixedVectorType::get(Type::getInt32Ty(Context), VectorNumElements);
+
+#define HANDLE_BINARY_INST(NUM, OPCODE, INSTCLASS)                             \
+  {                                                                            \
+    auto VPID = VPIntrinsic::getForOpcode(Instruction::OPCODE);                \
+    bool IsFP = (#INSTCLASS)[0] == 'F';                                        \
+    Type *ValueTy = IsFP ? FloatVecTy : IntVecTy;                              \
+    Value *Op = UndefValue::get(ValueTy);                                      \
+    auto *I = VBuild.createVectorInstruction(Instruction::OPCODE, ValueTy,     \
+                                             {Op, Op});                        \
+    ASSERT_TRUE(isa<VPIntrinsic>(I));                                          \
+    auto *VPIntrin = cast<VPIntrinsic>(I);                                     \
+    ASSERT_EQ(VPIntrin->getIntrinsicID(), VPID);                               \
+    ASSERT_TRUE(isAllTrueMask(VPIntrin->getMaskParam(), VectorNumElements));   \
+    ASSERT_TRUE(                                                               \
+        isLegalConstEVL(VPIntrin->getVectorLengthParam(), VectorNumElements)); \
+  }
+#include "llvm/IR/Instruction.def"
+}
+/// Check that creating vp.load/vp.store works.
+TEST_F(VectorBuilderTest, TestCreateLoadStore) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  VectorBuilder VBuild(Builder);
+  VBuild.setMask(Mask).setEVL(EVL);
+
+  auto *FloatVecTy =
+      FixedVectorType::get(Type::getFloatTy(Context), VectorNumElements);
+  auto *FloatVecPtrTy = FloatVecTy->getPointerTo();
+
+  Value *FloatVecPtr = UndefValue::get(FloatVecPtrTy);
+  Value *FloatVec = UndefValue::get(FloatVecTy);
+
+  // vp.load
+  auto LoadVPID = VPIntrinsic::getForOpcode(Instruction::Load);
+  auto *LoadIntrin = VBuild.createVectorInstruction(Instruction::Load,
+                                                    FloatVecTy, {FloatVecPtr});
+  ASSERT_TRUE(isa<VPIntrinsic>(LoadIntrin));
+  auto *VPLoad = cast<VPIntrinsic>(LoadIntrin);
+  ASSERT_EQ(VPLoad->getIntrinsicID(), LoadVPID);
+  ASSERT_EQ(VPLoad->getMemoryPointerParam(), FloatVecPtr);
+
+  // vp.store
+  auto *VoidTy = Builder.getVoidTy();
+  auto StoreVPID = VPIntrinsic::getForOpcode(Instruction::Store);
+  auto *StoreIntrin = VBuild.createVectorInstruction(Instruction::Store, VoidTy,
+                                                     {FloatVec, FloatVecPtr});
+  ASSERT_TRUE(isa<VPIntrinsic>(LoadIntrin));
+  auto *VPStore = cast<VPIntrinsic>(StoreIntrin);
+  ASSERT_EQ(VPStore->getIntrinsicID(), StoreVPID);
+  ASSERT_EQ(VPStore->getMemoryPointerParam(), FloatVecPtr);
+  ASSERT_EQ(VPStore->getMemoryDataParam(), FloatVec);
+}
+
+/// Check that the SilentlyReturnNone error handling mode works.
+TEST_F(VectorBuilderTest, TestFail_SilentlyReturnNone) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  auto *VoidTy = Builder.getVoidTy();
+  VectorBuilder VBuild(Builder, VectorBuilder::Behavior::SilentlyReturnNone);
+  VBuild.setMask(Mask).setEVL(EVL);
+  auto *Val = VBuild.createVectorInstruction(Instruction::Br, VoidTy, {});
+  ASSERT_EQ(Val, nullptr);
+}
+
+/// Check that the ReportAndFail error handling mode aborts as advertised.
+TEST_F(VectorBuilderTest, TestFail_ReportAndAbort) {
+  Function *F;
+  BasicBlock *BB;
+  Value *Mask, *EVL;
+  auto Mod = createBuilderModule(F, BB, Mask, EVL);
+
+  IRBuilder<> Builder(BB);
+  auto *VoidTy = Builder.getVoidTy();
+  VectorBuilder VBuild(Builder, VectorBuilder::Behavior::ReportAndAbort);
+  VBuild.setMask(Mask).setEVL(EVL);
+  ASSERT_DEATH({ VBuild.createVectorInstruction(Instruction::Br, VoidTy, {}); },
+               "No VPIntrinsic for this opcode");
+}
+
+} // end anonymous namespace