[X86] Add a pass to convert instruction chains between domains

[guyblank]

Summary

This pass attempts to find instruction chains in one domain and convert them to equivalent instructions in another domain, if it is found profitable.

The patch adds support only for GPR->Mask conversions.

[craig.topper]

Some initial comments. I need to look some more.

lib/Target/X86/X86DomainReassignment.cpp

67

class*

79

Prefer llvm_unreachable over assert(false)

118

Could we just pass TII to the methods that use it? If one of the virtual implementation doesn't need it they can ignore it. That way we don't have to store it in the classes?

154

Same question as above, can we pass TII and MRI to the virtual methods?

299

I'm not sure static state in a pass is a good practice. If we ever tried to multithread it would be a problem.

475

Should Worklist be a stack variable here that you can pass by reference to addToWorklist? Does it have lifetime beyond building the closure?

579

Isn't KMOVB and all other 'B' instructions part of the DQI feature set?

[delena]

lib/Target/X86/X86DomainReassignment.cpp

202

The COPY has only 2 operands, right?
Why do you need so complex computations here?

474

Please replace E with a meaningful variable name.

495

I think that you can iterate to MemOp. After the MemOp may be only immediate.

[zvi]

Please add tests that demonstrate cross-BB transformations.

[zvi]

Would it be beneficial to add .mir tests?

[zvi]

lib/Target/X86/X86DomainReassignment.cpp

44

Would these functions fit better in X86RegisterInfo?

50

Can this check be made cheaper and more maintainable by defining a super class for all K reg class and check for membership in the superclass?

246

How does the delete happen? Are we relying on MachneDCE?

318

*for

323

STLExtras.h provides a more convenient variant of find:

auto I = find(LegalDstDomains, RD);

333

return is_contained(LegalDstDomains, RD)

375

If the class declaration and definition and in the same unit, make the definition inline in class?

381

This methods does more than possibly adding to the worklist. Maybe visitRegister would be a better name?

404

EnclosedInstrs.find(MI) -> I

453

Should this be moved out to a common location?

546

Please add .mir tests for each replacer

675

0.0

lib/Target/X86/X86TargetMachine.cpp

412

identation

[guyblank]

added a test (gpr-to-mask.ll) to show some cross BB issues

[aaboud]

lib/Target/X86/X86DomainReassignment.cpp

56

Please, pick an order and stick to it. "RC == &X86::VK64RegClass" should be moved line 52.

97

Would it be better to rename this function as: "getExtraCost" or "getIncreasedCost"?

103

I think you do not need the "virtual" keyword here and in all places below. The "override" keyword is enough.

125

To make the comment more clear:
"unless it's dead." -> "unless that register is dead."

136

Are we sure we do not need to add any implicit operand?
in the "isLegal" function you allowed implicit operands that are not dead, when the DstOpcode implicitly define that physical register, so in such case we need to add that implicit operand to the new instruction, do not we?

160

Why do not we need to check that MI opcode and DstOpode have same operand types/semantic?
Otherwise, the loop in line 169 will fail!

197

Should not we return false if (MI->getOpcode() != TargetOpcode:COPY)?

253

Can you explain this KetTy? what are the "int" and the "unsigned" pair represents?
Please, add a comment.

306

How about passing the vector by reference? i.e., "&LegalDstDomins".

318

I think he meant "to all domains".

405

Please, add a comment explaining this case.

413

Please, add a comment explaining this behavior.

437

I do not understand this code, can you add a comment explaining the logic behind this code?

512

Does that make any sense to continue after we clear all Legal Dst Domains? Should not we just return?

523

Same question here, why do not we just return!?

648

If you clear these two containers for each function processing, then why they need to be static?
How about let this function creates them and pass them to all "Closure"s by reference? Can do that at line 665 below.

[craig.topper]

lib/Target/X86/X86DomainReassignment.cpp

495

There are definitely instructions that have a register after the memory op. For example vpblendvb

[guyblank]

Thank you all for the comments.

lib/Target/X86/X86DomainReassignment.cpp

44

i'm not sure, there are all the other register classes (GR_ABCD, and friends) which I did not include here. So putting this isGPR function, as is, in X86RegisterInfo would be a bit misleading I think.

136

BuildMI will add the implicit operands from the instruction description, which are the ones allowed to be not dead by "isLegal"

160

I think the responsibility for semantic equivalence is on users of this class (and all other converters).

But there is a hidden assumption here that the functions will be called only with the MI they were created with, when initializing the converters.
Does adding the source op code as a member of the converter and checking/asserting on it sound reasonable?

202

I'm not sure I understand why you say these are complex computations, can you suggest a simpler sequence?

246

No, this pass will delete the instruction.
ConvertInstr should returns true when the MI needs to be deleted.

453

can you suggest a location?
is x86InstrInfo suitable?

512

I don't think it matters much.
If we return here, we'll get to the other uses eventually, as part of another closure, and mark them as illegal (since they will try to add a register which is part of another closure).
So we might as well get to them now.

523

same answer as above.

546

WIP

579

ooops
Thanks!

[guyblank]

addressed comments

[guyblank]

Added mir test

[aaboud]

lib/Target/X86/X86DomainReassignment.cpp

136

Then please add a comment saying that implicit operands are added/handled by BuildMI, and we only need to add the explicit operands.

160

I think it is the best option we have.
Add the source opcode and make sure that isLegal will accept only MI with same source opcode, otherwise it will fail.
you have two options here:

1. actually return false if the MI opcode and the source opcode are not the same.
2. Only assert on that, but always return true. (this is acceptable, if you think that such case is an implementation issue, rather than user input issue).

In addition, you should add an assertion in the ConverInstr(), that the MI is legal:
assert(isLegal(MI) && "Illegal instruction!");

[guyblank]

Address comments by @aaboud and some other offline comments.

[guyblank]

ping

[zvi]

LGTM

[anna]

Hi

It looks like this pass has extreme compile time growth. On icelake machines (where we have AVX-512 enabled), we see llc taking about 18 seconds on an IR and this pass contributes about 17 seconds.

There were also prior reproducers/issues added here: https://github.com/llvm/llvm-project/issues/41517

lib/Target/X86/CMakeLists.txt

 
 add_public_tablegen_target(X86CommonTableGen)
 
 set(sources
   X86AsmPrinter.cpp
   X86CallFrameOptimization.cpp
   X86CallLowering.cpp
   X86CmovConversion.cpp
+  X86DomainReassignment.cpp
   X86ExpandPseudo.cpp
   X86FastISel.cpp
   X86FixupBWInsts.cpp
   X86FixupLEAs.cpp
   X86FixupSetCC.cpp
   X86FloatingPoint.cpp
   X86FrameLowering.cpp
   X86InstructionSelector.cpp

lib/Target/X86/X86.h

 FunctionPass *createX86CmovConverterPass();
 
 /// Return a Machine IR pass that selectively replaces
 /// certain byte and word instructions by equivalent 32 bit instructions,
 /// in order to eliminate partial register usage, false dependences on
 /// the upper portions of registers, and to save code size.
 FunctionPass *createX86FixupBWInsts();
 
+/// Return a Machine IR pass that reassigns instruction chains from one domain
+/// to another, when profitable.
+FunctionPass *createX86DomainReassignmentPass();
+
 void initializeFixupBWInstPassPass(PassRegistry &);
 
 /// This pass replaces EVEX encoded of AVX-512 instructiosn by VEX
 /// encoding when possible in order to reduce code size.
 FunctionPass *createX86EvexToVexInsts();
 
 InstructionSelector *createX86InstructionSelector(const X86TargetMachine &TM,
                                                   X86Subtarget &,
                                                   X86RegisterBankInfo &);
 
 void initializeEvexToVexInstPassPass(PassRegistry &);
 
 } // End llvm namespace
 
 #endif

lib/Target/X86/X86DomainReassignment.cpp

+//===--- X86DomainReassignment.cpp - Selectively switch register classes---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass attempts to find instruction chains (closures) in one domain,
+// and convert them to equivalent instructions in a different domain,
+// if profitable.
+//
+//===----------------------------------------------------------------------===//
+
+#include "X86.h"
+#include "X86InstrInfo.h"
+#include "X86Subtarget.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseMapInfo.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Target/TargetRegisterInfo.h"
+
+using namespace llvm;
+
+namespace llvm {
+void initializeX86DomainReassignmentPass(PassRegistry &);
+}
+
+#define DEBUG_TYPE "x86-domain-reassignment"
+
+STATISTIC(NumClosuresConverted, "Number of closures converted by the pass");
+
+static cl::opt<bool> DisableX86DomainReassignment(
+    "disable-x86-domain-reassignment", cl::Hidden,
+    cl::desc("X86: Disable Virtual Register Reassignment."), cl::init(false));
+
+namespace {
+enum RegDomain { NoDomain = -1, GPRDomain, MaskDomain, OtherDomain };
+
+static bool isGPR(const TargetRegisterClass *RC) {
+  return X86::GR64RegClass.hasSubClassEq(RC) ||
+         X86::GR32RegClass.hasSubClassEq(RC) ||
+         X86::GR16RegClass.hasSubClassEq(RC) ||
+         X86::GR8RegClass.hasSubClassEq(RC);
+}
+
+static bool isMask(const TargetRegisterClass *RC,
+                   const TargetRegisterInfo *TRI) {
+  return X86::VK16RegClass.hasSubClassEq(RC);
+}
+
+static RegDomain getDomain(const TargetRegisterClass *RC,
+                           const TargetRegisterInfo *TRI) {
+  if (isGPR(RC))
+    return GPRDomain;
+  if (isMask(RC, TRI))
+    return MaskDomain;
+  return OtherDomain;
+}
+
+/// Return a register class equivalent to \p SrcRC, in \p Domain.
+static const TargetRegisterClass *getDstRC(const TargetRegisterClass *SrcRC,
+                                           RegDomain Domain) {
+  assert(Domain == MaskDomain && "add domain");
+  if (SrcRC == &X86::GR8RegClass)
+    return &X86::VK8RegClass;
+  if (SrcRC == &X86::GR16RegClass)
+    return &X86::VK16RegClass;
+  if (SrcRC == &X86::GR32RegClass)
+    return &X86::VK32RegClass;
+  if (SrcRC == &X86::GR64RegClass)
+    return &X86::VK64RegClass;
+  llvm_unreachable("add register class");
+  return nullptr;
+}
+
+/// Abstract Instruction Converter class.
+class InstrConverterBase {
+protected:
+  unsigned SrcOpcode;
+
+public:
+  InstrConverterBase(unsigned SrcOpcode) : SrcOpcode(SrcOpcode) {}
+
+  virtual ~InstrConverterBase() {}
+
+  /// \returns true if \p MI is legal to convert.
+  virtual bool isLegal(const MachineInstr *MI,
+                       const TargetInstrInfo *TII) const {
+    assert(MI->getOpcode() == SrcOpcode &&
+           "Wrong instruction passed to converter");
+    return true;
+  }
+
+  /// Applies conversion to \p MI.
+  ///
+  /// \returns true if \p MI is no longer need, and can be deleted.
+  virtual bool convertInstr(MachineInstr *MI, const TargetInstrInfo *TII,
+                            MachineRegisterInfo *MRI) const = 0;
+
+  /// \returns the cost increment incurred by converting \p MI.
+  virtual double getExtraCost(const MachineInstr *MI,
+                              MachineRegisterInfo *MRI) const = 0;
+};
+
+/// An Instruction Converter which ignores the given instruction.
+/// For example, PHI instructions can be safely ignored since only the registers
+/// need to change.
+class InstrIgnore : public InstrConverterBase {
+public:
+  InstrIgnore(unsigned SrcOpcode) : InstrConverterBase(SrcOpcode) {}
+
+  bool convertInstr(MachineInstr *MI, const TargetInstrInfo *TII,
+                    MachineRegisterInfo *MRI) const override {
+    assert(isLegal(MI, TII) && "Cannot convert instruction");
+    return false;
+  }
+
+  double getExtraCost(const MachineInstr *MI,
+                      MachineRegisterInfo *MRI) const override {
+    return 0;
+  }
+};
+
+/// An Instruction Converter which replaces an instruction with another.
+class InstrReplacer : public InstrConverterBase {
+public:
+  /// Opcode of the destination instruction.
+  unsigned DstOpcode;
+
+  InstrReplacer(unsigned SrcOpcode, unsigned DstOpcode)
+      : InstrConverterBase(SrcOpcode), DstOpcode(DstOpcode) {}
+
+  bool isLegal(const MachineInstr *MI,
+               const TargetInstrInfo *TII) const override {
+    if (!InstrConverterBase::isLegal(MI, TII))
+      return false;
+    // It's illegal to replace an instruction that implicitly defines a register
+    // with an instruction that doesn't, unless that register dead.
+    for (auto &MO : MI->implicit_operands())
+      if (MO.isReg() && MO.isDef() && !MO.isDead() &&
+          !TII->get(DstOpcode).hasImplicitDefOfPhysReg(MO.getReg()))
+        return false;
+    return true;
+  }
+
+  bool convertInstr(MachineInstr *MI, const TargetInstrInfo *TII,
+                    MachineRegisterInfo *MRI) const override {
+    assert(isLegal(MI, TII) && "Cannot convert instruction");
+    MachineInstrBuilder Bld =
+        BuildMI(*MI->getParent(), MI, MI->getDebugLoc(), TII->get(DstOpcode));
+    // Transfer explicit operands from original instruction. Implicit operands
+    // are handled by BuildMI.
+    for (auto &Op : MI->explicit_operands())
+      Bld.add(Op);
+    return true;
+  }
+
+  double getExtraCost(const MachineInstr *MI,
+                      MachineRegisterInfo *MRI) const override {
+    // Assuming instructions have the same cost.
+    return 0;
+  }
+};
+
+/// An Instruction Converter which replaces an instruction with another, and
+/// adds a COPY from the new instruction's destination to the old one's.
+class InstrReplacerDstCOPY : public InstrConverterBase {
+public:
+  unsigned DstOpcode;
+
+  InstrReplacerDstCOPY(unsigned SrcOpcode, unsigned DstOpcode)
+      : InstrConverterBase(SrcOpcode), DstOpcode(DstOpcode) {}
+
+  bool convertInstr(MachineInstr *MI, const TargetInstrInfo *TII,
+                    MachineRegisterInfo *MRI) const override {
+    assert(isLegal(MI, TII) && "Cannot convert instruction");
+    MachineBasicBlock *MBB = MI->getParent();
+    auto &DL = MI->getDebugLoc();
+
+    unsigned Reg = MRI->createVirtualRegister(
+        TII->getRegClass(TII->get(DstOpcode), 0, MRI->getTargetRegisterInfo(),
+                         *MBB->getParent()));
+    MachineInstrBuilder Bld = BuildMI(*MBB, MI, DL, TII->get(DstOpcode), Reg);
+    for (unsigned Idx = 1, End = MI->getNumOperands(); Idx < End; ++Idx)
+      Bld.add(MI->getOperand(Idx));
+
+    BuildMI(*MBB, MI, DL, TII->get(TargetOpcode::COPY))
+        .add(MI->getOperand(0))
+        .addReg(Reg);
+
+    return true;
+  }
+
+  double getExtraCost(const MachineInstr *MI,
+                      MachineRegisterInfo *MRI) const override {
+    // Assuming instructions have the same cost, and that COPY is in the same
+    // domain so it will be eliminated.
+    return 0;
+  }
+};
+
+/// An Instruction Converter for replacing COPY instructions.
+class InstrCOPYReplacer : public InstrReplacer {
+public:
+  RegDomain DstDomain;
+
+  InstrCOPYReplacer(unsigned SrcOpcode, RegDomain DstDomain, unsigned DstOpcode)
+      : InstrReplacer(SrcOpcode, DstOpcode), DstDomain(DstDomain) {}
+
+  double getExtraCost(const MachineInstr *MI,
+                      MachineRegisterInfo *MRI) const override {
+    assert(MI->getOpcode() == TargetOpcode::COPY && "Expected a COPY");
+
+    for (auto &MO : MI->operands()) {
+      // Physical registers will not be converted. Assume that converting the
+      // COPY to the destination domain will eventually result in a actual
+      // instruction.
+      if (TargetRegisterInfo::isPhysicalRegister(MO.getReg()))
+        return 1;
+
+      RegDomain OpDomain = getDomain(MRI->getRegClass(MO.getReg()),
+                                     MRI->getTargetRegisterInfo());
+      // Converting a cross domain COPY to a same domain COPY should eliminate
+      // an insturction
+      if (OpDomain == DstDomain)
+        return -1;
+    }
+    return 0;
+  }
+};
+
+/// An Instruction Converter which replaces an instruction with a COPY.
+class InstrReplaceWithCopy : public InstrConverterBase {
+public:
+  // Source instruction operand Index, to be used as the COPY source.
+  unsigned SrcOpIdx;
+
+  InstrReplaceWithCopy(unsigned SrcOpcode, unsigned SrcOpIdx)
+      : InstrConverterBase(SrcOpcode), SrcOpIdx(SrcOpIdx) {}
+
+  bool convertInstr(MachineInstr *MI, const TargetInstrInfo *TII,
+                    MachineRegisterInfo *MRI) const override {
+    assert(isLegal(MI, TII) && "Cannot convert instruction");
+    BuildMI(*MI->getParent(), MI, MI->getDebugLoc(),
+            TII->get(TargetOpcode::COPY))
+        .add({MI->getOperand(0), MI->getOperand(SrcOpIdx)});
+    return true;
+  }
+
+  double getExtraCost(const MachineInstr *MI,
+                      MachineRegisterInfo *MRI) const override {
+    return 0;
+  }
+};
+
+/// An Instruction Converter which completely deletes an instruction.
+/// For example, IMPLICIT_DEF instructions can be deleted when converting from
+/// GPR to mask.
+class InstrDeleter : public InstrConverterBase {
+public:
+  InstrDeleter(unsigned SrcOpcode) : InstrConverterBase(SrcOpcode) {}
+
+  bool convertInstr(MachineInstr *MI, const TargetInstrInfo *TII,
+                    MachineRegisterInfo *MRI) const override {
+    assert(isLegal(MI, TII) && "Cannot convert instruction");
+    return true;
+  }
+
+  double getExtraCost(const MachineInstr *MI,
+                      MachineRegisterInfo *MRI) const override {
+    return 0;
+  }
+};
+
+// Key type to be used by the Instruction Converters map.
+// A converter is identified by <destination domain, source opcode>
+typedef std::pair<int, unsigned> InstrConverterBaseKeyTy;
+
+typedef DenseMap<InstrConverterBaseKeyTy, InstrConverterBase *>
+    InstrConverterBaseMap;
+
+/// A closure is a set of virtual register representing all of the edges in
+/// the closure, as well as all of the instructions connected by those edges.
+///
+/// A closure may encompass virtual registers in the same register bank that
+/// have different widths. For example, it may contain 32-bit GPRs as well as
+/// 64-bit GPRs.
+///
+/// A closure that computes an address (i.e. defines a virtual register that is
+/// used in a memory operand) excludes the instructions that contain memory
+/// operands using the address. Such an instruction will be included in a
+/// different closure that manipulates the loaded or stored value.
+class Closure {
+private:
+  const TargetInstrInfo *TII;
+  MachineRegisterInfo *MRI;
+
+  /// Virtual registers in the closure.
+  DenseSet<unsigned> Edges;
+
+  /// Instructions in the closure.
+  SmallVector<MachineInstr *, 8> Instrs;
+
+  /// A map of available Instruction Converters.
+  const InstrConverterBaseMap &Converters;
+
+  /// The register domain of this closure.
+  RegDomain Domain;
+
+  /// Domains which this closure can legally be reassigned to.
+  SmallVector<RegDomain, 2> LegalDstDomains;
+
+  SmallVector<RegDomain, 2> getLegalDstDomains() const {
+    return LegalDstDomains;
+  }
+
+  /// Enqueue \p Reg to be considered for addition to the closure.
+  void visitRegister(unsigned Reg, SmallVectorImpl<unsigned> &Worklist);
+
+  /// Add \p MI to this closure.
+  void encloseInstr(MachineInstr *MI);
+
+  /// Calculate the total cost of reassigning the closure to \p Domain.
+  double calculateCost(RegDomain Domain) const;
+
+  /// All edges that are included in some closure.
+  DenseSet<unsigned> &EnclosedEdges;
+
+  /// All instructions that are included in some closure.
+  DenseMap<MachineInstr *, Closure *> &EnclosedInstrs;
+
+public:
+  Closure(const TargetInstrInfo *TII, MachineRegisterInfo *MRI,
+          const InstrConverterBaseMap &Converters,
+          const SmallVector<RegDomain, 2> &LegalDstDomains,
+          DenseSet<unsigned> &EnclosedEdges,
+          DenseMap<MachineInstr *, Closure *> &EnclosedInstrs)
+      : TII(TII), MRI(MRI), Converters(Converters), Domain(NoDomain),
+        LegalDstDomains(LegalDstDomains), EnclosedEdges(EnclosedEdges),
+        EnclosedInstrs(EnclosedInstrs) {}
+
+  /// Starting from \Reg, expand the closure as much as possible.
+  void buildClosure(unsigned E);
+
+  /// /returns true if it is profitable to reassign the closure to \p Domain.
+  bool isReassignmentProfitable(RegDomain Domain) const;
+
+  /// Reassign the closure to \p Domain.
+  void Reassign(RegDomain Domain) const;
+
+  /// Mark this closure as illegal for reassignment to all domains.
+  void setAllIllegal() { LegalDstDomains.clear(); }
+
+  /// \returns true if this closure has domains which are legal to reassign to.
+  bool hasLegalDstDomain() const { return !LegalDstDomains.empty(); }
+
+  /// \returns true if is legal to reassign this closure to domain \p RD.
+  bool isLegal(RegDomain RD) const { return is_contained(LegalDstDomains, RD); }
+
+  bool empty() const { return Edges.empty(); }
+};
+
+class X86DomainReassignment : public MachineFunctionPass {
+public:
+  static char ID;
+
+  X86DomainReassignment() : MachineFunctionPass(ID) {
+    initializeX86DomainReassignmentPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+
+  StringRef getPassName() const override {
+    return "X86 Domain Reassignment Pass";
+  }
+
+private:
+  const X86Subtarget *STI;
+  MachineRegisterInfo *MRI;
+  const X86InstrInfo *TII;
+
+  /// A map of available Instruction Converters.
+  InstrConverterBaseMap Converters;
+
+  /// Initialize Converters map.
+  void initConverters();
+};
+
+char X86DomainReassignment::ID = 0;
+
+} // End anonymous namespace.
+
+void Closure::visitRegister(unsigned Reg, SmallVectorImpl<unsigned> &Worklist) {
+  if (EnclosedEdges.count(Reg))
+    return;
+
+  if (!TargetRegisterInfo::isVirtualRegister(Reg))
+    return;
+
+  if (!MRI->hasOneDef(Reg))
+    return;
+
+  RegDomain RD = getDomain(MRI->getRegClass(Reg), MRI->getTargetRegisterInfo());
+  // First edge in closure sets the domain.
+  if (Domain == NoDomain)
+    Domain = RD;
+
+  if (Domain != RD)
+    return;
+
+  Worklist.push_back(Reg);
+}
+
+void Closure::encloseInstr(MachineInstr *MI) {
+  auto I = EnclosedInstrs.find(MI);
+  if (I != EnclosedInstrs.end()) {
+    if (I->second != this)
+      // Instruction already belongs to another closure, avoid conflicts between
+      // closure and mark this closure as illegal.
+      setAllIllegal();
+    return;
+  }
+
+  EnclosedInstrs[MI] = this;
+  Instrs.push_back(MI);
+
+  // Mark closure as illegal for reassignment to domains, if there is no
+  // converter for the instruction or if the converter cannot convert the
+  // instruction.
+  erase_if(LegalDstDomains, [&](RegDomain D) {
+    InstrConverterBase *IC = Converters.lookup({D, MI->getOpcode()});
+    return !IC || !IC->isLegal(MI, TII);
+  });
+}
+
+double Closure::calculateCost(RegDomain DstDomain) const {
+  assert(isLegal(DstDomain) && "Cannot calculate cost for illegal closure");
+
+  double Cost = 0.0;
+  for (auto MI : Instrs)
+    Cost +=
+        Converters.lookup({DstDomain, MI->getOpcode()})->getExtraCost(MI, MRI);
+  return Cost;
+}
+
+bool Closure::isReassignmentProfitable(RegDomain Domain) const {
+  return calculateCost(Domain) < 0.0;
+}
+
+void Closure::Reassign(RegDomain Domain) const {
+  assert(isLegal(Domain) && "Cannot convert illegal closure");
+
+  // Iterate all instructions in the closure, convert each one using the
+  // appropriate converter.
+  SmallVector<MachineInstr *, 8> ToErase;
+  for (auto MI : Instrs)
+    if (Converters.lookup({Domain, MI->getOpcode()})
+            ->convertInstr(MI, TII, MRI))
+      ToErase.push_back(MI);
+
+  // Iterate all registers in the closure, replace them with registers in the
+  // destination domain.
+  for (unsigned Reg : Edges) {
+    MRI->setRegClass(Reg, getDstRC(MRI->getRegClass(Reg), Domain));
+    for (auto &MO : MRI->use_operands(Reg)) {
+      if (MO.isReg())
+        // Remove all subregister references as they are not valid in the
+        // destination domain.
+        MO.setSubReg(0);
+    }
+  }
+
+  for (auto MI : ToErase)
+    MI->eraseFromParent();
+}
+
+/// \returns true when \p Reg is used as part of an address calculation in \p
+/// MI.
+static bool usedAsAddr(const MachineInstr &MI, unsigned Reg,
+                       const TargetInstrInfo *TII) {
+  if (!MI.mayLoadOrStore())
+    return false;
+
+  const MCInstrDesc &Desc = TII->get(MI.getOpcode());
+  int MemOpStart = X86II::getMemoryOperandNo(Desc.TSFlags);
+  if (MemOpStart == -1)
+    return false;
+
+  MemOpStart += X86II::getOperandBias(Desc);
+  for (unsigned MemOpIdx = MemOpStart,
+                MemOpEnd = MemOpStart + X86::AddrNumOperands;
+       MemOpIdx < MemOpEnd; ++MemOpIdx) {
+    auto &Op = MI.getOperand(MemOpIdx);
+    if (Op.isReg() && Op.getReg() == Reg)
+      return true;
+  }
+  return false;
+}
+
+void Closure::buildClosure(unsigned Reg) {
+  SmallVector<unsigned, 4> Worklist;
+  visitRegister(Reg, Worklist);
+  while (!Worklist.empty()) {
+    unsigned CurReg = Worklist.pop_back_val();
+
+    // Register already in this closure.
+    if (!Edges.insert(CurReg).second)
+      continue;
+
+    MachineInstr *DefMI = MRI->getVRegDef(CurReg);
+    encloseInstr(DefMI);
+
+    // Add register used by the defining MI to the worklist.
+    // Do not add registers which are used in address calculation, they will be
+    // added to a different closure.
+    int OpEnd = DefMI->getNumOperands();
+    const MCInstrDesc &Desc = DefMI->getDesc();
+    int MemOp = X86II::getMemoryOperandNo(Desc.TSFlags);
+    if (MemOp != -1)
+      MemOp += X86II::getOperandBias(Desc);
+    for (int OpIdx = 0; OpIdx < OpEnd; ++OpIdx) {
+      if (OpIdx == MemOp) {
+        // skip address calculation.
+        OpIdx += (X86::AddrNumOperands - 1);
+        continue;
+      }
+      auto &Op = DefMI->getOperand(OpIdx);
+      if (!Op.isReg() || !Op.isUse())
+        continue;
+      visitRegister(Op.getReg(), Worklist);
+    }
+
+    // Expand closure through register uses.
+    for (auto &UseMI : MRI->use_nodbg_instructions(CurReg)) {
+      // We would like to avoid converting closures which calculare addresses,
+      // as this should remain in GPRs.
+      if (usedAsAddr(UseMI, CurReg, TII)) {
+        setAllIllegal();
+        continue;
+      }
+      encloseInstr(&UseMI);
+
+      for (auto &DefOp : UseMI.defs()) {
+        if (!DefOp.isReg())
+          continue;
+
+        unsigned DefReg = DefOp.getReg();
+        if (!TargetRegisterInfo::isVirtualRegister(DefReg)) {
+          setAllIllegal();
+          continue;
+        }
+        visitRegister(DefReg, Worklist);
+      }
+    }
+  }
+}
+
+void X86DomainReassignment::initConverters() {
+  Converters[{MaskDomain, TargetOpcode::PHI}] =
+      new InstrIgnore(TargetOpcode::PHI);
+
+  Converters[{MaskDomain, TargetOpcode::IMPLICIT_DEF}] =
+      new InstrDeleter(TargetOpcode::IMPLICIT_DEF);
+
+  Converters[{MaskDomain, TargetOpcode::INSERT_SUBREG}] =
+      new InstrReplaceWithCopy(TargetOpcode::INSERT_SUBREG, 2);
+
+  Converters[{MaskDomain, TargetOpcode::COPY}] =
+      new InstrCOPYReplacer(TargetOpcode::COPY, MaskDomain, TargetOpcode::COPY);
+
+  auto createReplacerDstCOPY = [&](unsigned From, unsigned To) {
+    Converters[{MaskDomain, From}] = new InstrReplacerDstCOPY(From, To);
+  };
+
+  createReplacerDstCOPY(X86::MOVZX32rm16, X86::KMOVWkm);
+  createReplacerDstCOPY(X86::MOVZX64rm16, X86::KMOVWkm);
+
+  createReplacerDstCOPY(X86::MOVZX32rr16, X86::KMOVWkk);
+  createReplacerDstCOPY(X86::MOVZX64rr16, X86::KMOVWkk);
+
+  if (STI->hasDQI()) {
+    createReplacerDstCOPY(X86::MOVZX16rm8, X86::KMOVBkm);
+    createReplacerDstCOPY(X86::MOVZX32rm8, X86::KMOVBkm);
+    createReplacerDstCOPY(X86::MOVZX64rm8, X86::KMOVBkm);
+
+    createReplacerDstCOPY(X86::MOVZX16rr8, X86::KMOVBkk);
+    createReplacerDstCOPY(X86::MOVZX32rr8, X86::KMOVBkk);
+    createReplacerDstCOPY(X86::MOVZX64rr8, X86::KMOVBkk);
+  }
+
+  auto createReplacer = [&](unsigned From, unsigned To) {
+    Converters[{MaskDomain, From}] = new InstrReplacer(From, To);
+  };
+
+  createReplacer(X86::MOV16rm, X86::KMOVWkm);
+  createReplacer(X86::MOV16mr, X86::KMOVWmk);
+  createReplacer(X86::MOV16rr, X86::KMOVWkk);
+  createReplacer(X86::SHR16ri, X86::KSHIFTRWri);
+  createReplacer(X86::SHL16ri, X86::KSHIFTLWri);
+  createReplacer(X86::NOT16r, X86::KNOTWrr);
+  createReplacer(X86::OR16rr, X86::KORWrr);
+  createReplacer(X86::AND16rr, X86::KANDWrr);
+  createReplacer(X86::XOR16rr, X86::KXORWrr);
+
+  if (STI->hasBWI()) {
+    createReplacer(X86::MOV32rm, X86::KMOVDkm);
+    createReplacer(X86::MOV64rm, X86::KMOVQkm);
+
+    createReplacer(X86::MOV32mr, X86::KMOVDmk);
+    createReplacer(X86::MOV64mr, X86::KMOVQmk);
+
+    createReplacer(X86::MOV32rr, X86::KMOVDkk);
+    createReplacer(X86::MOV64rr, X86::KMOVQkk);
+
+    createReplacer(X86::SHR32ri, X86::KSHIFTRDri);
+    createReplacer(X86::SHR64ri, X86::KSHIFTRQri);
+
+    createReplacer(X86::SHL32ri, X86::KSHIFTLDri);
+    createReplacer(X86::SHL64ri, X86::KSHIFTLQri);
+
+    createReplacer(X86::ADD32rr, X86::KADDDrr);
+    createReplacer(X86::ADD64rr, X86::KADDQrr);
+
+    createReplacer(X86::NOT32r, X86::KNOTDrr);
+    createReplacer(X86::NOT64r, X86::KNOTQrr);
+
+    createReplacer(X86::OR32rr, X86::KORDrr);
+    createReplacer(X86::OR64rr, X86::KORQrr);
+
+    createReplacer(X86::AND32rr, X86::KANDDrr);
+    createReplacer(X86::AND64rr, X86::KANDQrr);
+
+    createReplacer(X86::ANDN32rr, X86::KANDNDrr);
+    createReplacer(X86::ANDN64rr, X86::KANDNQrr);
+
+    createReplacer(X86::XOR32rr, X86::KXORDrr);
+    createReplacer(X86::XOR64rr, X86::KXORQrr);
+
+    createReplacer(X86::TEST32rr, X86::KTESTDrr);
+    createReplacer(X86::TEST64rr, X86::KTESTQrr);
+  }
+
+  if (STI->hasDQI()) {
+    createReplacer(X86::ADD8rr, X86::KADDBrr);
+    createReplacer(X86::ADD16rr, X86::KADDWrr);
+
+    createReplacer(X86::AND8rr, X86::KANDBrr);
+
+    createReplacer(X86::MOV8rm, X86::KMOVBkm);
+    createReplacer(X86::MOV8mr, X86::KMOVBmk);
+    createReplacer(X86::MOV8rr, X86::KMOVBkk);
+
+    createReplacer(X86::NOT8r, X86::KNOTBrr);
+
+    createReplacer(X86::OR8rr, X86::KORBrr);
+
+    createReplacer(X86::SHR8ri, X86::KSHIFTRBri);
+    createReplacer(X86::SHL8ri, X86::KSHIFTLBri);
+
+    createReplacer(X86::TEST8rr, X86::KTESTBrr);
+    createReplacer(X86::TEST16rr, X86::KTESTWrr);
+
+    createReplacer(X86::XOR8rr, X86::KXORBrr);
+  }
+}
+
+bool X86DomainReassignment::runOnMachineFunction(MachineFunction &MF) {
+  if (skipFunction(*MF.getFunction()))
+    return false;
+  if (DisableX86DomainReassignment)
+    return false;
+
+  DEBUG(dbgs() << "***** Machine Function before Domain Reassignment *****\n");
+  DEBUG(MF.print(dbgs()));
+
+  STI = &MF.getSubtarget<X86Subtarget>();
+  // GPR->K is the only transformation currently supported, bail out early if no
+  // AVX512.
+  if (!STI->hasAVX512())
+    return false;
+
+  MRI = &MF.getRegInfo();
+  assert(MRI->isSSA() && "Expected MIR to be in SSA form");
+
+  TII = STI->getInstrInfo();
+  initConverters();
+  bool Changed = false;
+
+  DenseSet<unsigned> EnclosedEdges;
+  DenseMap<MachineInstr *, Closure *> EnclosedInstrs;
+
+  std::vector<Closure> Closures;
+
+  // Go over all virtual registers and calculate a closure.
+  for (unsigned Idx = 0; Idx < MRI->getNumVirtRegs(); ++Idx) {
+    unsigned Reg = TargetRegisterInfo::index2VirtReg(Idx);
+
+    // GPR only current source domain supported.
+    if (!isGPR(MRI->getRegClass(Reg)))
+      continue;
+
+    // Register already in closure.
+    if (EnclosedEdges.count(Reg))
+      continue;
+
+    // Calculate closure starting with Reg.
+    Closure C(TII, MRI, Converters, {MaskDomain}, EnclosedEdges,
+              EnclosedInstrs);
+    C.buildClosure(Reg);
+
+    // Collect all closures that can potentially be converted.
+    if (!C.empty() && C.isLegal(MaskDomain))
+      Closures.push_back(std::move(C));
+  }
+
+  for (Closure &C : Closures)
+    if (C.isReassignmentProfitable(MaskDomain)) {
+      C.Reassign(MaskDomain);
+      ++NumClosuresConverted;
+      Changed = true;
+    }
+
+  for (auto I : Converters)
+    delete I.second;
+
+  DEBUG(dbgs() << "***** Machine Function after Domain Reassignment *****\n");
+  DEBUG(MF.print(dbgs()));
+
+  return Changed;
+}
+
+INITIALIZE_PASS(X86DomainReassignment, "x86-domain-reassignment",
+                "X86 Domain Reassignment Pass", false, false);
+
+/// Returns an instance of the Domain Reassignment pass.
+FunctionPass *llvm::createX86DomainReassignmentPass() {
+  return new X86DomainReassignment();
+}

lib/Target/X86/X86TargetMachine.cpp

static cl::opt<bool> EnableMachineCombinerPass("x86-machine-combiner",
                                cl::desc("Enable the machine combiner pass"),
                                cl::init(true), cl::Hidden);
 
 namespace llvm {
 
 void initializeWinEHStatePassPass(PassRegistry &);
 void initializeFixupLEAPassPass(PassRegistry &);
 void initializeX86ExecutionDepsFixPass(PassRegistry &);
+void initializeX86DomainReassignmentPass(PassRegistry &);
 
 } // end namespace llvm
 
 extern "C" void LLVMInitializeX86Target() {
   // Register the target.
   RegisterTargetMachine<X86TargetMachine> X(getTheX86_32Target());
   RegisterTargetMachine<X86TargetMachine> Y(getTheX86_64Target());
 
   PassRegistry &PR = *PassRegistry::getPassRegistry();
   initializeGlobalISel(PR);
   initializeWinEHStatePassPass(PR);
   initializeFixupBWInstPassPass(PR);
   initializeEvexToVexInstPassPass(PR);
   initializeFixupLEAPassPass(PR);
   initializeX86ExecutionDepsFixPass(PR);
+  initializeX86DomainReassignmentPass(PR);
 }
 
 static std::unique_ptr<TargetLoweringObjectFile> createTLOF(const Triple &TT) {
   if (TT.isOSBinFormatMachO()) {
     if (TT.getArch() == Triple::x86_64)
       return llvm::make_unique<X86_64MachoTargetObjectFile>();
     return llvm::make_unique<TargetLoweringObjectFileMachO>();
   }
public:
   void addIRPasses() override;
   bool addInstSelector() override;
   bool addIRTranslator() override;
   bool addLegalizeMachineIR() override;
   bool addRegBankSelect() override;
   bool addGlobalInstructionSelect() override;
   bool addILPOpts() override;
   bool addPreISel() override;
+  void addMachineSSAOptimization() override;
   void addPreRegAlloc() override;
   void addPostRegAlloc() override;
   void addPreEmitPass() override;
   void addPreSched2() override;
 };
 
 class X86ExecutionDepsFix : public ExecutionDepsFix {
 public:
if (getOptLevel() != CodeGenOpt::None) {
     addPass(&LiveRangeShrinkID);
     addPass(createX86FixupSetCC());
     addPass(createX86OptimizeLEAs());
     addPass(createX86CallFrameOptimization());
   }
 
   addPass(createX86WinAllocaExpander());
 }
+void X86PassConfig::addMachineSSAOptimization() {
+  addPass(createX86DomainReassignmentPass());
+  TargetPassConfig::addMachineSSAOptimization();
+}
 
 void X86PassConfig::addPostRegAlloc() {
   addPass(createX86FloatingPointStackifierPass());
 }
 
 void X86PassConfig::addPreSched2() { addPass(createX86ExpandPseudoPass()); }
 
 void X86PassConfig::addPreEmitPass() {

test/CodeGen/X86/avx512-insert-extract.ll

 ; KNL-NEXT:    vptestmd %zmm0, %zmm0, %k0
 ; KNL-NEXT:    kmovw %k0, %eax
 ; KNL-NEXT:    ## kill: %AX<def> %AX<kill> %EAX<kill>
 ; KNL-NEXT:    vzeroupper
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: test16:
 ; SKX:       ## BB#0:
-; SKX-NEXT:    movb (%rdi), %al
-; SKX-NEXT:    kmovd %esi, %k0
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpmovm2d %k1, %zmm0
-; SKX-NEXT:    vpmovm2d %k0, %zmm1
+; SKX-NEXT:    kmovb (%rdi), %k0
+; SKX-NEXT:    kmovd %esi, %k1
+; SKX-NEXT:    vpmovm2d %k0, %zmm0
+; SKX-NEXT:    vpmovm2d %k1, %zmm1
 ; SKX-NEXT:    vmovdqa32 {{.*#+}} zmm2 = [0,1,2,3,4,5,6,7,8,9,16,11,12,13,14,15]
 ; SKX-NEXT:    vpermi2d %zmm0, %zmm1, %zmm2
 ; SKX-NEXT:    vpmovd2m %zmm2, %k0
 ; SKX-NEXT:    kmovd %k0, %eax
 ; SKX-NEXT:    ## kill: %AX<def> %AX<kill> %EAX<kill>
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
   %x = load i1 , i1 * %addr, align 128

 ; KNL-NEXT:    vptestmq %zmm0, %zmm0, %k0
 ; KNL-NEXT:    kmovw %k0, %eax
 ; KNL-NEXT:    ## kill: %AL<def> %AL<kill> %EAX<kill>
 ; KNL-NEXT:    vzeroupper
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: test17:
 ; SKX:       ## BB#0:
-; SKX-NEXT:    movb (%rdi), %al
-; SKX-NEXT:    kmovd %esi, %k0
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    vpmovm2q %k1, %zmm0
-; SKX-NEXT:    vpmovm2q %k0, %zmm1
+; SKX-NEXT:    kmovb (%rdi), %k0
+; SKX-NEXT:    kmovd %esi, %k1
+; SKX-NEXT:    vpmovm2q %k0, %zmm0
+; SKX-NEXT:    vpmovm2q %k1, %zmm1
 ; SKX-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,2,3,8,5,6,7]
 ; SKX-NEXT:    vpermi2q %zmm0, %zmm1, %zmm2
 ; SKX-NEXT:    vpmovq2m %zmm2, %k0
 ; SKX-NEXT:    kmovd %k0, %eax
 ; SKX-NEXT:    ## kill: %AL<def> %AL<kill> %EAX<kill>
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
   %x = load i1 , i1 * %addr, align 128

test/CodeGen/X86/avx512-intrinsics-fast-isel.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -fast-isel -mtriple=i386-unknown-unknown -mattr=+avx512f | FileCheck %s --check-prefix=ALL --check-prefix=X32
 ; RUN: llc < %s -fast-isel -mtriple=x86_64-unknown-unknown -mattr=+avx512f | FileCheck %s --check-prefix=ALL --check-prefix=X64
 
 ; NOTE: This should use IR equivalent to what is generated by clang/test/CodeGen/avx512f-builtins.c
 
 define <8 x i64> @test_mm512_mask_set1_epi32(<8 x i64> %__O, i16 zeroext %__M, i32 %__A) {
 ; X32-LABEL: test_mm512_mask_set1_epi32:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovw %ecx, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastd %eax, %zmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_set1_epi32:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpbroadcastd %esi, %zmm0 {%k1}
 ; X64-NEXT:    retq
 entry:
   %vecinit.i.i = insertelement <16 x i32> undef, i32 %__A, i32 0
   %vecinit15.i.i = shufflevector <16 x i32> %vecinit.i.i, <16 x i32> undef, <16 x i32> zeroinitializer
   %0 = bitcast <8 x i64> %__O to <16 x i32>
   %1 = bitcast i16 %__M to <16 x i1>
   %2 = select <16 x i1> %1, <16 x i32> %vecinit15.i.i, <16 x i32> %0
   %3 = bitcast <16 x i32> %2 to <8 x i64>
   ret <8 x i64> %3
 }
 
 define <8 x i64> @test_mm512_maskz_set1_epi32(i16 zeroext %__M, i32 %__A)  {
 ; X32-LABEL: test_mm512_maskz_set1_epi32:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovw %ecx, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastd %eax, %zmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_set1_epi32:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpbroadcastd %esi, %zmm0 {%k1} {z}
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <4 x i32> %arg0, <4 x i32> undef, <16 x i32> zeroinitializer
   %res1 = bitcast <16 x i32> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_broadcastd_epi32(<8 x i64> %a0, i16 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm512_mask_broadcastd_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastd %xmm1, %zmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_broadcastd_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpbroadcastd %xmm1, %zmm0 {%k1}
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <16 x i32>
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %arg2 = bitcast <2 x i64> %a2 to <4 x i32>
   %res0 = shufflevector <4 x i32> %arg2, <4 x i32> undef, <16 x i32> zeroinitializer
   %res1 = select <16 x i1> %arg1, <16 x i32> %res0, <16 x i32> %arg0
   %res2 = bitcast <16 x i32> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_broadcastd_epi32(i16 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm512_maskz_broadcastd_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastd %xmm0, %zmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_broadcastd_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpbroadcastd %xmm0, %zmm0 {%k1} {z}
 ; X64-NEXT:    retq

 ; X64-NEXT:    retq
   %res = shufflevector <4 x float> %a0, <4 x float> undef, <16 x i32> zeroinitializer
   ret <16 x float> %res
 }
 
 define <16 x float> @test_mm512_mask_broadcastss_ps(<16 x float> %a0, i16 %a1, <4 x float> %a2) {
 ; X32-LABEL: test_mm512_mask_broadcastss_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vbroadcastss %xmm1, %zmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_broadcastss_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vbroadcastss %xmm1, %zmm0 {%k1}
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <4 x float> %a2, <4 x float> undef, <16 x i32> zeroinitializer
   %res1 = select <16 x i1> %arg1, <16 x float> %res0, <16 x float> %a0
   ret <16 x float> %res1
 }
 
 define <16 x float> @test_mm512_maskz_broadcastss_ps(i16 %a0, <4 x float> %a1) {
 ; X32-LABEL: test_mm512_maskz_broadcastss_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vbroadcastss %xmm0, %zmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_broadcastss_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vbroadcastss %xmm0, %zmm0 {%k1} {z}
 ; X64-NEXT:    retq

 ; X64-NEXT:    retq
   %res = shufflevector <16 x float> %a0, <16 x float> undef, <16 x i32> <i32 1, i32 1, i32 3, i32 3, i32 5, i32 5, i32 7, i32 7, i32 9, i32 9, i32 11, i32 11, i32 13, i32 13, i32 15, i32 15>
   ret <16 x float> %res
 }
 
 define <16 x float> @test_mm512_mask_movehdup_ps(<16 x float> %a0, i16 %a1, <16 x float> %a2) {
 ; X32-LABEL: test_mm512_mask_movehdup_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vmovshdup {{.*#+}} zmm0 {%k1} = zmm1[1,1,3,3,5,5,7,7,9,9,11,11,13,13,15,15]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_movehdup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vmovshdup {{.*#+}} zmm0 {%k1} = zmm1[1,1,3,3,5,5,7,7,9,9,11,11,13,13,15,15]
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <16 x float> %a2, <16 x float> undef, <16 x i32> <i32 1, i32 1, i32 3, i32 3, i32 5, i32 5, i32 7, i32 7, i32 9, i32 9, i32 11, i32 11, i32 13, i32 13, i32 15, i32 15>
   %res1 = select <16 x i1> %arg1, <16 x float> %res0, <16 x float> %a0
   ret <16 x float> %res1
 }
 
 define <16 x float> @test_mm512_maskz_movehdup_ps(i16 %a0, <16 x float> %a1) {
 ; X32-LABEL: test_mm512_maskz_movehdup_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vmovshdup {{.*#+}} zmm0 {%k1} {z} = zmm0[1,1,3,3,5,5,7,7,9,9,11,11,13,13,15,15]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_movehdup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vmovshdup {{.*#+}} zmm0 {%k1} {z} = zmm0[1,1,3,3,5,5,7,7,9,9,11,11,13,13,15,15]
 ; X64-NEXT:    retq

 ; X64-NEXT:    retq
   %res = shufflevector <16 x float> %a0, <16 x float> undef, <16 x i32> <i32 0, i32 0, i32 2, i32 2, i32 4, i32 4, i32 6, i32 6, i32 8, i32 8, i32 10, i32 10, i32 12, i32 12, i32 14, i32 14>
   ret <16 x float> %res
 }
 
 define <16 x float> @test_mm512_mask_moveldup_ps(<16 x float> %a0, i16 %a1, <16 x float> %a2) {
 ; X32-LABEL: test_mm512_mask_moveldup_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vmovsldup {{.*#+}} zmm0 {%k1} = zmm1[0,0,2,2,4,4,6,6,8,8,10,10,12,12,14,14]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_moveldup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vmovsldup {{.*#+}} zmm0 {%k1} = zmm1[0,0,2,2,4,4,6,6,8,8,10,10,12,12,14,14]
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <16 x float> %a2, <16 x float> undef, <16 x i32> <i32 0, i32 0, i32 2, i32 2, i32 4, i32 4, i32 6, i32 6, i32 8, i32 8, i32 10, i32 10, i32 12, i32 12, i32 14, i32 14>
   %res1 = select <16 x i1> %arg1, <16 x float> %res0, <16 x float> %a0
   ret <16 x float> %res1
 }
 
 define <16 x float> @test_mm512_maskz_moveldup_ps(i16 %a0, <16 x float> %a1) {
 ; X32-LABEL: test_mm512_maskz_moveldup_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vmovsldup {{.*#+}} zmm0 {%k1} {z} = zmm0[0,0,2,2,4,4,6,6,8,8,10,10,12,12,14,14]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_moveldup_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vmovsldup {{.*#+}} zmm0 {%k1} {z} = zmm0[0,0,2,2,4,4,6,6,8,8,10,10,12,12,14,14]
 ; X64-NEXT:    retq

 ; X64-NEXT:    retq
   %res = shufflevector <16 x float> %a0, <16 x float> undef, <16 x i32> <i32 2, i32 0, i32 0, i32 0, i32 6, i32 4, i32 4, i32 4, i32 10, i32 8, i32 8, i32 8, i32 14, i32 12, i32 12, i32 12>
   ret <16 x float> %res
 }
 
 define <16 x float> @test_mm512_mask_permute_ps(<16 x float> %a0, i16 %a1, <16 x float> %a2) {
 ; X32-LABEL: test_mm512_mask_permute_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = zmm1[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_permute_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} = zmm1[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <16 x float> %a2, <16 x float> undef, <16 x i32> <i32 2, i32 0, i32 0, i32 0, i32 6, i32 4, i32 4, i32 4, i32 10, i32 8, i32 8, i32 8, i32 14, i32 12, i32 12, i32 12>
   %res1 = select <16 x i1> %arg1, <16 x float> %res0, <16 x float> %a0
   ret <16 x float> %res1
 }
 
 define <16 x float> @test_mm512_maskz_permute_ps(i16 %a0, <16 x float> %a1) {
 ; X32-LABEL: test_mm512_maskz_permute_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_permute_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpermilps {{.*#+}} zmm0 {%k1} {z} = zmm0[2,0,0,0,6,4,4,4,10,8,8,8,14,12,12,12]
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <16 x i32> %arg0, <16 x i32> undef, <16 x i32> <i32 1, i32 0, i32 0, i32 0, i32 5, i32 4, i32 4, i32 4, i32 9, i32 8, i32 8, i32 8, i32 13, i32 12, i32 12, i32 12>
   %res1 = bitcast <16 x i32> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_shuffle_epi32(<8 x i64> %a0, i16 %a1, <8 x i64> %a2) {
 ; X32-LABEL: test_mm512_mask_shuffle_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpshufd {{.*#+}} zmm0 {%k1} = zmm1[1,0,0,0,5,4,4,4,9,8,8,8,13,12,12,12]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_shuffle_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpshufd {{.*#+}} zmm0 {%k1} = zmm1[1,0,0,0,5,4,4,4,9,8,8,8,13,12,12,12]
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <16 x i32>
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %arg2 = bitcast <8 x i64> %a2 to <16 x i32>
   %res0 = shufflevector <16 x i32> %arg2, <16 x i32> undef, <16 x i32> <i32 1, i32 0, i32 0, i32 0, i32 5, i32 4, i32 4, i32 4, i32 9, i32 8, i32 8, i32 8, i32 13, i32 12, i32 12, i32 12>
   %res1 = select <16 x i1> %arg1, <16 x i32> %res0, <16 x i32> %arg0
   %res2 = bitcast <16 x i32> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_shuffle_epi32(i16 %a0, <8 x i64> %a1) {
 ; X32-LABEL: test_mm512_maskz_shuffle_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpshufd {{.*#+}} zmm0 {%k1} {z} = zmm0[1,0,0,0,5,4,4,4,9,8,8,8,13,12,12,12]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_shuffle_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpshufd {{.*#+}} zmm0 {%k1} {z} = zmm0[1,0,0,0,5,4,4,4,9,8,8,8,13,12,12,12]
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <16 x i32> %arg0, <16 x i32> %arg1, <16 x i32> <i32 2, i32 18, i32 3, i32 19, i32 6, i32 22, i32 7, i32 23, i32 10, i32 26, i32 11, i32 27, i32 14, i32 30, i32 15, i32 31>
   %res1 = bitcast <16 x i32> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_unpackhi_epi32(<8 x i64> %a0, i16 %a1, <8 x i64> %a2, <8 x i64> %a3) {
 ; X32-LABEL: test_mm512_mask_unpackhi_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpckhdq {{.*#+}} zmm0 {%k1} = zmm1[2],zmm2[2],zmm1[3],zmm2[3],zmm1[6],zmm2[6],zmm1[7],zmm2[7],zmm1[10],zmm2[10],zmm1[11],zmm2[11],zmm1[14],zmm2[14],zmm1[15],zmm2[15]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_unpackhi_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpunpckhdq {{.*#+}} zmm0 {%k1} = zmm1[2],zmm2[2],zmm1[3],zmm2[3],zmm1[6],zmm2[6],zmm1[7],zmm2[7],zmm1[10],zmm2[10],zmm1[11],zmm2[11],zmm1[14],zmm2[14],zmm1[15],zmm2[15]
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <16 x i32>
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %arg2 = bitcast <8 x i64> %a2 to <16 x i32>
   %arg3 = bitcast <8 x i64> %a3 to <16 x i32>
   %res0 = shufflevector <16 x i32> %arg2, <16 x i32> %arg3, <16 x i32> <i32 2, i32 18, i32 3, i32 19, i32 6, i32 22, i32 7, i32 23, i32 10, i32 26, i32 11, i32 27, i32 14, i32 30, i32 15, i32 31>
   %res1 = select <16 x i1> %arg1, <16 x i32> %res0, <16 x i32> %arg0
   %res2 = bitcast <16 x i32> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_unpackhi_epi32(i16 %a0, <8 x i64> %a1, <8 x i64> %a2) {
 ; X32-LABEL: test_mm512_maskz_unpackhi_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpckhdq {{.*#+}} zmm0 {%k1} {z} = zmm0[2],zmm1[2],zmm0[3],zmm1[3],zmm0[6],zmm1[6],zmm0[7],zmm1[7],zmm0[10],zmm1[10],zmm0[11],zmm1[11],zmm0[14],zmm1[14],zmm0[15],zmm1[15]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_unpackhi_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpunpckhdq {{.*#+}} zmm0 {%k1} {z} = zmm0[2],zmm1[2],zmm0[3],zmm1[3],zmm0[6],zmm1[6],zmm0[7],zmm1[7],zmm0[10],zmm1[10],zmm0[11],zmm1[11],zmm0[14],zmm1[14],zmm0[15],zmm1[15]
 ; X64-NEXT:    retq

 ; X64-NEXT:    retq
   %res = shufflevector <16 x float> %a0, <16 x float> %a1, <16 x i32> <i32 2, i32 18, i32 3, i32 19, i32 6, i32 22, i32 7, i32 23, i32 10, i32 26, i32 11, i32 27, i32 14, i32 30, i32 15, i32 31>
   ret <16 x float> %res
 }
 
 define <16 x float> @test_mm512_mask_unpackhi_ps(<16 x float> %a0, i16 %a1, <16 x float> %a2, <16 x float> %a3) {
 ; X32-LABEL: test_mm512_mask_unpackhi_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vunpckhps {{.*#+}} zmm0 {%k1} = zmm1[2],zmm2[2],zmm1[3],zmm2[3],zmm1[6],zmm2[6],zmm1[7],zmm2[7],zmm1[10],zmm2[10],zmm1[11],zmm2[11],zmm1[14],zmm2[14],zmm1[15],zmm2[15]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_unpackhi_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vunpckhps {{.*#+}} zmm0 {%k1} = zmm1[2],zmm2[2],zmm1[3],zmm2[3],zmm1[6],zmm2[6],zmm1[7],zmm2[7],zmm1[10],zmm2[10],zmm1[11],zmm2[11],zmm1[14],zmm2[14],zmm1[15],zmm2[15]
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <16 x float> %a2, <16 x float> %a3, <16 x i32> <i32 2, i32 18, i32 3, i32 19, i32 6, i32 22, i32 7, i32 23, i32 10, i32 26, i32 11, i32 27, i32 14, i32 30, i32 15, i32 31>
   %res1 = select <16 x i1> %arg1, <16 x float> %res0, <16 x float> %a0
   ret <16 x float> %res1
 }
 
 define <16 x float> @test_mm512_maskz_unpackhi_ps(i16 %a0, <16 x float> %a1, <16 x float> %a2) {
 ; X32-LABEL: test_mm512_maskz_unpackhi_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vunpckhps {{.*#+}} zmm0 {%k1} {z} = zmm0[2],zmm1[2],zmm0[3],zmm1[3],zmm0[6],zmm1[6],zmm0[7],zmm1[7],zmm0[10],zmm1[10],zmm0[11],zmm1[11],zmm0[14],zmm1[14],zmm0[15],zmm1[15]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_unpackhi_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vunpckhps {{.*#+}} zmm0 {%k1} {z} = zmm0[2],zmm1[2],zmm0[3],zmm1[3],zmm0[6],zmm1[6],zmm0[7],zmm1[7],zmm0[10],zmm1[10],zmm0[11],zmm1[11],zmm0[14],zmm1[14],zmm0[15],zmm1[15]
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <16 x i32> %arg0, <16 x i32> %arg1, <16 x i32> <i32 0, i32 16, i32 1, i32 17, i32 4, i32 20, i32 5, i32 21, i32 8, i32 24, i32 9, i32 25, i32 12, i32 28, i32 13, i32 29>
   %res1 = bitcast <16 x i32> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_unpacklo_epi32(<8 x i64> %a0, i16 %a1, <8 x i64> %a2, <8 x i64> %a3) {
 ; X32-LABEL: test_mm512_mask_unpacklo_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpckldq {{.*#+}} zmm0 {%k1} = zmm1[0],zmm2[0],zmm1[1],zmm2[1],zmm1[4],zmm2[4],zmm1[5],zmm2[5],zmm1[8],zmm2[8],zmm1[9],zmm2[9],zmm1[12],zmm2[12],zmm1[13],zmm2[13]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_unpacklo_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpunpckldq {{.*#+}} zmm0 {%k1} = zmm1[0],zmm2[0],zmm1[1],zmm2[1],zmm1[4],zmm2[4],zmm1[5],zmm2[5],zmm1[8],zmm2[8],zmm1[9],zmm2[9],zmm1[12],zmm2[12],zmm1[13],zmm2[13]
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <16 x i32>
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %arg2 = bitcast <8 x i64> %a2 to <16 x i32>
   %arg3 = bitcast <8 x i64> %a3 to <16 x i32>
   %res0 = shufflevector <16 x i32> %arg2, <16 x i32> %arg3, <16 x i32> <i32 0, i32 16, i32 1, i32 17, i32 4, i32 20, i32 5, i32 21, i32 8, i32 24, i32 9, i32 25, i32 12, i32 28, i32 13, i32 29>
   %res1 = select <16 x i1> %arg1, <16 x i32> %res0, <16 x i32> %arg0
   %res2 = bitcast <16 x i32> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_unpacklo_epi32(i16 %a0, <8 x i64> %a1, <8 x i64> %a2) {
 ; X32-LABEL: test_mm512_maskz_unpacklo_epi32:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpckldq {{.*#+}} zmm0 {%k1} {z} = zmm0[0],zmm1[0],zmm0[1],zmm1[1],zmm0[4],zmm1[4],zmm0[5],zmm1[5],zmm0[8],zmm1[8],zmm0[9],zmm1[9],zmm0[12],zmm1[12],zmm0[13],zmm1[13]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_unpacklo_epi32:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vpunpckldq {{.*#+}} zmm0 {%k1} {z} = zmm0[0],zmm1[0],zmm0[1],zmm1[1],zmm0[4],zmm1[4],zmm0[5],zmm1[5],zmm0[8],zmm1[8],zmm0[9],zmm1[9],zmm0[12],zmm1[12],zmm0[13],zmm1[13]
 ; X64-NEXT:    retq

 ; X64-NEXT:    retq
   %res = shufflevector <16 x float> %a0, <16 x float> %a1, <16 x i32> <i32 0, i32 16, i32 1, i32 17, i32 4, i32 20, i32 5, i32 21, i32 8, i32 24, i32 9, i32 25, i32 12, i32 28, i32 13, i32 29>
   ret <16 x float> %res
 }
 
 define <16 x float> @test_mm512_mask_unpacklo_ps(<16 x float> %a0, i16 %a1, <16 x float> %a2, <16 x float> %a3) {
 ; X32-LABEL: test_mm512_mask_unpacklo_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vunpcklps {{.*#+}} zmm0 {%k1} = zmm1[0],zmm2[0],zmm1[1],zmm2[1],zmm1[4],zmm2[4],zmm1[5],zmm2[5],zmm1[8],zmm2[8],zmm1[9],zmm2[9],zmm1[12],zmm2[12],zmm1[13],zmm2[13]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_unpacklo_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vunpcklps {{.*#+}} zmm0 {%k1} = zmm1[0],zmm2[0],zmm1[1],zmm2[1],zmm1[4],zmm2[4],zmm1[5],zmm2[5],zmm1[8],zmm2[8],zmm1[9],zmm2[9],zmm1[12],zmm2[12],zmm1[13],zmm2[13]
 ; X64-NEXT:    retq
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %res0 = shufflevector <16 x float> %a2, <16 x float> %a3, <16 x i32> <i32 0, i32 16, i32 1, i32 17, i32 4, i32 20, i32 5, i32 21, i32 8, i32 24, i32 9, i32 25, i32 12, i32 28, i32 13, i32 29>
   %res1 = select <16 x i1> %arg1, <16 x float> %res0, <16 x float> %a0
   ret <16 x float> %res1
 }
 
 define <16 x float> @test_mm512_maskz_unpacklo_ps(i16 %a0, <16 x float> %a1, <16 x float> %a2) {
 ; X32-LABEL: test_mm512_maskz_unpacklo_ps:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovw %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vunpcklps {{.*#+}} zmm0 {%k1} {z} = zmm0[0],zmm1[0],zmm0[1],zmm1[1],zmm0[4],zmm1[4],zmm0[5],zmm1[5],zmm0[8],zmm1[8],zmm0[9],zmm1[9],zmm0[12],zmm1[12],zmm0[13],zmm1[13]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_unpacklo_ps:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovw %edi, %k1
 ; X64-NEXT:    vunpcklps {{.*#+}} zmm0 {%k1} {z} = zmm0[0],zmm1[0],zmm0[1],zmm1[1],zmm0[4],zmm1[4],zmm0[5],zmm1[5],zmm0[8],zmm1[8],zmm0[9],zmm1[9],zmm0[12],zmm1[12],zmm0[13],zmm1[13]
 ; X64-NEXT:    retq

test/CodeGen/X86/avx512-mask-op.ll

; AVX512DQ-NEXT:    retq
   %d = sext <64 x i1>%c to <64 x i8>
   ret <64 x i8>%d
 }
 
 define <8 x i1> @test18(i8 %a, i16 %y) {
 ; KNL-LABEL: test18:
 ; KNL:       ## BB#0:
 ; KNL-NEXT:    kmovw %edi, %k1
-; KNL-NEXT:    kmovw %esi, %k0
-; KNL-NEXT:    kshiftlw $7, %k0, %k2
-; KNL-NEXT:    kshiftrw $15, %k2, %k2
-; KNL-NEXT:    kmovw %k2, %eax
-; KNL-NEXT:    kshiftlw $6, %k0, %k0
+; KNL-NEXT:    kmovw %esi, %k2
+; KNL-NEXT:    kshiftlw $7, %k2, %k0
 ; KNL-NEXT:    kshiftrw $15, %k0, %k0
-; KNL-NEXT:    kmovw %k0, %ecx
+; KNL-NEXT:    kshiftlw $6, %k2, %k2
+; KNL-NEXT:    kshiftrw $15, %k2, %k2
 ; KNL-NEXT:    vpternlogq $255, %zmm0, %zmm0, %zmm0 {%k1} {z}
-; KNL-NEXT:    kmovw %ecx, %k1
-; KNL-NEXT:    vpternlogq $255, %zmm1, %zmm1, %zmm1 {%k1} {z}
+; KNL-NEXT:    vpternlogq $255, %zmm1, %zmm1, %zmm1 {%k2} {z}
 ; KNL-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,2,3,4,5,8,7]
 ; KNL-NEXT:    vpermi2q %zmm1, %zmm0, %zmm2
 ; KNL-NEXT:    vpsllq $63, %zmm2, %zmm0
-; KNL-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; KNL-NEXT:    kshiftlw $1, %k0, %k0
-; KNL-NEXT:    kshiftrw $1, %k0, %k0
-; KNL-NEXT:    kmovw %eax, %k1
-; KNL-NEXT:    kshiftlw $7, %k1, %k1
-; KNL-NEXT:    korw %k1, %k0, %k1
+; KNL-NEXT:    vptestmq %zmm0, %zmm0, %k1
+; KNL-NEXT:    kshiftlw $1, %k1, %k1
+; KNL-NEXT:    kshiftrw $1, %k1, %k1
+; KNL-NEXT:    kshiftlw $7, %k0, %k0
+; KNL-NEXT:    korw %k0, %k1, %k1
 ; KNL-NEXT:    vpternlogq $255, %zmm0, %zmm0, %zmm0 {%k1} {z}
 ; KNL-NEXT:    vpmovqw %zmm0, %xmm0
 ; KNL-NEXT:    vzeroupper
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: test18:
 ; SKX:       ## BB#0:
-; SKX-NEXT:    kmovd %edi, %k0
-; SKX-NEXT:    kmovd %esi, %k1
-; SKX-NEXT:    kshiftlw $7, %k1, %k2
+; SKX-NEXT:    kmovd %edi, %k1
+; SKX-NEXT:    kmovd %esi, %k2
+; SKX-NEXT:    kshiftlw $7, %k2, %k0
+; SKX-NEXT:    kshiftrw $15, %k0, %k0
+; SKX-NEXT:    kshiftlw $6, %k2, %k2
 ; SKX-NEXT:    kshiftrw $15, %k2, %k2
-; SKX-NEXT:    kmovd %k2, %eax
-; SKX-NEXT:    kshiftlw $6, %k1, %k1
-; SKX-NEXT:    kshiftrw $15, %k1, %k1
-; SKX-NEXT:    kmovd %k1, %ecx
-; SKX-NEXT:    vpmovm2q %k0, %zmm0
-; SKX-NEXT:    kmovd %ecx, %k0
-; SKX-NEXT:    vpmovm2q %k0, %zmm1
+; SKX-NEXT:    vpmovm2q %k1, %zmm0
+; SKX-NEXT:    vpmovm2q %k2, %zmm1
 ; SKX-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,2,3,4,5,8,7]
 ; SKX-NEXT:    vpermi2q %zmm1, %zmm0, %zmm2
-; SKX-NEXT:    vpmovq2m %zmm2, %k0
-; SKX-NEXT:    kshiftlb $1, %k0, %k0
-; SKX-NEXT:    kshiftrb $1, %k0, %k0
-; SKX-NEXT:    kmovd %eax, %k1
-; SKX-NEXT:    kshiftlb $7, %k1, %k1
-; SKX-NEXT:    korb %k1, %k0, %k0
+; SKX-NEXT:    vpmovq2m %zmm2, %k1
+; SKX-NEXT:    kshiftlb $1, %k1, %k1
+; SKX-NEXT:    kshiftrb $1, %k1, %k1
+; SKX-NEXT:    kshiftlb $7, %k0, %k0
+; SKX-NEXT:    korb %k0, %k1, %k0
 ; SKX-NEXT:    vpmovm2w %k0, %xmm0
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
 ;
 ; AVX512BW-LABEL: test18:
 ; AVX512BW:       ## BB#0:
 ; AVX512BW-NEXT:    kmovd %edi, %k1
-; AVX512BW-NEXT:    kmovd %esi, %k0
-; AVX512BW-NEXT:    kshiftlw $7, %k0, %k2
-; AVX512BW-NEXT:    kshiftrw $15, %k2, %k2
-; AVX512BW-NEXT:    kmovd %k2, %eax
-; AVX512BW-NEXT:    kshiftlw $6, %k0, %k0
+; AVX512BW-NEXT:    kmovd %esi, %k2
+; AVX512BW-NEXT:    kshiftlw $7, %k2, %k0
 ; AVX512BW-NEXT:    kshiftrw $15, %k0, %k0
-; AVX512BW-NEXT:    kmovd %k0, %ecx
+; AVX512BW-NEXT:    kshiftlw $6, %k2, %k2
+; AVX512BW-NEXT:    kshiftrw $15, %k2, %k2
 ; AVX512BW-NEXT:    vpternlogq $255, %zmm0, %zmm0, %zmm0 {%k1} {z}
-; AVX512BW-NEXT:    kmovd %ecx, %k1
-; AVX512BW-NEXT:    vpternlogq $255, %zmm1, %zmm1, %zmm1 {%k1} {z}
+; AVX512BW-NEXT:    vpternlogq $255, %zmm1, %zmm1, %zmm1 {%k2} {z}
 ; AVX512BW-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,2,3,4,5,8,7]
 ; AVX512BW-NEXT:    vpermi2q %zmm1, %zmm0, %zmm2
 ; AVX512BW-NEXT:    vpsllq $63, %zmm2, %zmm0
-; AVX512BW-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; AVX512BW-NEXT:    kshiftlw $1, %k0, %k0
-; AVX512BW-NEXT:    kshiftrw $1, %k0, %k0
-; AVX512BW-NEXT:    kmovd %eax, %k1
-; AVX512BW-NEXT:    kshiftlw $7, %k1, %k1
-; AVX512BW-NEXT:    korw %k1, %k0, %k0
+; AVX512BW-NEXT:    vptestmq %zmm0, %zmm0, %k1
+; AVX512BW-NEXT:    kshiftlw $1, %k1, %k1
+; AVX512BW-NEXT:    kshiftrw $1, %k1, %k1
+; AVX512BW-NEXT:    kshiftlw $7, %k0, %k0
+; AVX512BW-NEXT:    korw %k0, %k1, %k0
 ; AVX512BW-NEXT:    vpmovm2w %k0, %zmm0
 ; AVX512BW-NEXT:    ## kill: %XMM0<def> %XMM0<kill> %ZMM0<kill>
 ; AVX512BW-NEXT:    vzeroupper
 ; AVX512BW-NEXT:    retq
 ;
 ; AVX512DQ-LABEL: test18:
 ; AVX512DQ:       ## BB#0:
-; AVX512DQ-NEXT:    kmovw %edi, %k0
-; AVX512DQ-NEXT:    kmovw %esi, %k1
-; AVX512DQ-NEXT:    kshiftlw $7, %k1, %k2
+; AVX512DQ-NEXT:    kmovw %edi, %k1
+; AVX512DQ-NEXT:    kmovw %esi, %k2
+; AVX512DQ-NEXT:    kshiftlw $7, %k2, %k0
+; AVX512DQ-NEXT:    kshiftrw $15, %k0, %k0
+; AVX512DQ-NEXT:    kshiftlw $6, %k2, %k2
 ; AVX512DQ-NEXT:    kshiftrw $15, %k2, %k2
-; AVX512DQ-NEXT:    kmovw %k2, %eax
-; AVX512DQ-NEXT:    kshiftlw $6, %k1, %k1
-; AVX512DQ-NEXT:    kshiftrw $15, %k1, %k1
-; AVX512DQ-NEXT:    kmovw %k1, %ecx
-; AVX512DQ-NEXT:    vpmovm2q %k0, %zmm0
-; AVX512DQ-NEXT:    kmovw %ecx, %k0
-; AVX512DQ-NEXT:    vpmovm2q %k0, %zmm1
+; AVX512DQ-NEXT:    vpmovm2q %k1, %zmm0
+; AVX512DQ-NEXT:    vpmovm2q %k2, %zmm1
 ; AVX512DQ-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,2,3,4,5,8,7]
 ; AVX512DQ-NEXT:    vpermi2q %zmm1, %zmm0, %zmm2
-; AVX512DQ-NEXT:    vpmovq2m %zmm2, %k0
-; AVX512DQ-NEXT:    kshiftlb $1, %k0, %k0
-; AVX512DQ-NEXT:    kshiftrb $1, %k0, %k0
-; AVX512DQ-NEXT:    kmovw %eax, %k1
-; AVX512DQ-NEXT:    kshiftlb $7, %k1, %k1
-; AVX512DQ-NEXT:    korb %k1, %k0, %k0
+; AVX512DQ-NEXT:    vpmovq2m %zmm2, %k1
+; AVX512DQ-NEXT:    kshiftlb $1, %k1, %k1
+; AVX512DQ-NEXT:    kshiftrb $1, %k1, %k1
+; AVX512DQ-NEXT:    kshiftlb $7, %k0, %k0
+; AVX512DQ-NEXT:    korb %k0, %k1, %k0
 ; AVX512DQ-NEXT:    vpmovm2q %k0, %zmm0
 ; AVX512DQ-NEXT:    vpmovqw %zmm0, %xmm0
 ; AVX512DQ-NEXT:    vzeroupper
 ; AVX512DQ-NEXT:    retq
   %b = bitcast i8 %a to <8 x i1>
   %b1 = bitcast i16 %y to <16 x i1>
   %el1 = extractelement <16 x i1>%b1, i32 8
   %el2 = extractelement <16 x i1>%b1, i32 9

test/CodeGen/X86/avx512-schedule.ll

; SKX-NEXT:    retq
   %c = insertelement <64 x i1>%a, i1 %b, i32 5
   %d = sext <64 x i1>%c to <64 x i8>
   ret <64 x i8>%d
 }
 
 define <8 x i1> @vmov_test18(i8 %a, i16 %y) {
 ; CHECK-LABEL: vmov_test18:
 ; CHECK:       # BB#0:
-; CHECK-NEXT:    kmovd %edi, %k0
-; CHECK-NEXT:    kmovd %esi, %k1
-; CHECK-NEXT:    kshiftlw $7, %k1, %k2
+; CHECK-NEXT:    kmovd %edi, %k1
+; CHECK-NEXT:    kmovd %esi, %k2
+; CHECK-NEXT:    kshiftlw $7, %k2, %k0
+; CHECK-NEXT:    kshiftrw $15, %k0, %k0
+; CHECK-NEXT:    kshiftlw $6, %k2, %k2
 ; CHECK-NEXT:    kshiftrw $15, %k2, %k2
-; CHECK-NEXT:    kmovd %k2, %eax
-; CHECK-NEXT:    kshiftlw $6, %k1, %k1
-; CHECK-NEXT:    kshiftrw $15, %k1, %k1
-; CHECK-NEXT:    kmovd %k1, %ecx
-; CHECK-NEXT:    vpmovm2q %k0, %zmm0
-; CHECK-NEXT:    kmovd %ecx, %k0
-; CHECK-NEXT:    vpmovm2q %k0, %zmm1
+; CHECK-NEXT:    vpmovm2q %k1, %zmm0
+; CHECK-NEXT:    vpmovm2q %k2, %zmm1
 ; CHECK-NEXT:    vmovdqa64 {{.*#+}} zmm2 = [0,1,2,3,4,5,8,7] sched: [5:0.50]
 ; CHECK-NEXT:    vpermi2q %zmm1, %zmm0, %zmm2
-; CHECK-NEXT:    vpmovq2m %zmm2, %k0
-; CHECK-NEXT:    kshiftlb $1, %k0, %k0
-; CHECK-NEXT:    kshiftrb $1, %k0, %k0
-; CHECK-NEXT:    kmovd %eax, %k1
-; CHECK-NEXT:    kshiftlb $7, %k1, %k1
-; CHECK-NEXT:    korb %k1, %k0, %k0
+; CHECK-NEXT:    vpmovq2m %zmm2, %k1
+; CHECK-NEXT:    kshiftlb $1, %k1, %k1
+; CHECK-NEXT:    kshiftrb $1, %k1, %k1
+; CHECK-NEXT:    kshiftlb $7, %k0, %k0
+; CHECK-NEXT:    korb %k0, %k1, %k0
 ; CHECK-NEXT:    vpmovm2w %k0, %xmm0
 ; CHECK-NEXT:    vzeroupper # sched: [4:1.00]
 ; CHECK-NEXT:    retq # sched: [2:1.00]
 ; SKX-LABEL: vmov_test18:
 ; SKX:       ## BB#0:
 ; SKX-NEXT:    kmovd %edi, %k0
 ; SKX-NEXT:    kmovd %esi, %k1
 ; SKX-NEXT:    kshiftlw $7, %k1, %k2

test/CodeGen/X86/avx512bw-intrinsics-fast-isel.ll

; X64-NEXT:    retq
   %2 = bitcast <64 x i8> %1 to <8 x i64>
   ret <8 x i64> %2
 }
 
 define <8 x i64> @test_mm512_mask_set1_epi16(<8 x i64> %__O, i32 %__M, i16 signext %__A)  {
 ; X32-LABEL: test_mm512_mask_set1_epi16:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %eax, %zmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_set1_epi16:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %esi, %zmm0 {%k1}
 ; X64-NEXT:    retq
   entry:
   %vecinit.i.i = insertelement <32 x i16> undef, i16 %__A, i32 0
   %vecinit31.i.i = shufflevector <32 x i16> %vecinit.i.i, <32 x i16> undef, <32 x i32> zeroinitializer
   %0 = bitcast <8 x i64> %__O to <32 x i16>
   %1 = bitcast i32 %__M to <32 x i1>
   %2 = select <32 x i1> %1, <32 x i16> %vecinit31.i.i, <32 x i16> %0
   %3 = bitcast <32 x i16> %2 to <8 x i64>
   ret <8 x i64> %3
 }
 
 define <8 x i64> @test_mm512_maskz_set1_epi16(i32 %__M, i16 signext %__A)  {
 ; X32-LABEL: test_mm512_maskz_set1_epi16:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %eax, %zmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_set1_epi16:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %esi, %zmm0 {%k1} {z}
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <8 x i16> %arg0, <8 x i16> undef, <32 x i32> zeroinitializer
   %res1 = bitcast <32 x i16> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_broadcastw_epi16(<8 x i64> %a0, i32 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm512_mask_broadcastw_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %xmm1, %zmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_broadcastw_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %xmm1, %zmm0 {%k1}
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <32 x i16>
   %arg1 = bitcast i32 %a1 to <32 x i1>
   %arg2 = bitcast <2 x i64> %a2 to <8 x i16>
   %res0 = shufflevector <8 x i16> %arg2, <8 x i16> undef, <32 x i32> zeroinitializer
   %res1 = select <32 x i1> %arg1, <32 x i16> %res0, <32 x i16> %arg0
   %res2 = bitcast <32 x i16> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_broadcastw_epi16(i32 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm512_maskz_broadcastw_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %xmm0, %zmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_broadcastw_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %xmm0, %zmm0 {%k1} {z}
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <32 x i16> %arg0, <32 x i16> %arg1, <32 x i32> <i32 4, i32 36, i32 5, i32 37, i32 6, i32 38, i32 7, i32 39, i32 12, i32 44, i32 13, i32 45, i32 14, i32 46, i32 15, i32 47, i32 20, i32 52, i32 21, i32 53, i32 22, i32 54, i32 23, i32 55, i32 28, i32 60, i32 29, i32 61, i32 30, i32 62, i32 31, i32 63>
   %res1 = bitcast <32 x i16> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_unpackhi_epi16(<8 x i64> %a0, i32 %a1, <8 x i64> %a2, <8 x i64> %a3) {
 ; X32-LABEL: test_mm512_mask_unpackhi_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpckhwd {{.*#+}} zmm0 {%k1} = zmm1[4],zmm2[4],zmm1[5],zmm2[5],zmm1[6],zmm2[6],zmm1[7],zmm2[7],zmm1[12],zmm2[12],zmm1[13],zmm2[13],zmm1[14],zmm2[14],zmm1[15],zmm2[15],zmm1[20],zmm2[20],zmm1[21],zmm2[21],zmm1[22],zmm2[22],zmm1[23],zmm2[23],zmm1[28],zmm2[28],zmm1[29],zmm2[29],zmm1[30],zmm2[30],zmm1[31],zmm2[31]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_unpackhi_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpunpckhwd {{.*#+}} zmm0 {%k1} = zmm1[4],zmm2[4],zmm1[5],zmm2[5],zmm1[6],zmm2[6],zmm1[7],zmm2[7],zmm1[12],zmm2[12],zmm1[13],zmm2[13],zmm1[14],zmm2[14],zmm1[15],zmm2[15],zmm1[20],zmm2[20],zmm1[21],zmm2[21],zmm1[22],zmm2[22],zmm1[23],zmm2[23],zmm1[28],zmm2[28],zmm1[29],zmm2[29],zmm1[30],zmm2[30],zmm1[31],zmm2[31]
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <32 x i16>
   %arg1 = bitcast i32 %a1 to <32 x i1>
   %arg2 = bitcast <8 x i64> %a2 to <32 x i16>
   %arg3 = bitcast <8 x i64> %a3 to <32 x i16>
   %res0 = shufflevector <32 x i16> %arg2, <32 x i16> %arg3, <32 x i32> <i32 4, i32 36, i32 5, i32 37, i32 6, i32 38, i32 7, i32 39, i32 12, i32 44, i32 13, i32 45, i32 14, i32 46, i32 15, i32 47, i32 20, i32 52, i32 21, i32 53, i32 22, i32 54, i32 23, i32 55, i32 28, i32 60, i32 29, i32 61, i32 30, i32 62, i32 31, i32 63>
   %res1 = select <32 x i1> %arg1, <32 x i16> %res0, <32 x i16> %arg0
   %res2 = bitcast <32 x i16> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_unpackhi_epi16(i32 %a0, <8 x i64> %a1, <8 x i64> %a2) {
 ; X32-LABEL: test_mm512_maskz_unpackhi_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpckhwd {{.*#+}} zmm0 {%k1} {z} = zmm0[4],zmm1[4],zmm0[5],zmm1[5],zmm0[6],zmm1[6],zmm0[7],zmm1[7],zmm0[12],zmm1[12],zmm0[13],zmm1[13],zmm0[14],zmm1[14],zmm0[15],zmm1[15],zmm0[20],zmm1[20],zmm0[21],zmm1[21],zmm0[22],zmm1[22],zmm0[23],zmm1[23],zmm0[28],zmm1[28],zmm0[29],zmm1[29],zmm0[30],zmm1[30],zmm0[31],zmm1[31]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_unpackhi_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpunpckhwd {{.*#+}} zmm0 {%k1} {z} = zmm0[4],zmm1[4],zmm0[5],zmm1[5],zmm0[6],zmm1[6],zmm0[7],zmm1[7],zmm0[12],zmm1[12],zmm0[13],zmm1[13],zmm0[14],zmm1[14],zmm0[15],zmm1[15],zmm0[20],zmm1[20],zmm0[21],zmm1[21],zmm0[22],zmm1[22],zmm0[23],zmm1[23],zmm0[28],zmm1[28],zmm0[29],zmm1[29],zmm0[30],zmm1[30],zmm0[31],zmm1[31]
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <32 x i16> %arg0, <32 x i16> %arg1, <32 x i32> <i32 0, i32 32, i32 1, i32 33, i32 2, i32 34, i32 3, i32 35, i32 8, i32 40, i32 9, i32 41, i32 10, i32 42, i32 11, i32 43, i32 16, i32 48, i32 17, i32 49, i32 18, i32 50, i32 19, i32 51, i32 24, i32 56, i32 25, i32 57, i32 26, i32 58, i32 27, i32 59>
   %res1 = bitcast <32 x i16> %res0 to <8 x i64>
   ret <8 x i64> %res1
 }
 
 define <8 x i64> @test_mm512_mask_unpacklo_epi16(<8 x i64> %a0, i32 %a1, <8 x i64> %a2, <8 x i64> %a3) {
 ; X32-LABEL: test_mm512_mask_unpacklo_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpcklwd {{.*#+}} zmm0 {%k1} = zmm1[0],zmm2[0],zmm1[1],zmm2[1],zmm1[2],zmm2[2],zmm1[3],zmm2[3],zmm1[8],zmm2[8],zmm1[9],zmm2[9],zmm1[10],zmm2[10],zmm1[11],zmm2[11],zmm1[16],zmm2[16],zmm1[17],zmm2[17],zmm1[18],zmm2[18],zmm1[19],zmm2[19],zmm1[24],zmm2[24],zmm1[25],zmm2[25],zmm1[26],zmm2[26],zmm1[27],zmm2[27]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_mask_unpacklo_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpunpcklwd {{.*#+}} zmm0 {%k1} = zmm1[0],zmm2[0],zmm1[1],zmm2[1],zmm1[2],zmm2[2],zmm1[3],zmm2[3],zmm1[8],zmm2[8],zmm1[9],zmm2[9],zmm1[10],zmm2[10],zmm1[11],zmm2[11],zmm1[16],zmm2[16],zmm1[17],zmm2[17],zmm1[18],zmm2[18],zmm1[19],zmm2[19],zmm1[24],zmm2[24],zmm1[25],zmm2[25],zmm1[26],zmm2[26],zmm1[27],zmm2[27]
 ; X64-NEXT:    retq
   %arg0 = bitcast <8 x i64> %a0 to <32 x i16>
   %arg1 = bitcast i32 %a1 to <32 x i1>
   %arg2 = bitcast <8 x i64> %a2 to <32 x i16>
   %arg3 = bitcast <8 x i64> %a3 to <32 x i16>
   %res0 = shufflevector <32 x i16> %arg2, <32 x i16> %arg3, <32 x i32> <i32 0, i32 32, i32 1, i32 33, i32 2, i32 34, i32 3, i32 35, i32 8, i32 40, i32 9, i32 41, i32 10, i32 42, i32 11, i32 43, i32 16, i32 48, i32 17, i32 49, i32 18, i32 50, i32 19, i32 51, i32 24, i32 56, i32 25, i32 57, i32 26, i32 58, i32 27, i32 59>
   %res1 = select <32 x i1> %arg1, <32 x i16> %res0, <32 x i16> %arg0
   %res2 = bitcast <32 x i16> %res1 to <8 x i64>
   ret <8 x i64> %res2
 }
 
 define <8 x i64> @test_mm512_maskz_unpacklo_epi16(i32 %a0, <8 x i64> %a1, <8 x i64> %a2) {
 ; X32-LABEL: test_mm512_maskz_unpacklo_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpunpcklwd {{.*#+}} zmm0 {%k1} {z} = zmm0[0],zmm1[0],zmm0[1],zmm1[1],zmm0[2],zmm1[2],zmm0[3],zmm1[3],zmm0[8],zmm1[8],zmm0[9],zmm1[9],zmm0[10],zmm1[10],zmm0[11],zmm1[11],zmm0[16],zmm1[16],zmm0[17],zmm1[17],zmm0[18],zmm1[18],zmm0[19],zmm1[19],zmm0[24],zmm1[24],zmm0[25],zmm1[25],zmm0[26],zmm1[26],zmm0[27],zmm1[27]
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm512_maskz_unpacklo_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpunpcklwd {{.*#+}} zmm0 {%k1} {z} = zmm0[0],zmm1[0],zmm0[1],zmm1[1],zmm0[2],zmm1[2],zmm0[3],zmm1[3],zmm0[8],zmm1[8],zmm0[9],zmm1[9],zmm0[10],zmm1[10],zmm0[11],zmm1[11],zmm0[16],zmm1[16],zmm0[17],zmm1[17],zmm0[18],zmm1[18],zmm0[19],zmm1[19],zmm0[24],zmm1[24],zmm0[25],zmm1[25],zmm0[26],zmm1[26],zmm0[27],zmm1[27]
 ; X64-NEXT:    retq

test/CodeGen/X86/avx512bwvl-intrinsics-fast-isel.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -fast-isel -mtriple=i386-unknown-unknown -mattr=+avx512f,+avx512bw,+avx512vl | FileCheck %s --check-prefix=ALL --check-prefix=X32
 ; RUN: llc < %s -fast-isel -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512bw,+avx512vl | FileCheck %s --check-prefix=ALL --check-prefix=X64
 
 ; NOTE: This should use IR equivalent to what is generated by clang/test/CodeGen/avx512vlbw-builtins.c
 
 define <2 x i64> @test_mm_mask_set1_epi8(<2 x i64> %__O, i16 zeroext %__M, i8 signext %__A) local_unnamed_addr #0 {
 ; X32-LABEL: test_mm_mask_set1_epi8:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %eax, %xmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_set1_epi8:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %esi, %xmm0 {%k1}
 ; X64-NEXT:    retq
 entry:
   %vecinit.i.i = insertelement <16 x i8> undef, i8 %__A, i32 0
   %vecinit15.i.i = shufflevector <16 x i8> %vecinit.i.i, <16 x i8> undef, <16 x i32> zeroinitializer
   %0 = bitcast <2 x i64> %__O to <16 x i8>
   %1 = bitcast i16 %__M to <16 x i1>
   %2 = select <16 x i1> %1, <16 x i8> %vecinit15.i.i, <16 x i8> %0
   %3 = bitcast <16 x i8> %2 to <2 x i64>
   ret <2 x i64> %3
 }
 
 define <2 x i64> @test_mm_maskz_set1_epi8(i16 zeroext %__M, i8 signext %__A)  {
 ; X32-LABEL: test_mm_maskz_set1_epi8:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %eax, %xmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_set1_epi8:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %esi, %xmm0 {%k1} {z}
 ; X64-NEXT:    retq
 entry:
   %vecinit.i.i = insertelement <16 x i8> undef, i8 %__A, i32 0
   %vecinit15.i.i = shufflevector <16 x i8> %vecinit.i.i, <16 x i8> undef, <16 x i32> zeroinitializer
   %0 = bitcast i16 %__M to <16 x i1>
   %1 = select <16 x i1> %0, <16 x i8> %vecinit15.i.i, <16 x i8> zeroinitializer
   %2 = bitcast <16 x i8> %1 to <2 x i64>
   ret <2 x i64> %2
 }
 
 define <4 x i64> @test_mm256_mask_set1_epi8(<4 x i64> %__O, i32 %__M, i8 signext %__A){
 ; X32-LABEL: test_mm256_mask_set1_epi8:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %eax, %ymm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_set1_epi8:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %esi, %ymm0 {%k1}
 ; X64-NEXT:    retq
 entry:
   %vecinit.i.i = insertelement <32 x i8> undef, i8 %__A, i32 0
   %vecinit31.i.i = shufflevector <32 x i8> %vecinit.i.i, <32 x i8> undef, <32 x i32> zeroinitializer
   %0 = bitcast <4 x i64> %__O to <32 x i8>
   %1 = bitcast i32 %__M to <32 x i1>
   %2 = select <32 x i1> %1, <32 x i8> %vecinit31.i.i, <32 x i8> %0
   %3 = bitcast <32 x i8> %2 to <4 x i64>
   ret <4 x i64> %3
 }
 
 define <4 x i64> @test_mm256_maskz_set1_epi8(i32 %__M, i8 signext %__A)  {
 ; X32-LABEL: test_mm256_maskz_set1_epi8:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movb {{[0-9]+}}(%esp), %al
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %eax, %ymm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_set1_epi8:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %esi, %ymm0 {%k1} {z}
 ; X64-NEXT:    retq
 entry:
   %vecinit.i.i = insertelement <32 x i8> undef, i8 %__A, i32 0
   %vecinit31.i.i = shufflevector <32 x i8> %vecinit.i.i, <32 x i8> undef, <32 x i32> zeroinitializer
   %0 = bitcast i32 %__M to <32 x i1>
   %1 = select <32 x i1> %0, <32 x i8> %vecinit31.i.i, <32 x i8> zeroinitializer
   %2 = bitcast <32 x i8> %1 to <4 x i64>
   ret <4 x i64> %2
 }
 
 define <4 x i64> @test_mm256_mask_set1_epi16(<4 x i64> %__O, i16 zeroext %__M, i16 signext %__A)  {
 ; X32-LABEL: test_mm256_mask_set1_epi16:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %eax, %ymm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_set1_epi16:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %esi, %ymm0 {%k1}
 ; X64-NEXT:    retq
 entry:
   %vecinit.i.i = insertelement <16 x i16> undef, i16 %__A, i32 0
   %vecinit15.i.i = shufflevector <16 x i16> %vecinit.i.i, <16 x i16> undef, <16 x i32> zeroinitializer
   %0 = bitcast <4 x i64> %__O to <16 x i16>
   %1 = bitcast i16 %__M to <16 x i1>
   %2 = select <16 x i1> %1, <16 x i16> %vecinit15.i.i, <16 x i16> %0
   %3 = bitcast <16 x i16> %2 to <4 x i64>
   ret <4 x i64> %3
 }
 
 define <4 x i64> @test_mm256_maskz_set1_epi16(i16 zeroext %__M, i16 signext %__A) {
 ; X32-LABEL: test_mm256_maskz_set1_epi16:
 ; X32:       # BB#0: # %entry
 ; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    kmovd %ecx, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %eax, %ymm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_set1_epi16:
 ; X64:       # BB#0: # %entry
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %esi, %ymm0 {%k1} {z}
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <16 x i8> %arg0, <16 x i8> undef, <16 x i32> zeroinitializer
   %res1 = bitcast <16 x i8> %res0 to <2 x i64>
   ret <2 x i64> %res1
 }
 
 define <2 x i64> @test_mm_mask_broadcastb_epi8(<2 x i64> %a0, i16 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm_mask_broadcastb_epi8:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %xmm1, %xmm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_mask_broadcastb_epi8:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %xmm1, %xmm0 {%k1}
 ; X64-NEXT:    retq
   %arg0 = bitcast <2 x i64> %a0 to <16 x i8>
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %arg2 = bitcast <2 x i64> %a2 to <16 x i8>
   %res0 = shufflevector <16 x i8> %arg2, <16 x i8> undef, <16 x i32> zeroinitializer
   %res1 = select <16 x i1> %arg1, <16 x i8> %res0, <16 x i8> %arg0
   %res2 = bitcast <16 x i8> %res1 to <2 x i64>
   ret <2 x i64> %res2
 }
 
 define <2 x i64> @test_mm_maskz_broadcastb_epi8(i16 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm_maskz_broadcastb_epi8:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %xmm0, %xmm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm_maskz_broadcastb_epi8:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %xmm0, %xmm0 {%k1} {z}
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <16 x i8> %arg0, <16 x i8> undef, <32 x i32> zeroinitializer
   %res1 = bitcast <32 x i8> %res0 to <4 x i64>
   ret <4 x i64> %res1
 }
 
 define <4 x i64> @test_mm256_mask_broadcastb_epi8(<4 x i64> %a0, i32 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm256_mask_broadcastb_epi8:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %xmm1, %ymm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_broadcastb_epi8:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %xmm1, %ymm0 {%k1}
 ; X64-NEXT:    retq
   %arg0 = bitcast <4 x i64> %a0 to <32 x i8>
   %arg1 = bitcast i32 %a1 to <32 x i1>
   %arg2 = bitcast <2 x i64> %a2 to <16 x i8>
   %res0 = shufflevector <16 x i8> %arg2, <16 x i8> undef, <32 x i32> zeroinitializer
   %res1 = select <32 x i1> %arg1, <32 x i8> %res0, <32 x i8> %arg0
   %res2 = bitcast <32 x i8> %res1 to <4 x i64>
   ret <4 x i64> %res2
 }
 
 define <4 x i64> @test_mm256_maskz_broadcastb_epi8(i32 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm256_maskz_broadcastb_epi8:
 ; X32:       # BB#0:
-; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovd {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastb %xmm0, %ymm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_broadcastb_epi8:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastb %xmm0, %ymm0 {%k1} {z}
 ; X64-NEXT:    retq
; X64-NEXT:    retq
   %res0 = shufflevector <8 x i16> %arg0, <8 x i16> undef, <16 x i32> zeroinitializer
   %res1 = bitcast <16 x i16> %res0 to <4 x i64>
   ret <4 x i64> %res1
 }
 
 define <4 x i64> @test_mm256_mask_broadcastw_epi16(<4 x i64> %a0, i16 %a1, <2 x i64> %a2) {
 ; X32-LABEL: test_mm256_mask_broadcastw_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %xmm1, %ymm0 {%k1}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_mask_broadcastw_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %xmm1, %ymm0 {%k1}
 ; X64-NEXT:    retq
   %arg0 = bitcast <4 x i64> %a0 to <16 x i16>
   %arg1 = bitcast i16 %a1 to <16 x i1>
   %arg2 = bitcast <2 x i64> %a2 to <8 x i16>
   %res0 = shufflevector <8 x i16> %arg2, <8 x i16> undef, <16 x i32> zeroinitializer
   %res1 = select <16 x i1> %arg1, <16 x i16> %res0, <16 x i16> %arg0
   %res2 = bitcast <16 x i16> %res1 to <4 x i64>
   ret <4 x i64> %res2
 }
 
 define <4 x i64> @test_mm256_maskz_broadcastw_epi16(i16 %a0, <2 x i64> %a1) {
 ; X32-LABEL: test_mm256_maskz_broadcastw_epi16:
 ; X32:       # BB#0:
-; X32-NEXT:    movzwl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    kmovd %eax, %k1
+; X32-NEXT:    kmovw {{[0-9]+}}(%esp), %k1
 ; X32-NEXT:    vpbroadcastw %xmm0, %ymm0 {%k1} {z}
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test_mm256_maskz_broadcastw_epi16:
 ; X64:       # BB#0:
 ; X64-NEXT:    kmovd %edi, %k1
 ; X64-NEXT:    vpbroadcastw %xmm0, %ymm0 {%k1} {z}
 ; X64-NEXT:    retq

test/CodeGen/X86/domain-reassignment.mir

+# RUN: llc -run-pass x86-domain-reassignment -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512bw,+avx512dq -o - %s | FileCheck %s
+--- |
+  ; ModuleID = '../test/CodeGen/X86/gpr-to-mask.ll'
+  source_filename = "../test/CodeGen/X86/gpr-to-mask.ll"
+  target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+  target triple = "x86_64-unknown-unknown"
+  
+  define void @test_fcmp_storefloat(i1 %cond, float* %fptr, float %f1, float %f2, float %f3, float %f4, float %f5, float %f6) #0 {
+  entry:
+    br i1 %cond, label %if, label %else
+  
+  if:                                               ; preds = %entry
+    %cmp1 = fcmp oeq float %f3, %f4
+    br label %exit
+  
+  else:                                             ; preds = %entry
+    %cmp2 = fcmp oeq float %f5, %f6
+    br label %exit
+  
+  exit:                                             ; preds = %else, %if
+    %val = phi i1 [ %cmp1, %if ], [ %cmp2, %else ]
+    %selected = select i1 %val, float %f1, float %f2
+    store float %selected, float* %fptr
+    ret void
+  }
+
+  define void @test_8bitops() #0 {
+    ret void
+  }
+  define void @test_16bitops() #0 {
+    ret void
+  }
+  define void @test_32bitops() #0 {
+    ret void
+  }
+  define void @test_64bitops() #0 {
+    ret void
+  }
+  define void @test_16bitext() #0 {
+    ret void
+  }
+  define void @test_32bitext() #0 {
+    ret void
+  }
+  define void @test_64bitext() #0 {
+    ret void
+  }
+...
+---
+name:            test_fcmp_storefloat
+# CHECK-LABEL: name: test_fcmp_storefloat
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  # CHECK: - { id: 0, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 1, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 2, class: vk8, preferred-register: '' }
+  - { id: 0, class: gr8, preferred-register: '' }
+  - { id: 1, class: gr8, preferred-register: '' }
+  - { id: 2, class: gr8, preferred-register: '' }
+  - { id: 3, class: gr32, preferred-register: '' }
+  - { id: 4, class: gr64, preferred-register: '' }
+  - { id: 5, class: fr32x, preferred-register: '' }
+  - { id: 6, class: fr32x, preferred-register: '' }
+  - { id: 7, class: fr32x, preferred-register: '' }
+  - { id: 8, class: fr32x, preferred-register: '' }
+  - { id: 9, class: fr32x, preferred-register: '' }
+  - { id: 10, class: fr32x, preferred-register: '' }
+  - { id: 11, class: gr8, preferred-register: '' }
+  - { id: 12, class: vk1, preferred-register: '' }
+  # CHECK: - { id: 13, class: vk32, preferred-register: '' }
+  - { id: 13, class: gr32, preferred-register: '' }
+  - { id: 14, class: vk1, preferred-register: '' }
+  # CHECK: - { id: 15, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 16, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 17, class: vk32, preferred-register: '' }
+  - { id: 15, class: gr32, preferred-register: '' }
+  - { id: 16, class: gr32, preferred-register: '' }
+  - { id: 17, class: gr32, preferred-register: '' }
+  - { id: 18, class: vk1wm, preferred-register: '' }
+  - { id: 19, class: vr128x, preferred-register: '' }
+  - { id: 20, class: fr128, preferred-register: '' }
+  - { id: 21, class: fr128, preferred-register: '' }
+  - { id: 22, class: fr32x, preferred-register: '' }
+liveins:         
+  - { reg: '%edi', virtual-reg: '%3' }
+  - { reg: '%rsi', virtual-reg: '%4' }
+  - { reg: '%xmm0', virtual-reg: '%5' }
+  - { reg: '%xmm1', virtual-reg: '%6' }
+  - { reg: '%xmm2', virtual-reg: '%7' }
+  - { reg: '%xmm3', virtual-reg: '%8' }
+  - { reg: '%xmm4', virtual-reg: '%9' }
+  - { reg: '%xmm5', virtual-reg: '%10' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0.entry:
+    successors: %bb.1.if(0x40000000), %bb.2.else(0x40000000)
+    liveins: %edi, %rsi, %xmm0, %xmm1, %xmm2, %xmm3, %xmm4, %xmm5
+  
+    %10 = COPY %xmm5
+    %9 = COPY %xmm4
+    %8 = COPY %xmm3
+    %7 = COPY %xmm2
+    %6 = COPY %xmm1
+    %5 = COPY %xmm0
+    %4 = COPY %rsi
+    %3 = COPY %edi
+    %11 = COPY %3.sub_8bit
+    TEST8ri killed %11, 1, implicit-def %eflags
+    JE_1 %bb.2.else, implicit %eflags
+    JMP_1 %bb.1.if
+  
+  bb.1.if:
+    successors: %bb.3.exit(0x80000000)
+  
+    %14 = VCMPSSZrr %7, %8, 0
+
+    ; check that cross domain copies are replaced with same domain copies.
+    ; CHECK: %15 = COPY %14
+    ; CHECK: %0 = COPY %15
+    
+    %15 = COPY %14
+    %0 = COPY %15.sub_8bit
+    JMP_1 %bb.3.exit
+  
+  bb.2.else:
+    successors: %bb.3.exit(0x80000000)
+    %12 = VCMPSSZrr %9, %10, 0
+
+    ; check that cross domain copies are replaced with same domain copies.
+    ; CHECK: %13 = COPY %12
+    ; CHECK: %1 = COPY %13
+
+    %13 = COPY %12
+    %1 = COPY %13.sub_8bit
+  
+  bb.3.exit:
+
+    ; check PHI, IMPLICIT_DEF, and INSERT_SUBREG replacers.
+    ; CHECK: %2 = PHI %1, %bb.2.else, %0, %bb.1.if
+    ; CHECK: %16 = COPY %2
+    ; CHECK: %18 = COPY %16
+  
+    %2 = PHI %1, %bb.2.else, %0, %bb.1.if
+    %17 = IMPLICIT_DEF
+    %16 = INSERT_SUBREG %17, %2, 1
+    %18 = COPY %16
+    %19 = COPY %6
+    %21 = IMPLICIT_DEF
+    %20 = VMOVSSZrrk %19, killed %18, killed %21, %5
+    %22 = COPY %20
+    VMOVSSZmr %4, 1, _, 0, _, killed %22 :: (store 4 into %ir.fptr)
+    RET 0
+
+...
+---
+name:            test_8bitops
+# CHECK-LABEL: name: test_8bitops
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vr512, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  - { id: 5, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 8, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 9, class: vk32, preferred-register: '' }
+  - { id: 6, class: gr32, preferred-register: '' }
+  - { id: 7, class: gr8, preferred-register: '' }
+  - { id: 8, class: gr32, preferred-register: '' }
+  - { id: 9, class: gr32, preferred-register: '' }
+  - { id: 10, class: vk8wm, preferred-register: '' }
+  - { id: 11, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 12, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 13, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 14, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 15, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 16, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 17, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 18, class: vk8, preferred-register: '' }
+  - { id: 12, class: gr8, preferred-register: '' }
+  - { id: 13, class: gr8, preferred-register: '' }
+  - { id: 14, class: gr8, preferred-register: '' }
+  - { id: 15, class: gr8, preferred-register: '' }
+  - { id: 16, class: gr8, preferred-register: '' }
+  - { id: 17, class: gr8, preferred-register: '' }
+  - { id: 18, class: gr8, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+  - { reg: '%zmm2', virtual-reg: '%3' }
+  - { reg: '%zmm3', virtual-reg: '%4' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1, %zmm2, %zmm3
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+    %3 = COPY %zmm2
+    %4 = COPY %zmm3
+  
+    %5 = VCMPPDZrri %3, %4, 0
+    ; CHECK: %6 = COPY %5
+    ; CHECK: %7 = COPY %6
+    %6 = COPY %5
+    %7 = COPY %6.sub_8bit
+
+    ; CHECK: %12 = KSHIFTRBri %7, 2
+    ; CHECK: %13 = KSHIFTLBri %12, 1
+    ; CHECK: %14 = KNOTBrr %13
+    ; CHECK: %15 = KORBrr %14, %12
+    ; CHECK: %16 = KANDBrr %15, %13
+    ; CHECK: %17 = KXORBrr %16, %12
+    ; CHECK: %18 = KADDBrr %17, %14
+    %12 = SHR8ri %7, 2, implicit-def dead %eflags
+    %13 = SHL8ri %12, 1, implicit-def dead %eflags
+    %14 = NOT8r %13
+    %15 = OR8rr %14, %12, implicit-def dead %eflags
+    %16 = AND8rr %15, %13, implicit-def dead %eflags
+    %17 = XOR8rr %16, %12, implicit-def dead %eflags
+    %18 = ADD8rr %17, %14, implicit-def dead %eflags
+  
+    ; CHECK: %9 = COPY %18
+    ; CHECK: %10 = COPY %9
+    %8 = IMPLICIT_DEF
+    %9 = INSERT_SUBREG %8, %18, 1
+    %10 = COPY %9
+    %11 = VMOVAPDZrrk %2, killed %10, %1
+    VMOVAPDZmr %0, 1, _, 0, _, killed %11 
+
+    ; CHECK: KTESTBrr %18, %18, implicit-def %eflags
+    TEST8rr %18, %18, implicit-def %eflags
+    JE_1 %bb.1, implicit %eflags
+    JMP_1 %bb.2
+
+  bb.1:
+
+  bb.2:
+    RET 0
+
+...
+---
+name:            test_16bitops
+# CHECK-LABEL: name: test_16bitops
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vr512, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  - { id: 5, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 8, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 9, class: vk32, preferred-register: '' }
+  - { id: 6, class: gr32, preferred-register: '' }
+  - { id: 7, class: gr16, preferred-register: '' }
+  - { id: 8, class: gr32, preferred-register: '' }
+  - { id: 9, class: gr32, preferred-register: '' }
+  - { id: 10, class: vk16wm, preferred-register: '' }
+  - { id: 11, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 12, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 13, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 14, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 15, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 16, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 17, class: vk16, preferred-register: '' }
+  - { id: 12, class: gr16, preferred-register: '' }
+  - { id: 13, class: gr16, preferred-register: '' }
+  - { id: 14, class: gr16, preferred-register: '' }
+  - { id: 15, class: gr16, preferred-register: '' }
+  - { id: 16, class: gr16, preferred-register: '' }
+  - { id: 17, class: gr16, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+  - { reg: '%zmm2', virtual-reg: '%3' }
+  - { reg: '%zmm3', virtual-reg: '%4' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1, %zmm2, %zmm3
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+    %3 = COPY %zmm2
+    %4 = COPY %zmm3
+  
+    %5 = VCMPPSZrri %3, %4, 0
+    ; CHECK: %6 = COPY %5
+    ; CHECK: %7 = COPY %6
+    %6 = COPY %5
+    %7 = COPY %6.sub_16bit
+
+    ; CHECK: %12 = KSHIFTRWri %7, 2
+    ; CHECK: %13 = KSHIFTLWri %12, 1
+    ; CHECK: %14 = KNOTWrr %13
+    ; CHECK: %15 = KORWrr %14, %12
+    ; CHECK: %16 = KANDWrr %15, %13
+    ; CHECK: %17 = KXORWrr %16, %12
+    %12 = SHR16ri %7, 2, implicit-def dead %eflags
+    %13 = SHL16ri %12, 1, implicit-def dead %eflags
+    %14 = NOT16r %13
+    %15 = OR16rr %14, %12, implicit-def dead %eflags
+    %16 = AND16rr %15, %13, implicit-def dead %eflags
+    %17 = XOR16rr %16, %12, implicit-def dead %eflags
+  
+    ; CHECK: %9 = COPY %17
+    ; CHECK: %10 = COPY %9
+    %8 = IMPLICIT_DEF
+    %9 = INSERT_SUBREG %8, %17, 3
+    %10 = COPY %9
+    %11 = VMOVAPSZrrk %2, killed %10, %1
+    VMOVAPSZmr %0, 1, _, 0, _, killed %11 
+
+    ; CHECK: KTESTWrr %17, %17, implicit-def %eflags
+    TEST16rr %17, %17, implicit-def %eflags
+    JE_1 %bb.1, implicit %eflags
+    JMP_1 %bb.2
+
+  bb.1:
+
+  bb.2:
+    RET 0
+
+...
+---
+name:            test_32bitops
+# CHECK-LABEL: name: test_32bitops
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vk32wm, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 5, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 8, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 9, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 10, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 11, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 12, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 13, class: vk32, preferred-register: '' }
+  - { id: 5, class: gr32, preferred-register: '' }
+  - { id: 6, class: gr32, preferred-register: '' }
+  - { id: 7, class: gr32, preferred-register: '' }
+  - { id: 8, class: gr32, preferred-register: '' }
+  - { id: 9, class: gr32, preferred-register: '' }
+  - { id: 10, class: gr32, preferred-register: '' }
+  - { id: 11, class: gr32, preferred-register: '' }
+  - { id: 12, class: gr32, preferred-register: '' }
+  - { id: 13, class: gr32, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+  
+    ; CHECK: %5 = KMOVDkm %0, 1, _, 0, _
+    ; CHECK: %6 = KSHIFTRDri %5, 2
+    ; CHECK: %7 = KSHIFTLDri %6, 1
+    ; CHECK: %8 = KNOTDrr %7
+    ; CHECK: %9 = KORDrr %8, %6
+    ; CHECK: %10 = KANDDrr %9, %7
+    ; CHECK: %11 = KXORDrr %10, %6
+    ; CHECK: %12 = KANDNDrr %11, %9
+    ; CHECK: %13 = KADDDrr %12, %11
+    %5 = MOV32rm %0, 1, _, 0, _
+    %6 = SHR32ri %5, 2, implicit-def dead %eflags
+    %7 = SHL32ri %6, 1, implicit-def dead %eflags
+    %8 = NOT32r %7
+    %9 = OR32rr %8, %6, implicit-def dead %eflags
+    %10 = AND32rr %9, %7, implicit-def dead %eflags
+    %11 = XOR32rr %10, %6, implicit-def dead %eflags
+    %12 = ANDN32rr %11, %9, implicit-def dead %eflags
+    %13 = ADD32rr %12, %11, implicit-def dead %eflags
+  
+    ; CHECK: %3 = COPY %13
+    %3 = COPY %13
+    %4 = VMOVDQU16Zrrk %2, killed %3, %1
+    VMOVDQA32Zmr %0, 1, _, 0, _, killed %4
+
+    ; CHECK: KTESTDrr %13, %13, implicit-def %eflags
+    TEST32rr %13, %13, implicit-def %eflags
+    JE_1 %bb.1, implicit %eflags
+    JMP_1 %bb.2
+
+  bb.1:
+
+  bb.2:
+    RET 0
+
+...
+---
+name:            test_64bitops
+# CHECK-LABEL: name: test_64bitops
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vk64wm, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 5, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 8, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 9, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 10, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 11, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 12, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 13, class: vk64, preferred-register: '' }
+  - { id: 5, class: gr64, preferred-register: '' }
+  - { id: 6, class: gr64, preferred-register: '' }
+  - { id: 7, class: gr64, preferred-register: '' }
+  - { id: 8, class: gr64, preferred-register: '' }
+  - { id: 9, class: gr64, preferred-register: '' }
+  - { id: 10, class: gr64, preferred-register: '' }
+  - { id: 11, class: gr64, preferred-register: '' }
+  - { id: 12, class: gr64, preferred-register: '' }
+  - { id: 13, class: gr64, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+  
+    ; CHECK: %5 = KMOVQkm %0, 1, _, 0, _
+    ; CHECK: %6 = KSHIFTRQri %5, 2
+    ; CHECK: %7 = KSHIFTLQri %6, 1
+    ; CHECK: %8 = KNOTQrr %7
+    ; CHECK: %9 = KORQrr %8, %6
+    ; CHECK: %10 = KANDQrr %9, %7
+    ; CHECK: %11 = KXORQrr %10, %6
+    ; CHECK: %12 = KANDNQrr %11, %9
+    ; CHECK: %13 = KADDQrr %12, %11
+    %5 = MOV64rm %0, 1, _, 0, _
+    %6 = SHR64ri %5, 2, implicit-def dead %eflags
+    %7 = SHL64ri %6, 1, implicit-def dead %eflags
+    %8 = NOT64r %7
+    %9 = OR64rr %8, %6, implicit-def dead %eflags
+    %10 = AND64rr %9, %7, implicit-def dead %eflags
+    %11 = XOR64rr %10, %6, implicit-def dead %eflags
+    %12 = ANDN64rr %11, %9, implicit-def dead %eflags
+    %13 = ADD64rr %12, %11, implicit-def dead %eflags
+  
+    ; CHECK: %3 = COPY %13
+    %3 = COPY %13
+    %4 = VMOVDQU8Zrrk %2, killed %3, %1
+    VMOVDQA32Zmr %0, 1, _, 0, _, killed %4
+
+    ; CHECK: KTESTQrr %13, %13, implicit-def %eflags
+    TEST64rr %13, %13, implicit-def %eflags
+    JE_1 %bb.1, implicit %eflags
+    JMP_1 %bb.2
+
+  bb.1:
+
+  bb.2:
+    RET 0
+
+...
+---
+name:            test_16bitext
+# CHECK-LABEL: name: test_16bitext
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vk16wm, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 5, class: vk16, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk16, preferred-register: '' }
+  - { id: 5, class: gr16, preferred-register: '' }
+  - { id: 6, class: gr16, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk8, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+  
+    ; CHECK: %7 = KMOVBkm %0, 1, _, 0, _
+    ; CHECK: %5 = COPY %7
+    ; CHECK: %6 = KNOTWrr %5
+    %5 = MOVZX16rm8 %0, 1, _, 0, _
+    %6 = NOT16r %5
+
+    ; CHECK: %3 = COPY %6
+    %3 = COPY %6
+    %4 = VMOVAPSZrrk %2, killed %3, %1
+    VMOVAPSZmr %0, 1, _, 0, _, killed %4
+    RET 0
+
+...
+---
+name:            test_32bitext
+# CHECK-LABEL: name: test_32bitext
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vk64wm, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 5, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk32, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk32, preferred-register: '' }
+  - { id: 5, class: gr32, preferred-register: '' }
+  - { id: 6, class: gr32, preferred-register: '' }
+  - { id: 7, class: gr32, preferred-register: '' }
+  # CHECK: - { id: 8, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 9, class: vk16, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+  
+    ; CHECK: %8 = KMOVBkm %0, 1, _, 0, _
+    ; CHECK: %5 = COPY %8
+    ; CHECK: %9 = KMOVWkm %0, 1, _, 0, _
+    ; CHECK: %6 = COPY %9
+    ; CHECK: %7 = KADDDrr %5, %6
+    %5 = MOVZX32rm8 %0, 1, _, 0, _
+    %6 = MOVZX32rm16 %0, 1, _, 0, _
+    %7 = ADD32rr %5, %6, implicit-def dead %eflags
+
+    ; CHECK: %3 = COPY %7
+    %3 = COPY %7
+    %4 = VMOVDQU16Zrrk %2, killed %3, %1
+    VMOVDQA32Zmr %0, 1, _, 0, _, killed %4
+    RET 0
+
+...
+---
+name:            test_64bitext
+# CHECK-LABEL: name: test_64bitext
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+registers:       
+  - { id: 0, class: gr64, preferred-register: '' }
+  - { id: 1, class: vr512, preferred-register: '' }
+  - { id: 2, class: vr512, preferred-register: '' }
+  - { id: 3, class: vk64wm, preferred-register: '' }
+  - { id: 4, class: vr512, preferred-register: '' }
+  # CHECK: - { id: 5, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 6, class: vk64, preferred-register: '' }
+  # CHECK: - { id: 7, class: vk64, preferred-register: '' }
+  - { id: 5, class: gr64, preferred-register: '' }
+  - { id: 6, class: gr64, preferred-register: '' }
+  - { id: 7, class: gr64, preferred-register: '' }
+  # CHECK: - { id: 8, class: vk8, preferred-register: '' }
+  # CHECK: - { id: 9, class: vk16, preferred-register: '' }
+liveins:         
+  - { reg: '%rdi', virtual-reg: '%0' }
+  - { reg: '%zmm0', virtual-reg: '%1' }
+  - { reg: '%zmm1', virtual-reg: '%2' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    0
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      
+stack:           
+constants:       
+body:             |
+  bb.0:
+    liveins: %rdi, %zmm0, %zmm1
+  
+    %0 = COPY %rdi
+    %1 = COPY %zmm0
+    %2 = COPY %zmm1
+  
+    ; CHECK: %8 = KMOVBkm %0, 1, _, 0, _
+    ; CHECK: %5 = COPY %8
+    ; CHECK: %9 = KMOVWkm %0, 1, _, 0, _
+    ; CHECK: %6 = COPY %9
+    ; CHECK: %7 = KADDQrr %5, %6
+    %5 = MOVZX64rm8 %0, 1, _, 0, _
+    %6 = MOVZX64rm16 %0, 1, _, 0, _
+    %7 = ADD64rr %5, %6, implicit-def dead %eflags
+
+    ; CHECK: %3 = COPY %7
+    %3 = COPY %7
+    %4 = VMOVDQU8Zrrk %2, killed %3, %1
+    VMOVDQA32Zmr %0, 1, _, 0, _, killed %4
+    RET 0
+
+...

test/CodeGen/X86/gpr-to-mask.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512bw < %s | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=+avx512f,+avx512bw,+avx512dq < %s | FileCheck %s
 
 define void @test_fcmp_storefloat(i1 %cond, float* %fptr, float %f1, float %f2, float %f3, float %f4, float %f5, float %f6) {
 ; CHECK-LABEL: test_fcmp_storefloat:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB0_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    vcmpeqss %xmm3, %xmm2, %k0
+; CHECK-NEXT:    vcmpeqss %xmm3, %xmm2, %k1
 ; CHECK-NEXT:    jmp .LBB0_3
 ; CHECK-NEXT:  .LBB0_2: # %else
-; CHECK-NEXT:    vcmpeqss %xmm5, %xmm4, %k0
+; CHECK-NEXT:    vcmpeqss %xmm5, %xmm4, %k1
 ; CHECK-NEXT:  .LBB0_3: # %exit
-; CHECK-NEXT:    kmovd %k0, %eax
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovss %xmm0, %xmm0, %xmm1 {%k1}
 ; CHECK-NEXT:    vmovss %xmm1, (%rsi)
 ; CHECK-NEXT:    retq
 entry:
   br i1 %cond, label %if, label %else
 
 if:
   %cmp1 = fcmp oeq float %f3, %f4

 }
 
 define void @test_load_add(i1 %cond, float* %fptr, i1* %iptr1, i1* %iptr2, float %f1, float %f2)  {
 ; CHECK-LABEL: test_load_add:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB2_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    movb (%rcx), %al
-; CHECK-NEXT:    addb (%rdx), %al
+; CHECK-NEXT:    kmovb (%rdx), %k0
+; CHECK-NEXT:    kmovb (%rcx), %k1
+; CHECK-NEXT:    kaddb %k1, %k0, %k1
 ; CHECK-NEXT:    jmp .LBB2_3
 ; CHECK-NEXT:  .LBB2_2: # %else
-; CHECK-NEXT:    movb (%rcx), %al
+; CHECK-NEXT:    kmovb (%rcx), %k1
 ; CHECK-NEXT:  .LBB2_3: # %exit
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovss %xmm0, %xmm0, %xmm1 {%k1}
 ; CHECK-NEXT:    vmovss %xmm1, (%rsi)
 ; CHECK-NEXT:    retq
 entry:
   br i1 %cond, label %if, label %else
 
 if:
   %loaded1 = load i1, i1* %iptr1

 }
 
 define void @test_load_i1(i1 %cond, float* %fptr, i1* %iptr1, i1* %iptr2, float %f1, float %f2)  {
 ; CHECK-LABEL: test_load_i1:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB3_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    movb (%rdx), %al
+; CHECK-NEXT:    kmovb (%rdx), %k1
 ; CHECK-NEXT:    jmp .LBB3_3
 ; CHECK-NEXT:  .LBB3_2: # %else
-; CHECK-NEXT:    movb (%rcx), %al
+; CHECK-NEXT:    kmovb (%rcx), %k1
 ; CHECK-NEXT:  .LBB3_3: # %exit
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovss %xmm0, %xmm0, %xmm1 {%k1}
 ; CHECK-NEXT:    vmovss %xmm1, (%rsi)
 ; CHECK-NEXT:    retq
 entry:
   br i1 %cond, label %if, label %else
 
 if:
   %loaded1 = load i1, i1* %iptr1

 define void @test_shl1(i1 %cond, i8* %ptr1, i8* %ptr2, <8 x float> %fvec1, <8 x float> %fvec2, <8 x float>* %fptrvec) {
 ; CHECK-LABEL: test_shl1:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    # kill: %YMM1<def> %YMM1<kill> %ZMM1<def>
 ; CHECK-NEXT:    # kill: %YMM0<def> %YMM0<kill> %ZMM0<def>
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB5_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    movb (%rsi), %al
-; CHECK-NEXT:    addb %al, %al
+; CHECK-NEXT:    kmovb (%rsi), %k0
+; CHECK-NEXT:    kaddb %k0, %k0, %k1
 ; CHECK-NEXT:    jmp .LBB5_3
 ; CHECK-NEXT:  .LBB5_2: # %else
-; CHECK-NEXT:    movb (%rdx), %al
+; CHECK-NEXT:    kmovb (%rdx), %k1
 ; CHECK-NEXT:  .LBB5_3: # %exit
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovaps %zmm0, %zmm1 {%k1}
 ; CHECK-NEXT:    vmovaps %ymm1, (%rcx)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   br i1 %cond, label %if, label %else
 
 if:

 define void @test_shr2(i1 %cond, i8* %ptr1, i8* %ptr2, <8 x float> %fvec1, <8 x float> %fvec2, <8 x float>* %fptrvec) {
 ; CHECK-LABEL: test_shr2:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    # kill: %YMM1<def> %YMM1<kill> %ZMM1<def>
 ; CHECK-NEXT:    # kill: %YMM0<def> %YMM0<kill> %ZMM0<def>
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB7_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    movb (%rsi), %al
-; CHECK-NEXT:    shrb $2, %al
+; CHECK-NEXT:    kmovb (%rsi), %k0
+; CHECK-NEXT:    kshiftrb $2, %k0, %k1
 ; CHECK-NEXT:    jmp .LBB7_3
 ; CHECK-NEXT:  .LBB7_2: # %else
-; CHECK-NEXT:    movb (%rdx), %al
+; CHECK-NEXT:    kmovb (%rdx), %k1
 ; CHECK-NEXT:  .LBB7_3: # %exit
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovaps %zmm0, %zmm1 {%k1}
 ; CHECK-NEXT:    vmovaps %ymm1, (%rcx)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   br i1 %cond, label %if, label %else
 
 if:

 define void @test_shl(i1 %cond, i8* %ptr1, i8* %ptr2, <8 x float> %fvec1, <8 x float> %fvec2, <8 x float>* %fptrvec) {
 ; CHECK-LABEL: test_shl:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    # kill: %YMM1<def> %YMM1<kill> %ZMM1<def>
 ; CHECK-NEXT:    # kill: %YMM0<def> %YMM0<kill> %ZMM0<def>
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB8_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    movb (%rsi), %al
-; CHECK-NEXT:    shlb $6, %al
+; CHECK-NEXT:    kmovb (%rsi), %k0
+; CHECK-NEXT:    kshiftlb $6, %k0, %k1
 ; CHECK-NEXT:    jmp .LBB8_3
 ; CHECK-NEXT:  .LBB8_2: # %else
-; CHECK-NEXT:    movb (%rdx), %al
+; CHECK-NEXT:    kmovb (%rdx), %k1
 ; CHECK-NEXT:  .LBB8_3: # %exit
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovaps %zmm0, %zmm1 {%k1}
 ; CHECK-NEXT:    vmovaps %ymm1, (%rcx)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   br i1 %cond, label %if, label %else
 
 if:
exit:
   ret void
 }
 
 define void @test_add(i1 %cond, i8* %ptr1, i8* %ptr2, <8 x float> %fvec1, <8 x float> %fvec2, <8 x float>* %fptrvec) {
 ; CHECK-LABEL: test_add:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    # kill: %YMM1<def> %YMM1<kill> %ZMM1<def>
 ; CHECK-NEXT:    # kill: %YMM0<def> %YMM0<kill> %ZMM0<def>
-; CHECK-NEXT:    movb (%rsi), %al
-; CHECK-NEXT:    movb (%rdx), %dl
+; CHECK-NEXT:    kmovb (%rsi), %k0
+; CHECK-NEXT:    kmovb (%rdx), %k1
 ; CHECK-NEXT:    testb $1, %dil
 ; CHECK-NEXT:    je .LBB9_2
 ; CHECK-NEXT:  # BB#1: # %if
-; CHECK-NEXT:    andb %dl, %al
+; CHECK-NEXT:    kandb %k1, %k0, %k1
 ; CHECK-NEXT:    jmp .LBB9_3
 ; CHECK-NEXT:  .LBB9_2: # %else
-; CHECK-NEXT:    addb %dl, %al
+; CHECK-NEXT:    kaddb %k1, %k0, %k1
 ; CHECK-NEXT:  .LBB9_3: # %exit
-; CHECK-NEXT:    kmovd %eax, %k1
 ; CHECK-NEXT:    vmovaps %zmm0, %zmm1 {%k1}
 ; CHECK-NEXT:    vmovaps %ymm1, (%rcx)
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %loaded1 = load i8, i8* %ptr1
   %loaded2 = load i8, i8* %ptr2
   br i1 %cond, label %if, label %else