GlobalISel: Enable the legalization of G_MERGE_VALUES and G_UNMERGE_VALUES

[volkan]

Summary

LegalizerInfo assumes all G_MERGE_VALUES and G_UNMERGE_VALUES instructions are legal, so it is not possible to legalize vector operations on illegal vector types. This patch fixes the problem by removing the related check and adding default actions for G_MERGE_VALUES and G_UNMERGE_VALUES.

[kristof.beyls]

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

67–88

This is in the class that contains target-independent info on how to legalize operations and types.
Therefore, I don't understand how this could be the right place to specify that specific vector widths are legal and others aren't.
I think this must go in target-specific code.

207–208

This is very generic code, and having explicit checks for specific Gopcodes feels wrong.
Should the if test instead of reading

if ((MI.getOpcode() == TargetOpcode::G_MERGE_VALUES ||
         MI.getOpcode() == TargetOpcode::G_UNMERGE_VALUES) &&
        TypeIdx == 1)

be more something like the following?
With some of the helper functions maybe still needing to be written:

if (hasVariableNumberOfOperands(MI.getOpcode() && TypeIdx == numberOfTypeIndices(MI.getOpcode())

Also, I fail to understand the need for this logic here with the little documentation that exists about G_MERGE_VALUES and G_UNMERGE_VALUES in include/llvm/Target/GenericOpcodes.td.
Are all of the operands in the variable-number-of-operands in G_MERGE_VALUES of the same type?
If so why do you need to get the Type of the last operand, instead of just any other operand?
If not, how can picking an action based on the type of the last operand by correct for the other operands that are of a different type?
It probably shows that I'm pretty confused here. I think I'd need quite a bit of explanation starting from exactly what constraints exist on G_(UN)MERGE_VALUES operands, the relationship with the TypeIdx, and going from there, maybe an example of what issue this code is trying to solve?

[volkan]

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

67–88

I thought about this, but then I decided to handle the common types here. My concern was targets use these operations in order to legalize some other operations. For example, in test/CodeGen/X86/GlobalISel/legalize-add.mir, there is a function called test_add_i64 as below.

bb.1 (%ir-block.0):
  %0(s64) = IMPLICIT_DEF
  %1(s64) = IMPLICIT_DEF
  %2(s64) = G_ADD %0, %1
  RET 0

Here, Legalizer builds G_MERGE_VALUES and G_UNMERGE_VALUES to extract 32-bit values, but they are not actually legal. The patch Aditya proposed (https://reviews.llvm.org/D39267) should fix this problem by processing the legalizer artifacts after legalizing all other instructions.

I'll move this to target-specific code and create another diff.

207–208

Are all of the operands in the variable-number-of-operands in G_MERGE_VALUES of the same type?

Yes, G_MERGE_VALUES concatenates multiple registers of the same size into a wider register and G_UNMERGE_VALUES extracts multiple registers with the specified size. These instructions has variable number of operands, but there is only one source type and one destination type as all sources for G_MERGE_VALUES and all destination for G_UNMERGE_VALUES must be the same type.

Should the if test instead of reading be more something like the following?

That would be incorrect because other instruction with variable number of operands may have other operands with different types. For example:

def G_OPCODE : Instruction {
  let OutOperandList = (outs type0:$dst);
  let InOperandList = (ins type1:$src0, type2:src1, ..., variable_ops);
}

So, this check is okay only for these instructions because they have size constraints.

I think I'd need quite a bit of explanation starting from exactly what constraints exist on G_(UN)MERGE_VALUES operands, the relationship with the TypeIdx, and going from there, maybe an example of what issue this code is trying to solve?

Simply, I'm trying to set legalizer actions for these instruction, but not for every operand. The existing definition for G_MERGE_VALUES is:

def G_MERGE_VALUES : Instruction {
  let OutOperandList = (outs type0:$dst);
  let InOperandList = (ins variable_ops);
  let hasSideEffects = 0;
}

In this case, there is no way to legalize source operands because MI.getDesc().getNumOperands() returns 1. We can fix this by replacing MI.getDesc().getNumOperands() with MI.getNumOperands(), but this means we need to set actions for every possible case.

I'll add a comment to explain why we are doing this.

[volkan]

Updated based on Kristof's comments.

[dsanders]

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

77

Isn't this one redundant? We've already said 2<=Size<8 are unsupported

207–208

Do we really need to treat G_MERGE_VALUE's specially here? G_UNMERGE_VALUES needs it because type1 is only on the last operand, but for G_MERGE_VALUES, type1 is on operands 1 and higher.

Should the if test instead of reading be more something like the following?

That would be incorrect because other instruction with variable number of operands
may have other operands with different types. For example:

def G_OPCODE : Instruction {

let OutOperandList = (outs type0:$dst);
let InOperandList = (ins type1:$src0, type2:src1, ..., variable_ops);

}
So, this check is okay only for these instructions because they have size constraints.

So long as variable_ops is equivalent to specifying typeN (where N is 1 higher than the highest number used elsewhere in the definition) a variable number of times, I think Kristof's if-statement would work for your example. Either way, I agree with the main point that we should be driving this with a generic mechanism rather than a special case.
Also, it seems odd to me that we have type constraints that aren't described.

Maybe, we should have a variable_ops_typeN that indicates the type constraint and leaves some information in the MCInstrDesc we can inspect during this code.

[volkan]

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

207–208

Do we really need to treat G_MERGE_VALUE's specially here? G_UNMERGE_VALUES needs it because type1 is only on the last operand, but for G_MERGE_VALUES, type1 is on operands 1 and higher.

Right, it seems like I was trying to get the last operand for G_MERGE_VALUES for no reason. I'll remove the special case for G_MERGE_VALUES.

So long as variable_ops is equivalent to specifying typeN (where N is 1 higher than the highest number used elsewhere in the definition) a variable number of times, I think Kristof's if-statement would work for your example.

I think it would not work because $src1 doesn't have to be the same type with variable_ops. MRI.getType(MI.getOperand(LastTypeIdx).getReg()) must return type2.

[volkan]

Removed the special case for G_MERGE_VALUES.

[kristof.beyls]

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

207–208

To make this easier to read, I think it may make sense to factor this out into a separate function, e.g. with a name like "getTyFromTypeIdx(MI, TypeIdx)". I'm not sure on the name, but doing it this way would make this code a lot easier to understand, I think.

lib/Target/AArch64/AArch64LegalizerInfo.cpp

355

I haven't thought this through at all, but I'm a bit surprised v8s64 also isn't legal if v6s64 is?
Similarly for S256 if s192 is legal?

[volkan]

• Handled missing cases for AArch64.
• Rebased.

[volkan]

lib/Target/AArch64/AArch64LegalizerInfo.cpp

355

Added missing cases.

[volkan]

Ping.

[dsanders]

include/llvm/Target/GenericOpcodes.td

612

I know this isn't from your patch but Concatenante -> Concatenate

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

185–189

This is a nitpick but we should early-return here

test/CodeGen/AArch64/GlobalISel/legalize-merge-values.mir

2

Could you add a comment explaining the intent of this test?

test/CodeGen/AArch64/GlobalISel/legalize-nonpowerof2eltsvec.mir

21–34

I'm not sure why this test has changed from s32's to s64's. Could you explain?

Also, the name is inconsistent with the types used

[volkan]

Updated based on Daniel's comments.

[volkan]

test/CodeGen/AArch64/GlobalISel/legalize-nonpowerof2eltsvec.mir

21–34

I'm not sure why this test has changed from s32's to s64's. Could you explain?

It's because <3 x s32> is not legal.

[dsanders]

LGTM

test/CodeGen/AArch64/GlobalISel/legalize-nonpowerof2eltsvec.mir

21–34

That makes sense. In that case, changing the name is enough

include/llvm/Target/GenericOpcodes.td

def G_EXTRACT : Instruction {
   let InOperandList = (ins type1:$src, unknown:$offset);
   let hasSideEffects = 0;
 }
 
 // Extract multiple registers specified size, starting from blocks given by
 // indexes. This will almost certainly be mapped to sub-register COPYs after
 // register banks have been selected.
 def G_UNMERGE_VALUES : Instruction {
-  let OutOperandList = (outs);
-  let InOperandList = (ins variable_ops);
+  let OutOperandList = (outs type0:$dst0, variable_ops);
+  let InOperandList = (ins type1:$src);
   let hasSideEffects = 0;
 }
 
 // Insert a smaller register into a larger one at the specified bit-index.
 def G_INSERT : Instruction {
   let OutOperandList = (outs type0:$dst);
   let InOperandList = (ins type0:$src, type1:$op, unknown:$offset);
   let hasSideEffects = 0;
 }
 
-/// Concatenante multiple registers of the same size into a wider register.
+/// Concatenate multiple registers of the same size into a wider register.
 def G_MERGE_VALUES : Instruction {
   let OutOperandList = (outs type0:$dst);
-  let InOperandList = (ins variable_ops);
+  let InOperandList = (ins type1:$src0, variable_ops);
   let hasSideEffects = 0;
 }
 
 // Intrinsic without side effects.
 def G_INTRINSIC : Instruction {
   let OutOperandList = (outs);
   let InOperandList = (ins unknown:$intrin, variable_ops);
   let hasSideEffects = 0;

lib/CodeGen/GlobalISel/LegalizerInfo.cpp

setLegalizeScalarToDifferentSizeStrategy(
       TargetOpcode::G_BRCOND, 0, widenToLargerTypesUnsupportedOtherwise);
   setLegalizeScalarToDifferentSizeStrategy(
       TargetOpcode::G_INSERT, 0, narrowToSmallerAndUnsupportedIfTooSmall);
   setLegalizeScalarToDifferentSizeStrategy(
       TargetOpcode::G_EXTRACT, 0, narrowToSmallerAndUnsupportedIfTooSmall);
   setLegalizeScalarToDifferentSizeStrategy(
       TargetOpcode::G_EXTRACT, 1, narrowToSmallerAndUnsupportedIfTooSmall);
   setScalarAction(TargetOpcode::G_FNEG, 0, {{1, Lower}});
 }
 
 void LegalizerInfo::computeTables() {
   assert(TablesInitialized == false);
 
   for (unsigned OpcodeIdx = 0; OpcodeIdx <= LastOp - FirstOp; ++OpcodeIdx) {
     const unsigned Opcode = FirstOp + OpcodeIdx;
     for (unsigned TypeIdx = 0; TypeIdx != SpecifiedActions[OpcodeIdx].size();
          ++TypeIdx) {
       // 0. Collect information specified through the setAction API, i.e.
       // for specific bit sizes.
       // For scalar types:
       SizeAndActionsVec ScalarSpecifiedActions;
       // For pointer types:
       std::map<uint16_t, SizeAndActionsVec> AddressSpace2SpecifiedActions;
       // For vector types:
       std::map<uint16_t, SizeAndActionsVec> ElemSize2SpecifiedActions;
       for (auto LLT2Action : SpecifiedActions[OpcodeIdx][TypeIdx]) {
         const LLT Type = LLT2Action.first;
         const LegalizeAction Action = LLT2Action.second;
 
         auto SizeAction = std::make_pair(Type.getSizeInBits(), Action);
         if (Type.isPointer())
           AddressSpace2SpecifiedActions[Type.getAddressSpace()].push_back(
               SizeAction);
         else if (Type.isVector())
           ElemSize2SpecifiedActions[Type.getElementType().getSizeInBits()]
               .push_back(SizeAction);
         else
           ScalarSpecifiedActions.push_back(SizeAction);

 // probably going to need specialized lookup structures for various types before
 // we have any hope of doing well with something like <13 x i3>. Even the common
 // cases should do better than what we have now.
 std::pair<LegalizerInfo::LegalizeAction, LLT>
 LegalizerInfo::getAction(const InstrAspect &Aspect) const {
   assert(TablesInitialized && "backend forgot to call computeTables");
   // These *have* to be implemented for now, they're the fundamental basis of
   // how everything else is transformed.
-
-  // FIXME: the long-term plan calls for expansion in terms of load/store (if
-  // they're not legal).
-  if (Aspect.Opcode == TargetOpcode::G_MERGE_VALUES ||
-      Aspect.Opcode == TargetOpcode::G_UNMERGE_VALUES)
-    return std::make_pair(Legal, Aspect.Type);
-
   if (Aspect.Type.isScalar() || Aspect.Type.isPointer())
     return findScalarLegalAction(Aspect);
   assert(Aspect.Type.isVector());
   return findVectorLegalAction(Aspect);
 }
 
+/// Helper function to get LLT for the given type index.
+static LLT getTypeFromTypeIdx(const MachineInstr &MI,
+                              const MachineRegisterInfo &MRI, unsigned OpIdx,
+                              unsigned TypeIdx) {
+  assert(TypeIdx < MI.getNumOperands() && "Unexpected TypeIdx");
+  // G_UNMERGE_VALUES has variable number of operands, but there is only
+  // one source type and one destination type as all destinations must be the
+  // same type. So, get the last operand if TypeIdx == 1.
+  if (MI.getOpcode() == TargetOpcode::G_UNMERGE_VALUES && TypeIdx == 1)
+    return MRI.getType(MI.getOperand(MI.getNumOperands() - 1).getReg());
+  return MRI.getType(MI.getOperand(OpIdx).getReg());
+}
+
 std::tuple<LegalizerInfo::LegalizeAction, unsigned, LLT>
 LegalizerInfo::getAction(const MachineInstr &MI,
                          const MachineRegisterInfo &MRI) const {
   SmallBitVector SeenTypes(8);
   const MCOperandInfo *OpInfo = MI.getDesc().OpInfo;
   // FIXME: probably we'll need to cache the results here somehow?
   for (unsigned i = 0; i < MI.getDesc().getNumOperands(); ++i) {
     if (!OpInfo[i].isGenericType())
       continue;
 
     // We must only record actions once for each TypeIdx; otherwise we'd
     // try to legalize operands multiple times down the line.
     unsigned TypeIdx = OpInfo[i].getGenericTypeIndex();
     if (SeenTypes[TypeIdx])
       continue;
 
     SeenTypes.set(TypeIdx);
 
-    LLT Ty = MRI.getType(MI.getOperand(i).getReg());
+    LLT Ty = getTypeFromTypeIdx(MI, MRI, i, TypeIdx);
     auto Action = getAction({MI.getOpcode(), TypeIdx, Ty});
     if (Action.first != Legal)
       return std::make_tuple(Action.first, TypeIdx, Action.second);
   }
   return std::make_tuple(Legal, 0, LLT{});
 }
 
 bool LegalizerInfo::isLegal(const MachineInstr &MI,
                             const MachineRegisterInfo &MRI) const {

lib/Target/AArch64/AArch64LegalizerInfo.cpp

AArch64LegalizerInfo::AArch64LegalizerInfo(const AArch64Subtarget &ST) {
   // as the destination.
   setAction({G_VAARG, 1, p0}, Legal);
 
   for (auto Ty : {s8, s16, s32, s64, p0})
     setAction({G_VAARG, Ty}, Custom);
 
   if (ST.hasLSE()) {
     for (auto Ty : {s8, s16, s32, s64})
       setAction({G_ATOMIC_CMPXCHG, Ty}, Legal);
     setAction({G_ATOMIC_CMPXCHG, 1, p0}, Legal);
 
     for (unsigned Op :
          {G_ATOMICRMW_XCHG, G_ATOMICRMW_ADD, G_ATOMICRMW_SUB, G_ATOMICRMW_AND,
           G_ATOMICRMW_OR, G_ATOMICRMW_XOR, G_ATOMICRMW_MIN, G_ATOMICRMW_MAX,
           G_ATOMICRMW_UMIN, G_ATOMICRMW_UMAX}) {
       for (auto Ty : {s8, s16, s32, s64}) {
         setAction({Op, Ty}, Legal);
       }
       setAction({Op, 1, p0}, Legal);
     }
   }
 
+  // Merge/Unmerge
+  for (unsigned Op : {G_MERGE_VALUES, G_UNMERGE_VALUES})
+    for (int Sz : {8, 16, 32, 64, 128, 192, 256, 384, 512}) {
+      LLT ScalarTy = LLT::scalar(Sz);
+      setAction({Op, ScalarTy}, Legal);
+      setAction({Op, 1, ScalarTy}, Legal);
+      if (Sz < 32)
+        continue;
+      for (int EltSize = 8; EltSize <= 64; EltSize *= 2) {
+        if (EltSize >= Sz)
+          continue;
+        LLT VecTy = LLT::vector(Sz / EltSize, EltSize);
+        setAction({Op, VecTy}, Legal);
+        setAction({Op, 1, VecTy}, Legal);
+      }
+    }
+
   computeTables();
 }
 
 bool AArch64LegalizerInfo::legalizeCustom(MachineInstr &MI,
                                           MachineRegisterInfo &MRI,
                                           MachineIRBuilder &MIRBuilder) const {
   switch (MI.getOpcode()) {
   default:

lib/Target/ARM/ARMLegalizerInfo.cpp

if (!ST.useSoftFloat() && ST.hasVFP2()) {
     else
       setFCmpLibcallsGNU();
   }
 
   for (unsigned Op : {G_FREM, G_FPOW})
     for (auto Ty : {s32, s64})
       setAction({Op, Ty}, Libcall);
 
+  // Merge/Unmerge
+  for (const auto &Ty : {s32, s64}) {
+    setAction({G_MERGE_VALUES, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, 1, Ty}, Legal);
+  }
+  for (const auto &Ty : {s16, s32}) {
+    setAction({G_MERGE_VALUES, 1, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, Ty}, Legal);
+  }
   computeTables();
 }
 
 void ARMLegalizerInfo::setFCmpLibcallsAEABI() {
   // FCMP_TRUE and FCMP_FALSE don't need libcalls, they should be
   // default-initialized.
   FCmp32Libcalls.resize(CmpInst::LAST_FCMP_PREDICATE + 1);
   FCmp32Libcalls[CmpInst::FCMP_OEQ] = {

lib/Target/X86/X86LegalizerInfo.cpp

 
 void X86LegalizerInfo::setLegalizerInfo32bit() {
 
   const LLT p0 = LLT::pointer(0, TM.getPointerSize() * 8);
   const LLT s1 = LLT::scalar(1);
   const LLT s8 = LLT::scalar(8);
   const LLT s16 = LLT::scalar(16);
   const LLT s32 = LLT::scalar(32);
+  const LLT s64 = LLT::scalar(64);
 
   for (auto Ty : {p0, s1, s8, s16, s32})
     setAction({G_IMPLICIT_DEF, Ty}, Legal);
 
   for (auto Ty : {s8, s16, s32, p0})
     setAction({G_PHI, Ty}, Legal);
 
   for (unsigned BinOp : {G_ADD, G_SUB, G_MUL, G_AND, G_OR, G_XOR})
for (auto Ty : {s8, s16, s32}) {
     setAction({G_ANYEXT, Ty}, Legal);
   }
 
   // Comparison
   setAction({G_ICMP, s1}, Legal);
 
   for (auto Ty : {s8, s16, s32, p0})
     setAction({G_ICMP, 1, Ty}, Legal);
+
+  // Merge/Unmerge
+  for (const auto &Ty : {s16, s32, s64}) {
+    setAction({G_MERGE_VALUES, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, 1, Ty}, Legal);
+  }
+  for (const auto &Ty : {s8, s16, s32}) {
+    setAction({G_MERGE_VALUES, 1, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, Ty}, Legal);
+  }
 }
 
 void X86LegalizerInfo::setLegalizerInfo64bit() {
 
   if (!Subtarget.is64Bit())
     return;
 
   const LLT s64 = LLT::scalar(64);
+  const LLT s128 = LLT::scalar(128);
 
   setAction({G_IMPLICIT_DEF, s64}, Legal);
 
   setAction({G_PHI, s64}, Legal);
 
   for (unsigned BinOp : {G_ADD, G_SUB, G_MUL, G_AND, G_OR, G_XOR})
     setAction({BinOp, s64}, Legal);
 
   for (unsigned MemOp : {G_LOAD, G_STORE})
     setAction({MemOp, s64}, Legal);
 
   // Pointer-handling
   setAction({G_GEP, 1, s64}, Legal);
 
   // Constants
   setAction({TargetOpcode::G_CONSTANT, s64}, Legal);
 
   // Extensions
   for (unsigned extOp : {G_ZEXT, G_SEXT, G_ANYEXT}) {
     setAction({extOp, s64}, Legal);
   }
 
   // Comparison
   setAction({G_ICMP, 1, s64}, Legal);
+
+  // Merge/Unmerge
+  setAction({G_MERGE_VALUES, s128}, Legal);
+  setAction({G_UNMERGE_VALUES, 1, s128}, Legal);
+  setAction({G_MERGE_VALUES, 1, s128}, Legal);
+  setAction({G_UNMERGE_VALUES, s128}, Legal);
 }
 
 void X86LegalizerInfo::setLegalizerInfoSSE1() {
   if (!Subtarget.hasSSE1())
     return;
 
   const LLT s32 = LLT::scalar(32);
+  const LLT s64 = LLT::scalar(64);
   const LLT v4s32 = LLT::vector(4, 32);
   const LLT v2s64 = LLT::vector(2, 64);
 
   for (unsigned BinOp : {G_FADD, G_FSUB, G_FMUL, G_FDIV})
     for (auto Ty : {s32, v4s32})
       setAction({BinOp, Ty}, Legal);
 
   for (unsigned MemOp : {G_LOAD, G_STORE})
     for (auto Ty : {v4s32, v2s64})
       setAction({MemOp, Ty}, Legal);
 
   // Constants
   setAction({TargetOpcode::G_FCONSTANT, s32}, Legal);
+
+  // Merge/Unmerge
+  for (const auto &Ty : {v4s32, v2s64}) {
+    setAction({G_MERGE_VALUES, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, 1, Ty}, Legal);
+  }
+  setAction({G_MERGE_VALUES, 1, s64}, Legal);
+  setAction({G_UNMERGE_VALUES, s64}, Legal);
 }
 
 void X86LegalizerInfo::setLegalizerInfoSSE2() {
   if (!Subtarget.hasSSE2())
     return;
 
   const LLT s32 = LLT::scalar(32);
   const LLT s64 = LLT::scalar(64);
   const LLT v16s8 = LLT::vector(16, 8);
   const LLT v8s16 = LLT::vector(8, 16);
   const LLT v4s32 = LLT::vector(4, 32);
   const LLT v2s64 = LLT::vector(2, 64);
 
+  const LLT v32s8 = LLT::vector(32, 8);
+  const LLT v16s16 = LLT::vector(16, 16);
+  const LLT v8s32 = LLT::vector(8, 32);
+  const LLT v4s64 = LLT::vector(4, 64);
+
   for (unsigned BinOp : {G_FADD, G_FSUB, G_FMUL, G_FDIV})
     for (auto Ty : {s64, v2s64})
       setAction({BinOp, Ty}, Legal);
 
   for (unsigned BinOp : {G_ADD, G_SUB})
     for (auto Ty : {v16s8, v8s16, v4s32, v2s64})
       setAction({BinOp, Ty}, Legal);
 
   setAction({G_MUL, v8s16}, Legal);
 
   setAction({G_FPEXT, s64}, Legal);
   setAction({G_FPEXT, 1, s32}, Legal);
 
   // Constants
   setAction({TargetOpcode::G_FCONSTANT, s64}, Legal);
+
+  // Merge/Unmerge
+  for (const auto &Ty :
+       {v16s8, v32s8, v8s16, v16s16, v4s32, v8s32, v2s64, v4s64}) {
+    setAction({G_MERGE_VALUES, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, 1, Ty}, Legal);
+  }
+  for (const auto &Ty : {v16s8, v8s16, v4s32, v2s64}) {
+    setAction({G_MERGE_VALUES, 1, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, Ty}, Legal);
+  }
 }
 
 void X86LegalizerInfo::setLegalizerInfoSSE41() {
   if (!Subtarget.hasSSE41())
     return;
 
   const LLT v4s32 = LLT::vector(4, 32);
 
   setAction({G_MUL, v4s32}, Legal);
 }
 
 void X86LegalizerInfo::setLegalizerInfoAVX() {
   if (!Subtarget.hasAVX())
     return;
 
   const LLT v16s8 = LLT::vector(16, 8);
   const LLT v8s16 = LLT::vector(8, 16);
   const LLT v4s32 = LLT::vector(4, 32);
   const LLT v2s64 = LLT::vector(2, 64);
 
   const LLT v32s8 = LLT::vector(32, 8);
+  const LLT v64s8 = LLT::vector(64, 8);
   const LLT v16s16 = LLT::vector(16, 16);
+  const LLT v32s16 = LLT::vector(32, 16);
   const LLT v8s32 = LLT::vector(8, 32);
+  const LLT v16s32 = LLT::vector(16, 32);
   const LLT v4s64 = LLT::vector(4, 64);
+  const LLT v8s64 = LLT::vector(8, 64);
 
   for (unsigned MemOp : {G_LOAD, G_STORE})
     for (auto Ty : {v8s32, v4s64})
       setAction({MemOp, Ty}, Legal);
 
   for (auto Ty : {v32s8, v16s16, v8s32, v4s64}) {
     setAction({G_INSERT, Ty}, Legal);
     setAction({G_EXTRACT, 1, Ty}, Legal);
   }
   for (auto Ty : {v16s8, v8s16, v4s32, v2s64}) {
     setAction({G_INSERT, 1, Ty}, Legal);
     setAction({G_EXTRACT, Ty}, Legal);
   }
+  // Merge/Unmerge
+  for (const auto &Ty :
+       {v32s8, v64s8, v16s16, v32s16, v8s32, v16s32, v4s64, v8s64}) {
+    setAction({G_MERGE_VALUES, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, 1, Ty}, Legal);
+  }
+  for (const auto &Ty :
+       {v16s8, v32s8, v8s16, v16s16, v4s32, v8s32, v2s64, v4s64}) {
+    setAction({G_MERGE_VALUES, 1, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, Ty}, Legal);
+  }
 }
 
 void X86LegalizerInfo::setLegalizerInfoAVX2() {
   if (!Subtarget.hasAVX2())
     return;
 
   const LLT v32s8 = LLT::vector(32, 8);
   const LLT v16s16 = LLT::vector(16, 16);
   const LLT v8s32 = LLT::vector(8, 32);
   const LLT v4s64 = LLT::vector(4, 64);
 
+  const LLT v64s8 = LLT::vector(64, 8);
+  const LLT v32s16 = LLT::vector(32, 16);
+  const LLT v16s32 = LLT::vector(16, 32);
+  const LLT v8s64 = LLT::vector(8, 64);
+
   for (unsigned BinOp : {G_ADD, G_SUB})
     for (auto Ty : {v32s8, v16s16, v8s32, v4s64})
       setAction({BinOp, Ty}, Legal);
 
   for (auto Ty : {v16s16, v8s32})
     setAction({G_MUL, Ty}, Legal);
+
+  // Merge/Unmerge
+  for (const auto &Ty : {v64s8, v32s16, v16s32, v8s64}) {
+    setAction({G_MERGE_VALUES, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, 1, Ty}, Legal);
+  }
+  for (const auto &Ty : {v32s8, v16s16, v8s32, v4s64}) {
+    setAction({G_MERGE_VALUES, 1, Ty}, Legal);
+    setAction({G_UNMERGE_VALUES, Ty}, Legal);
+  }
 }
 
 void X86LegalizerInfo::setLegalizerInfoAVX512() {
   if (!Subtarget.hasAVX512())
     return;
 
   const LLT v16s8 = LLT::vector(16, 8);
   const LLT v8s16 = LLT::vector(8, 16);

test/CodeGen/AArch64/GlobalISel/legalize-combines.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -O0 -run-pass=legalizer -global-isel %s -o - | FileCheck %s
 
 --- |
   target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64--"
   define void @test_combines_2() { ret void }
   define void @test_combines_3() { ret void }
   define void @test_combines_4() { ret void }
   define void @test_combines_5() { ret void }
-  define void @test_combines_6() { ret void }
 ...
 
 ---
 name:            test_combines_2
 body: |
   bb.0:
     liveins: %w0
 
bb.0:
     %0:_(s32) = COPY %w0
 
     %1:_(s32) = G_ADD %0, %0
     %2:_(s64) = G_MERGE_VALUES %0, %1
     %3:_(s32), %4:_(s32) = G_UNMERGE_VALUES %2
     %5:_(s32) = G_ADD %3, %4
     %w0 = COPY %5
 ...
-
----
-name:            test_combines_6
-body: |
-  bb.0:
-    liveins: %w0
-
-    ; Check that we replace all the uses of a G_EXTRACT.
-    ; CHECK-LABEL: name: test_combines_6
-    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY %w0
-    ; CHECK: [[MUL:%[0-9]+]]:_(s32) = G_MUL [[COPY]], [[COPY]]
-    ; CHECK: [[ADD:%[0-9]+]]:_(s32) = G_ADD [[COPY]], [[MUL]]
-    %0:_(s32) = COPY %w0
-
-    %1:_(s32) = G_MERGE_VALUES %0
-    %2:_(s32) = G_UNMERGE_VALUES %1
-    %3:_(s32) = G_MUL %2, %2
-    %4:_(s32) = G_ADD %2, %3
-    %w0 = COPY %4
-...

test/CodeGen/AArch64/GlobalISel/legalize-merge-values.mir

+# RUN: llc -O0 -run-pass=legalizer -global-isel -global-isel-abort=0 -pass-remarks-missed='gisel*' %s -o - 2>&1 | FileCheck %s
+
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+  target triple = "aarch64--"
+  define void @test_merge_s4() {
+    ret void
+  }
+...
+
+---
+name:            test_merge_s4
+registers:
+  - { id: 0, class: _ }
+  - { id: 1, class: _ }
+  - { id: 2, class: _ }
+  - { id: 3, class: _ }
+  - { id: 4, class: _ }
+body: |
+  bb.0:
+    %0(s64) = G_CONSTANT i64 0
+    %1(s4) = G_TRUNC %0(s64)
+    ; Previously, LegalizerInfo was assuming all G_MERGE_VALUES and G_UNMERGE_VALUES
+    ; instructions are legal. Make sure that is no longer happening.
+    ; CHECK: unable to legalize instruction: {{.*}} G_MERGE_VALUES
+    %2(s8) = G_MERGE_VALUES %1(s4), %1(s4)
+    %3(s8) = COPY %2(s8)
+    %4(s64) = G_ANYEXT %3(s8)
+    %x0 = COPY %4(s64)
+...

test/CodeGen/AArch64/GlobalISel/legalize-nonpowerof2eltsvec.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -O0 -run-pass=legalizer -global-isel %s -o - | FileCheck %s
 
 --- |
   target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
   target triple = "aarch64--"
-  define void @test_legalize_merge_v3s32() {
+  define void @test_legalize_merge_v3s64() {
     ret void
   }
 ...
 ---
-name:            test_legalize_merge_v3s32
+name:            test_legalize_merge_v3s64
 registers:
   - { id: 0, class: _ }
   - { id: 1, class: _ }
   - { id: 2, class: _ }
   - { id: 3, class: _ }
+  - { id: 4, class: _ }
+  - { id: 5, class: _ }
 body: |
   bb.0:
-    liveins: %w0, %w1, %w2
-    ; CHECK-LABEL: name: test_legalize_merge_v3s32
-    ; CHECK: [[COPY:%[0-9]+]]:_(s32) = COPY %w0
-    ; CHECK: [[COPY1:%[0-9]+]]:_(s32) = COPY %w1
-    ; CHECK: [[COPY2:%[0-9]+]]:_(s32) = COPY %w2
-    ; CHECK: %w0 = COPY [[COPY]](s32)
-    ; CHECK: %w1 = COPY [[COPY1]](s32)
-    ; CHECK: %w2 = COPY [[COPY2]](s32)
-    %0(s32) = COPY %w0
-    %1(s32) = COPY %w1
-    %2(s32) = COPY %w2
-    %3(<3 x s32>) = G_MERGE_VALUES %0(s32), %1(s32), %2(s32)
-    %4:_(s32), %5:_(s32), %6:_(s32) = G_UNMERGE_VALUES %3
-    %w0 = COPY %4
-    %w1 = COPY %5
-    %w2 = COPY %6
+    liveins: %w0
+    ; CHECK-LABEL: name: test_legalize_merge_v3s64
+    ; CHECK: [[COPY:%[0-9]+]]:_(s64) = COPY %x0
+    ; CHECK: [[MV:%[0-9]+]]:_(<3 x s64>) = G_MERGE_VALUES [[COPY]](s64), [[COPY]](s64), [[COPY]](s64)
+    ; CHECK: [[COPY1:%[0-9]+]]:_(<3 x s64>) = COPY [[MV]](<3 x s64>)
+    ; CHECK: [[UV:%[0-9]+]]:_(s64), [[UV1:%[0-9]+]]:_(s64), [[UV2:%[0-9]+]]:_(s64) = G_UNMERGE_VALUES [[COPY1]](<3 x s64>)
+    ; CHECK: %x0 = COPY [[UV]](s64)
+    %0(s64) = COPY %x0
+    %1(<3 x s64>) = G_MERGE_VALUES %0(s64), %0(s64), %0(s64)
+    %2(<3 x s64>) = COPY %1(<3 x s64>)
+    %3(s64), %4(s64), %5(s64) = G_UNMERGE_VALUES %2(<3 x s64>)
+    %x0 = COPY %3(s64)
 ...

test/CodeGen/AArch64/GlobalISel/legalize-unmerge-values.mir

+# RUN: llc -O0 -run-pass=legalizer -global-isel -global-isel-abort=0 -pass-remarks-missed='gisel*' %s -o - 2>&1 | FileCheck %s
+
+--- |
+  target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+  target triple = "aarch64--"
+  define void @test_unmerge_s4() {
+    ret void
+  }
+...
+
+---
+name:            test_unmerge_s4
+registers:
+  - { id: 0, class: _ }
+  - { id: 1, class: _ }
+  - { id: 2, class: _ }
+  - { id: 3, class: _ }
+body: |
+  bb.0:
+    %0(s8) = G_CONSTANT i8 0
+    ; Previously, LegalizerInfo was assuming all G_MERGE_VALUES and G_UNMERGE_VALUES
+    ; instructions are legal. Make sure that is no longer happening.
+    ; CHECK: unable to legalize instruction: {{.*}} G_UNMERGE_VALUES
+    %1(s4), %2(s4)= G_UNMERGE_VALUES %0(s8)
+    %3(s64) = G_ANYEXT %1(s4)
+    %x0 = COPY %3(s64)
+
+...

test/CodeGen/AArch64/GlobalISel/no-regclass.mir

 body:             |
   bb.0:
     liveins: %w0
     ; CHECK-LABEL: name: unused_reg
     ; CHECK: liveins: %w0
     ; CHECK: [[COPY:%[0-9]+]]:gpr32all = COPY %w0
     ; CHECK: %w0 = COPY [[COPY]]
     %0:gpr(s32) = COPY %w0
-    %1:gpr(s32) = G_MERGE_VALUES %0(s32)
-    %2:gpr(s32) = G_UNMERGE_VALUES %1(s32)
+    %1:gpr(s64) = G_MERGE_VALUES %0(s32), %0(s32)
+    %2:gpr(s32), %3:gpr(s32) = G_UNMERGE_VALUES %1(s64)
     %w0 = COPY %2(s32)
 ...