[AMDGPU] Divergence driven instruction selection. Shift operations.

[alex-t]

Summary

This change enables VOP3 shifts to be explicitly selected dependent on the divergence.

Tests: CodeGen/AMDGPU passed

[rampitec]

lib/Target/AMDGPU/VOP3Instructions.td

399

Why not GCNPat?
Why divergence is not checked?
Why do you need it at all if you have patgen enabled for these instructions above?

587

Does it really belong to this patch?

[alex-t]

Pattern changed to GCNPat, divergence check added.

lib/Target/AMDGPU/VOP3Instructions.td

399

On VI+ we have no V_LSHL_B64 only V_LSHLREV_B64, same for sra, srl.
So, we cannot select in case we have, let's say, shl with src0 i64 and src1 i32.
The aim of these patterns to swap operands. Current implementation does this in
SIInstrInfo::moveToVALU

case AMDGPU::S_LSHL_B64:
  if (ST.getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS) {
    NewOpcode = AMDGPU::V_LSHLREV_B64;
    **swapOperands(Inst);**
  }

587

Sure it does. As soon as I start selecting this _b64 the LIT tests failed because of the odd "_e64" suffix printed by AMDGPUInstrPrinter. It prints it for all with VOP3 flag.
The flag was not set before this change because the instructions were created in moveToVALU.
I filed the bug https://bugs.llvm.org/show_bug.cgi?id=39086 for adding the flag to indicate that the instruction does not have 32bit encoding.

[rampitec]

lib/Target/AMDGPU/VOP3Instructions.td

399

Then what the code "def V_LSHLREV_B64 : VOP3Inst <"v_lshlrev_b64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I32_I64>>, shl>" 8 lines above does? Isn't it the same?

587

How can it be possible for moveToVALU to create a VOP3 instruction without a VOP3 flag?
VOP3_Pseudo sets VOP3 flag. InstSI copies it into TSFlags. VOP3_Real copies TSFlags from a pseudo.

[alex-t]

lib/Target/AMDGPU/VOP3Instructions.td

399

No. The code you've mentioned defines the pattern for:

Divergent node that has

1. shl operator
2. i64 result, i32 src0 and i64 src1

it is okay for v_lshlREV - it shifts src1 64 bits to i32 src0 positions

what if we have the opposite order of operands? On SICI we'd select v_lshl since
that order fits. On VI we have only v_lshlREV so we need to swap operands.

[alex-t]

lib/Target/AMDGPU/VOP3Instructions.td

587

InstSI copies VOP3 to TSFlags and your code checks exactly TSFlags & SIInstrFlags::VOP3

[rampitec]

lib/Target/AMDGPU/VOP3Instructions.td

392

It sounds like from your explanation below and the logic of the getVOP3Pat this will create a bogus pattern with wrong operand order. Only one pattern shall exist for (shl i64:x, i32:y) and it seems to be the pattern below. At best this one will never match.

587

So the explanation is wrong. It has VOP3 in TSFlags with or without your changes. It is worth nothing where the instruction is created. The suffix was not printed because the control did not even come to the AMDGPUInstPrinter::printVOPDst, and it did not trigger because these instructions did not have VOPDstOperand in the td. Now what you have changed is the operand definition. What you need to fix the problem instead of the hack is to return DstRC back to the RegisterOperand<VReg_64> in the profile.

[alex-t]

lib/Target/AMDGPU/VOP3Instructions.td

587

I did not change the VOP3_Profile.
The reason was in the order of the profiles instantiation: VOP_PAT_GEN<VOP3_Profile<...>> VOP_PAT_GEN does not inherit VOP3_Profile let statements. It just inherits types.

The fix is: VOP3_Profile<VOP_PAT_GEN<...> and to add to VOP3_Profile

"let NeedPatGen = P.NeedPatGen"

[alex-t]

lib/Target/AMDGPU/VOP3Instructions.td

392

If we have VI and only one this:

def V_LSHLREV_B64 : VOP3Inst <"v_lshlrev_b64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I32_I64>>, shl>;

then "shl i64 i32" will never match.

[rampitec]

lib/Target/AMDGPU/VOP3Instructions.td

392

You have the match for it right below:

(getDivergentFrag<shl>.ret i64:$x, i32:$y)

This is exactly shl i64, i32.

[alex-t]

Fixes according the discussion results.

[alex-t]

lib/Target/AMDGPU/VOP3Instructions.td

392

The reason of misunderstanding was that I assumed that it could be

shl i32 i64 input on VI

If it could be we'd really need 2 different patterns for the same instruction V_LSHLREV_B32

one for

shl i32 i64
``` and another for

shl i64 i32

As soon as I understood that nobody can swap shl dag node operands, everything become clear.

[rampitec]

LGTM

lib/Target/AMDGPU/SOPInstructions.td

 let AddedComplexity = 1 in {
 
 let Defs = [SCC] in {
 // TODO: b64 versions require VOP3 change since v_lshlrev_b64 is VOP3
 def S_LSHL_B32 : SOP2_32 <"s_lshl_b32",
   [(set i32:$sdst, (UniformBinFrag<shl> i32:$src0, i32:$src1))]
 >;
 def S_LSHL_B64 : SOP2_64_32 <"s_lshl_b64",
-  [(set i64:$sdst, (shl i64:$src0, i32:$src1))]
+  [(set i64:$sdst, (UniformBinFrag<shl> i64:$src0, i32:$src1))]
 >;
 def S_LSHR_B32 : SOP2_32 <"s_lshr_b32",
   [(set i32:$sdst, (UniformBinFrag<srl> i32:$src0, i32:$src1))]
 >;
 def S_LSHR_B64 : SOP2_64_32 <"s_lshr_b64",
-  [(set i64:$sdst, (srl i64:$src0, i32:$src1))]
+  [(set i64:$sdst, (UniformBinFrag<srl> i64:$src0, i32:$src1))]
 >;
 def S_ASHR_I32 : SOP2_32 <"s_ashr_i32",
   [(set i32:$sdst, (UniformBinFrag<sra> i32:$src0, i32:$src1))]
 >;
 def S_ASHR_I64 : SOP2_64_32 <"s_ashr_i64",
-  [(set i64:$sdst, (sra i64:$src0, i32:$src1))]
+  [(set i64:$sdst, (UniformBinFrag<sra> i64:$src0, i32:$src1))]
 >;
 } // End Defs = [SCC]
 
 def S_BFM_B32 : SOP2_32 <"s_bfm_b32",
   [(set i32:$sdst, (UniformBinFrag<AMDGPUbfm> i32:$src0, i32:$src1))]>;
 def S_BFM_B64 : SOP2_64_32_32 <"s_bfm_b64">;
 
 // TODO: S_MUL_I32 require V_MUL_LO_I32 from VOP3 change

lib/Target/AMDGPU/VOP3Instructions.td

 //===----------------------------------------------------------------------===//
 
 class getVOP3ModPat<VOPProfile P, SDPatternOperator node> {
   dag src0 = !if(P.HasOMod,
     (VOP3Mods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp, i32:$omod),
     (VOP3Mods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp));
 
   list<dag> ret3 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT src0),
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT src0),
           (P.Src1VT (VOP3Mods P.Src1VT:$src1, i32:$src1_modifiers)),
           (P.Src2VT (VOP3Mods P.Src2VT:$src2, i32:$src2_modifiers))))];
 
   list<dag> ret2 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT src0),
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT src0),
           (P.Src1VT (VOP3Mods P.Src1VT:$src1, i32:$src1_modifiers))))];
 
   list<dag> ret1 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT src0)))];
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT src0)))];
 
   list<dag> ret = !if(!eq(P.NumSrcArgs, 3), ret3,
                   !if(!eq(P.NumSrcArgs, 2), ret2,
                   ret1));
 }
 
 class getVOP3PModPat<VOPProfile P, SDPatternOperator node> {
   list<dag> ret3 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT !if(P.HasClamp, (VOP3PMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp),
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT !if(P.HasClamp, (VOP3PMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp),
                                     (VOP3PMods P.Src0VT:$src0, i32:$src0_modifiers))),
           (P.Src1VT (VOP3PMods P.Src1VT:$src1, i32:$src1_modifiers)),
           (P.Src2VT (VOP3PMods P.Src2VT:$src2, i32:$src2_modifiers))))];
 
   list<dag> ret2 = [(set P.DstVT:$vdst,
-    (node !if(P.HasClamp, (P.Src0VT (VOP3PMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp)),
+    (DivergentFragOrOp<node, P>.ret !if(P.HasClamp, (P.Src0VT (VOP3PMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp)),
                           (P.Src0VT (VOP3PMods P.Src0VT:$src0, i32:$src0_modifiers))),
           (P.Src1VT (VOP3PMods P.Src1VT:$src1, i32:$src1_modifiers))))];
 
   list<dag> ret1 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT (VOP3PMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp))))];
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT (VOP3PMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp))))];
 
   list<dag> ret = !if(!eq(P.NumSrcArgs, 3), ret3,
                   !if(!eq(P.NumSrcArgs, 2), ret2,
                   ret1));
 }
 
 class getVOP3OpSelPat<VOPProfile P, SDPatternOperator node> {
   list<dag> ret3 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT !if(P.HasClamp, (VOP3OpSel0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp),
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT !if(P.HasClamp, (VOP3OpSel0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp),
                                     (VOP3OpSel P.Src0VT:$src0, i32:$src0_modifiers))),
           (P.Src1VT (VOP3OpSel P.Src1VT:$src1, i32:$src1_modifiers)),
           (P.Src2VT (VOP3OpSel P.Src2VT:$src2, i32:$src2_modifiers))))];
 
   list<dag> ret2 = [(set P.DstVT:$vdst,
-    (node !if(P.HasClamp, (P.Src0VT (VOP3OpSel0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp)),
+    (DivergentFragOrOp<node, P>.ret !if(P.HasClamp, (P.Src0VT (VOP3OpSel0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp)),
                           (P.Src0VT (VOP3OpSel P.Src0VT:$src0, i32:$src0_modifiers))),
           (P.Src1VT (VOP3OpSel P.Src1VT:$src1, i32:$src1_modifiers))))];
 
   list<dag> ret1 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT (VOP3OpSel0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp))))];
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT (VOP3OpSel0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp))))];
 
   list<dag> ret = !if(!eq(P.NumSrcArgs, 3), ret3,
                   !if(!eq(P.NumSrcArgs, 2), ret2,
                   ret1));
 }
 
 class getVOP3OpSelModPat<VOPProfile P, SDPatternOperator node> {
   list<dag> ret3 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT !if(P.HasClamp, (VOP3OpSelMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp),
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT !if(P.HasClamp, (VOP3OpSelMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp),
                                     (VOP3OpSelMods P.Src0VT:$src0, i32:$src0_modifiers))),
           (P.Src1VT (VOP3OpSelMods P.Src1VT:$src1, i32:$src1_modifiers)),
           (P.Src2VT (VOP3OpSelMods P.Src2VT:$src2, i32:$src2_modifiers))))];
 
   list<dag> ret2 = [(set P.DstVT:$vdst,
-    (node !if(P.HasClamp, (P.Src0VT (VOP3OpSelMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp)),
+    (DivergentFragOrOp<node, P>.ret !if(P.HasClamp, (P.Src0VT (VOP3OpSelMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp)),
                           (P.Src0VT (VOP3OpSelMods P.Src0VT:$src0, i32:$src0_modifiers))),
           (P.Src1VT (VOP3OpSelMods P.Src1VT:$src1, i32:$src1_modifiers))))];
 
   list<dag> ret1 = [(set P.DstVT:$vdst,
-    (node (P.Src0VT (VOP3OpSelMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp))))];
+    (DivergentFragOrOp<node, P>.ret (P.Src0VT (VOP3OpSelMods0 P.Src0VT:$src0, i32:$src0_modifiers, i1:$clamp))))];
 
   list<dag> ret = !if(!eq(P.NumSrcArgs, 3), ret3,
                   !if(!eq(P.NumSrcArgs, 2), ret2,
                   ret1));
 }
 
 class getVOP3Pat<VOPProfile P, SDPatternOperator node> {
-  list<dag> ret3 = [(set P.DstVT:$vdst, (node P.Src0VT:$src0, P.Src1VT:$src1, P.Src2VT:$src2))];
-  list<dag> ret2 = [(set P.DstVT:$vdst, (node P.Src0VT:$src0, P.Src1VT:$src1))];
-  list<dag> ret1 = [(set P.DstVT:$vdst, (node P.Src0VT:$src0))];
+  list<dag> ret3 = [(set P.DstVT:$vdst, (DivergentFragOrOp<node, P>.ret P.Src0VT:$src0, P.Src1VT:$src1, P.Src2VT:$src2))];
+  list<dag> ret2 = [(set P.DstVT:$vdst, (DivergentFragOrOp<node, P>.ret P.Src0VT:$src0, P.Src1VT:$src1))];
+  list<dag> ret1 = [(set P.DstVT:$vdst, (DivergentFragOrOp<node, P>.ret P.Src0VT:$src0))];
   list<dag> ret = !if(!eq(P.NumSrcArgs, 3), ret3,
                   !if(!eq(P.NumSrcArgs, 2), ret2,
                   ret1));
 }
 
 class getVOP3ClampPat<VOPProfile P, SDPatternOperator node> {
   list<dag> ret3 = [(set P.DstVT:$vdst, (node P.Src0VT:$src0, P.Src1VT:$src1, P.Src2VT:$src2, i1:$clamp))];
   list<dag> ret2 = [(set P.DstVT:$vdst, (node P.Src0VT:$src0, P.Src1VT:$src1, i1:$clamp))];

 } // End Constraints = "@earlyclobber $vdst"
 
 def V_TRIG_PREOP_F64 : VOP3Inst <"v_trig_preop_f64", VOP3_Profile<VOP_F64_F64_I32>, AMDGPUtrig_preop> {
   let SchedRW = [WriteDouble];
 }
 
 let SchedRW = [Write64Bit] in {
 // These instructions only exist on SI and CI
-let SubtargetPredicate = isSICI in {
-def V_LSHL_B64 : VOP3Inst <"v_lshl_b64", VOP3_Profile<VOP_I64_I64_I32>>;
-def V_LSHR_B64 : VOP3Inst <"v_lshr_b64", VOP3_Profile<VOP_I64_I64_I32>>;
-def V_ASHR_I64 : VOP3Inst <"v_ashr_i64", VOP3_Profile<VOP_I64_I64_I32>>;
+let SubtargetPredicate = isSICI, Predicates = [isSICI] in {
+def V_LSHL_B64 : VOP3Inst <"v_lshl_b64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I64_I32>>, shl>;
+def V_LSHR_B64 : VOP3Inst <"v_lshr_b64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I64_I32>>, srl>;
+def V_ASHR_I64 : VOP3Inst <"v_ashr_i64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I64_I32>>, sra>;
 def V_MULLIT_F32 : VOP3Inst <"v_mullit_f32", VOP3_Profile<VOP_F32_F32_F32_F32>>;
-} // End SubtargetPredicate = isSICI
+} // End SubtargetPredicate = isSICI, Predicates = [isSICI]
 
-let SubtargetPredicate = isVI in {
-def V_LSHLREV_B64 : VOP3Inst <"v_lshlrev_b64", VOP3_Profile<VOP_I64_I32_I64>>;
-def V_LSHRREV_B64 : VOP3Inst <"v_lshrrev_b64", VOP3_Profile<VOP_I64_I32_I64>>;
-def V_ASHRREV_I64 : VOP3Inst <"v_ashrrev_i64", VOP3_Profile<VOP_I64_I32_I64>>;
-} // End SubtargetPredicate = isVI
+let SubtargetPredicate = isVI, Predicates = [isVI] in {
+def V_LSHLREV_B64 : VOP3Inst <"v_lshlrev_b64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I32_I64>>, shl>;
+def V_LSHRREV_B64 : VOP3Inst <"v_lshrrev_b64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I32_I64>>, srl>;
+def V_ASHRREV_I64 : VOP3Inst <"v_ashrrev_i64", VOP_PAT_GEN<VOP3_Profile<VOP_I64_I32_I64>>, sra>;
+} // End SubtargetPredicate = isVI, Predicates = [isVI]
 } // End SchedRW = [Write64Bit]
 
+let Predicates = [isVI] in {
+def : AMDGPUPat <
+ (shl i64:$x, i32:$y),
+ (V_LSHLREV_B64 $y, $x)
+>;
+def : AMDGPUPat <
+ (srl i64:$x, i32:$y),
+ (V_LSHRREV_B64 $y, $x)
+>;
+def : AMDGPUPat <
+ (sra i64:$x, i32:$y),
+ (V_ASHRREV_I64 $y, $x)
+>; 
+}
+
+
 let SubtargetPredicate = isCIVI in {
 
 let Constraints = "@earlyclobber $vdst", SchedRW = [WriteQuarterRate32] in {
 def V_QSAD_PK_U16_U8 : VOP3Inst <"v_qsad_pk_u16_u8", VOP3_Profile<VOP_I64_I64_I32_I64, VOP3_CLAMP>>;
 def V_MQSAD_U32_U8 : VOP3Inst <"v_mqsad_u32_u8", VOP3_Profile<VOP_V4I32_I64_I32_V4I32, VOP3_CLAMP>>;
 } // End Constraints = "@earlyclobber $vdst", SchedRW = [WriteQuarterRate32]
 
 let isCommutable = 1 in {
def _si : VOP3_Real<!cast<VOP3_Pseudo>(NAME), SIEncodingFamily.SI>,
             VOP3e_si <op, !cast<VOP3_Pseudo>(NAME).Pfl>;
 }
 
 multiclass VOP3be_Real_si<bits<9> op> {
   def _si : VOP3_Real<!cast<VOP3_Pseudo>(NAME), SIEncodingFamily.SI>,
             VOP3be_si <op, !cast<VOP3_Pseudo>(NAME).Pfl>;
 }
 
+multiclass VOP3_Real_e64only_si <bits<9> op> {
+  def _si:
+    VOP3_Real<!cast<VOP3_Pseudo>(NAME), SIEncodingFamily.SI>,
+    VOP3e_si <op, !cast<VOP3_Pseudo>(NAME).Pfl> {
+      // Hack to stop printing _e64
+      VOP3_Pseudo ps = !cast<VOP3_Pseudo>(NAME);
+      let OutOperandList = (outs VReg_64:$vdst);
+      let AsmString = ps.Mnemonic # " " # ps.AsmOperands;
+    }
+}
+
 } // End AssemblerPredicates = [isSICI], DecoderNamespace = "SICI"
 
 defm V_MAD_LEGACY_F32   : VOP3_Real_si <0x140>;
 defm V_MAD_F32          : VOP3_Real_si <0x141>;
 defm V_MAD_I32_I24      : VOP3_Real_si <0x142>;
 defm V_MAD_U32_U24      : VOP3_Real_si <0x143>;
 defm V_CUBEID_F32       : VOP3_Real_si <0x144>;
 defm V_CUBESC_F32       : VOP3_Real_si <0x145>;

 defm V_MED3_U32         : VOP3_Real_si <0x159>;
 defm V_SAD_U8           : VOP3_Real_si <0x15a>;
 defm V_SAD_HI_U8        : VOP3_Real_si <0x15b>;
 defm V_SAD_U16          : VOP3_Real_si <0x15c>;
 defm V_SAD_U32          : VOP3_Real_si <0x15d>;
 defm V_CVT_PK_U8_F32    : VOP3_Real_si <0x15e>;
 defm V_DIV_FIXUP_F32    : VOP3_Real_si <0x15f>;
 defm V_DIV_FIXUP_F64    : VOP3_Real_si <0x160>;
-defm V_LSHL_B64         : VOP3_Real_si <0x161>;
-defm V_LSHR_B64         : VOP3_Real_si <0x162>;
-defm V_ASHR_I64         : VOP3_Real_si <0x163>;
+defm V_LSHL_B64         : VOP3_Real_e64only_si <0x161>;
+defm V_LSHR_B64         : VOP3_Real_e64only_si <0x162>;
+defm V_ASHR_I64         : VOP3_Real_e64only_si <0x163>;
 defm V_ADD_F64          : VOP3_Real_si <0x164>;
 defm V_MUL_F64          : VOP3_Real_si <0x165>;
 defm V_MIN_F64          : VOP3_Real_si <0x166>;
 defm V_MAX_F64          : VOP3_Real_si <0x167>;
 defm V_LDEXP_F64        : VOP3_Real_si <0x168>;
 defm V_MUL_LO_U32       : VOP3_Real_si <0x169>;
 defm V_MUL_HI_U32       : VOP3_Real_si <0x16a>;
 defm V_MUL_LO_I32       : VOP3_Real_si <0x16b>;
def _vi : VOP3_Real<!cast<VOP_Pseudo>(NAME), SIEncodingFamily.VI>,
             VOP3OpSel_gfx9 <op, !cast<VOP_Pseudo>(NAME).Pfl>;
 }
 
 multiclass VOP3Interp_Real_vi<bits<10> op> {
   def _vi : VOP3_Real<!cast<VOP_Pseudo>(NAME), SIEncodingFamily.VI>,
             VOP3Interp_vi <op, !cast<VOP_Pseudo>(NAME).Pfl>;
 }
 
+multiclass VOP3_Real_e64only_vi <bits<10> op> {
+  def _vi:
+    VOP3_Real<!cast<VOP3_Pseudo>(NAME), SIEncodingFamily.VI>,
+    VOP3e_vi <op, !cast<VOP3_Pseudo>(NAME).Pfl> {
+      // Hack to stop printing _e64
+      VOP3_Pseudo ps = !cast<VOP3_Pseudo>(NAME);
+      let OutOperandList = (outs VReg_64:$vdst);
+      let AsmString = ps.Mnemonic # " " # ps.AsmOperands;
+    }
+}
+
 } // End AssemblerPredicates = [isVI], DecoderNamespace = "VI"
 
 let AssemblerPredicates = [isVIOnly], DecoderNamespace = "VI" in {
 
 multiclass VOP3_F16_Real_vi<bits<10> op> {
   def _vi : VOP3_Real<!cast<VOP3_Pseudo>(NAME), SIEncodingFamily.VI>,
             VOP3e_vi <op, !cast<VOP3_Pseudo>(NAME).Pfl>;
 }

 }
 
 defm V_MUL_HI_U32       : VOP3_Real_vi <0x286>;
 defm V_MUL_HI_I32       : VOP3_Real_vi <0x287>;
 
 defm V_READLANE_B32     : VOP3_Real_vi <0x289>;
 defm V_WRITELANE_B32    : VOP3_Real_vi <0x28a>;
 
-defm V_LSHLREV_B64      : VOP3_Real_vi <0x28f>;
-defm V_LSHRREV_B64      : VOP3_Real_vi <0x290>;
-defm V_ASHRREV_I64      : VOP3_Real_vi <0x291>;
+defm V_LSHLREV_B64      : VOP3_Real_e64only_vi <0x28f>;
+defm V_LSHRREV_B64      : VOP3_Real_e64only_vi <0x290>;
+defm V_ASHRREV_I64      : VOP3_Real_e64only_vi <0x291>;
 defm V_TRIG_PREOP_F64   : VOP3_Real_vi <0x292>;
 
 defm V_LSHL_ADD_U32 : VOP3_Real_vi <0x1fd>;
 defm V_ADD_LSHL_U32 : VOP3_Real_vi <0x1fe>;
 defm V_ADD3_U32 : VOP3_Real_vi <0x1ff>;
 defm V_LSHL_OR_B32 : VOP3_Real_vi <0x200>;
 defm V_AND_OR_B32 : VOP3_Real_vi <0x201>;
 defm V_OR3_B32 : VOP3_Real_vi <0x202>;

lib/Target/AMDGPU/VOPInstructions.td

class VOPPatGen<SDPatternOperator Op, VOPProfile P> {
 
   list<dag> ret =  [!con(Outs, (set Ins))];
 }
 
 class VOPPatOrNull<SDPatternOperator Op, VOPProfile P> {
   list<dag> ret = !if(!ne(P.NeedPatGen,PatGenMode.NoPattern), VOPPatGen<Op, P>.ret, []);
 }
 
+class DivergentFragOrOp<SDPatternOperator Op, VOPProfile P> {
+  SDPatternOperator ret = !if(!eq(P.NeedPatGen,PatGenMode.Pattern),
+   !if(!isa<SDNode>(Op), getDivergentFrag<Op>.ret, Op), Op);
+}
+
 include "VOPCInstructions.td"
 include "VOP1Instructions.td"
 include "VOP2Instructions.td"
 include "VOP3Instructions.td"
 include "VOP3PInstructions.td"