AMDGPU: Refactor exp instructions

[arsenm]

Summary

Structure the definitions a bit more like the other classes.

The main change here is to split EXP with the done bit set
to a separate opcode, so we can set hasSideEffects = 1
if the done bit is in use since this has the special constraint
that it should be the last exp in the shader.

Previously all exp instructions were inferred to have unmodeled
side effects.

[mareko]

DONE doesn't always mean it's the last EXP in the shader. It's only true for pixel shaders. For vertex shaders, it means it's the last Position export. Param vertex shader exports are unaffected and can be moved arbitrarily.

[axeldavy]

I think the DONE bit should be added at one of the last passes of compilation.

Especially we want to be free at Scheduling step to move around the exp instructions and have them the order we want.
I guess the export may take some time, so earlier the exp, the best.

We can also introduce in the SI scheduler some way to incitate to have the position exports first for vertex shaders.

[• tstellarAMD]

I'm see cannot select error for EXPORT_DONE with this test case:

define void @main([9 x <16 x i8>] addrspace(2)* byval, [17 x <16 x i8>] addrspace(2)* byval, [17 x <4 x i32>] addrspace(2)* byval, [34 x <8 x i32>] addrspace(2)* byval, float inreg, i32 inreg, <2 x i32>, <2 x i32>, <2 x i32>, <3 x i32>, <2 x i32>, <2 x i32>, <2 x i32>, float, float, float, float, float, float, i32, float, float) #0 {
main_body:

call void @llvm.SI.export(i32 0, i32 1, i32 1, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0)
ret void

}

declare void @llvm.SI.export(i32, i32, i32, i32, i32, i32, i32, i32, i32)

attributes #0 = { "ShaderType"="0" "enable-no-nans-fp-math"="true" }

[mareko]

Please ignore what I said. I've talked with a Catalyst guy and he said we should emit exports as late in the shader as possible. The export memory can be allocated in the first export instruction and it limits the number of waves that can be run from that point. It's okay to reorder exports, but please don't emit them too early.

[arsenm]

In D12067#269437, @mareko wrote:

Please ignore what I said. I've talked with a Catalyst guy and he said we should emit exports as late in the shader as possible. The export memory can be allocated in the first export instruction and it limits the number of waves that can be run from that point. It's okay to reorder exports, but please don't emit them too early.

This patch shouldn't effect the scheduling of exports much if any

[arsenm]

What needs to be done on this? The cannot select error is a frontend bug

[arsenm]

Update for trunk. Set mayLoad = 1 for EXP_DONE instead of hasSideEffects

[arsenm]

Change integer type

[mareko]

Could you briefly describe what effect this will have on behavior? (e.g. scheduling)

[axeldavy]

In D12067#484345, @arsenm wrote:

Update for trunk. Set mayLoad = 1 for EXP_DONE instead of hasSideEffects

Could you explain why ? To me it looks like hasSideEffects is the only way to guarantee the exp with the done bit is last.

Btw you forgot to update the comments in the part "Export opcodes" to reflect that change.

[arsenm]

In D12067#486407, @axeldavy wrote:

In D12067#484345, @arsenm wrote:

Update for trunk. Set mayLoad = 1 for EXP_DONE instead of hasSideEffects

Could you explain why ? To me it looks like hasSideEffects is the only way to guarantee the exp with the done bit is last.

Btw you forgot to update the comments in the part "Export opcodes" to reflect that change.

If exp_done is reading from unknown memory, it can't be reordered before the stores to unknown memory

[arsenm]

In D12067#486222, @mareko wrote:

Could you briefly describe what effect this will have on behavior? (e.g. scheduling)

I don't think it should have that much impact on scheduling, there might be some changes around the done part

[arsenm]

In D12067#486407, @axeldavy wrote:

In D12067#484345, @arsenm wrote:

Update for trunk. Set mayLoad = 1 for EXP_DONE instead of hasSideEffects

Could you explain why ? To me it looks like hasSideEffects is the only way to guarantee the exp with the done bit is last.

Btw you forgot to update the comments in the part "Export opcodes" to reflect that change.

I see some regressions with the mayLoad bit set for exp_done, but without it I see minimal changes: https://ghostbin.com/paste/b6ypo

I can drop the separate bit setting part of this patch for now and revisit it later

[arsenm]

Update for trunk

[arsenm]

Reorder operands

[nhaehnle]

The TODOs need to be addressed, and for the FIXMEs in the intrinsic definition, I'd say we're better off just adding a real llvm.amdgcn.export (+ .done?) intrinsic in the future. Some more inline comments, but overall I think this patch is fine.

lib/Target/AMDGPU/AMDGPUInstrInfo.td

288–290

SDNPMayLoad instead of SDNPSideEffect for consistency.

lib/Target/AMDGPU/SIInstrInfo.td

530–554

s/hasSideEffects/mayLoad/

[arsenm]

In D12067#599858, @nhaehnle wrote:

The TODOs need to be addressed, and for the FIXMEs in the intrinsic definition, I'd say we're better off just adding a real llvm.amdgcn.export (+ .done?) intrinsic in the future. Some more inline comments, but overall I think this patch is fine.

The only TODOs I see left are addressed in the D12068

[mareko]

FYI, I'm working on a pass that moves all EXP instructions to the end of shaders after the machine scheduler. We don't want any scheduling optimizations for those.

[arsenm]

Address comments, merge changes from other patch

[arsenm]

Attach right patch. Bit inputs are temporarily printed as -1 until the next patch which fixes the printing

[nhaehnle]

LGTM

[arsenm]

r288695 with waitcnt insert fix

lib/Target/AMDGPU/AMDGPUISelLowering.h

enum NodeType : unsigned {
   MUL_I24,
   MULHI_U24,
   MULHI_I24,
   MAD_U24,
   MAD_I24,
   MUL_LOHI_I24,
   MUL_LOHI_U24,
   TEXTURE_FETCH,
-  EXPORT,
+  EXPORT, // exp on SI+
+  EXPORT_DONE, // exp on SI+ with done bit set
+  R600_EXPORT,
   CONST_ADDRESS,
   REGISTER_LOAD,
   REGISTER_STORE,
   LOAD_INPUT,
   SAMPLE,
   SAMPLEB,
   SAMPLED,
   SAMPLEL,

lib/Target/AMDGPU/AMDGPUISelLowering.cpp

const char* AMDGPUTargetLowering::getTargetNodeName(unsigned Opcode) const {
   NODE_NAME_CASE(MULHI_U24)
   NODE_NAME_CASE(MULHI_I24)
   NODE_NAME_CASE(MUL_LOHI_U24)
   NODE_NAME_CASE(MUL_LOHI_I24)
   NODE_NAME_CASE(MAD_U24)
   NODE_NAME_CASE(MAD_I24)
   NODE_NAME_CASE(TEXTURE_FETCH)
   NODE_NAME_CASE(EXPORT)
+  NODE_NAME_CASE(EXPORT_DONE)
+  NODE_NAME_CASE(R600_EXPORT)
   NODE_NAME_CASE(CONST_ADDRESS)
   NODE_NAME_CASE(REGISTER_LOAD)
   NODE_NAME_CASE(REGISTER_STORE)
   NODE_NAME_CASE(LOAD_INPUT)
   NODE_NAME_CASE(SAMPLE)
   NODE_NAME_CASE(SAMPLEB)
   NODE_NAME_CASE(SAMPLED)
   NODE_NAME_CASE(SAMPLEL)

lib/Target/AMDGPU/AMDGPUInstrInfo.td

 def AMDGPUinterp_p1 : SDNode<"AMDGPUISD::INTERP_P1",
                       SDTypeProfile<1, 3, [SDTCisFP<0>]>,
                       [SDNPInGlue, SDNPOutGlue]>;
 
 def AMDGPUinterp_p2 : SDNode<"AMDGPUISD::INTERP_P2",
                       SDTypeProfile<1, 4, [SDTCisFP<0>]>,
                       [SDNPInGlue]>;
 
+
 def AMDGPUkill : SDNode<"AMDGPUISD::KILL", AMDGPUKillSDT,
   [SDNPHasChain, SDNPSideEffect]>;
 
+// SI+ export
+def AMDGPUExportOp : SDTypeProfile<0, 8, [
+  SDTCisInt<0>, // i8 en
+  SDTCisInt<1>, // i1 vm
+  // skip done
+  SDTCisInt<2>, // i8 tgt
+  SDTCisSameAs<3, 1>, // i1 compr
+  SDTCisFP<4>,        // f32 src0
+  SDTCisSameAs<5, 4>, // f32 src1
+  SDTCisSameAs<6, 4>, // f32 src2
+  SDTCisSameAs<7, 4>  // f32 src3
+]>;
+
+def AMDGPUexport: SDNode<"AMDGPUISD::EXPORT", AMDGPUExportOp,
+  [SDNPHasChain, SDNPMayStore]>;
+
+def AMDGPUexport_done: SDNode<"AMDGPUISD::EXPORT_DONE", AMDGPUExportOp,
+  [SDNPHasChain, SDNPMayLoad, SDNPMayStore]>;
+
+
+def R600ExportOp : SDTypeProfile<0, 7, [SDTCisFP<0>, SDTCisInt<1>]>;
+
+def R600_EXPORT: SDNode<"AMDGPUISD::R600_EXPORT", R600ExportOp,
+  [SDNPHasChain, SDNPSideEffect]>;
+
 //===----------------------------------------------------------------------===//
 // Flow Control Profile Types
 //===----------------------------------------------------------------------===//
 // Branch instruction where second and third are basic blocks
 def SDTIL_BRCond : SDTypeProfile<0, 2, [
     SDTCisVT<0, OtherVT>
     ]>;
 

lib/Target/AMDGPU/R600ISelLowering.cpp

case AMDGPUIntrinsic::r600_store_swizzle: {
         Op.getOperand(2), // Export Value
         Op.getOperand(3), // ArrayBase
         Op.getOperand(4), // Type
         DAG.getConstant(0, DL, MVT::i32), // SWZ_X
         DAG.getConstant(1, DL, MVT::i32), // SWZ_Y
         DAG.getConstant(2, DL, MVT::i32), // SWZ_Z
         DAG.getConstant(3, DL, MVT::i32) // SWZ_W
       };
-      return DAG.getNode(AMDGPUISD::EXPORT, DL, Op.getValueType(), Args);
+      return DAG.getNode(AMDGPUISD::R600_EXPORT, DL, Op.getValueType(), Args);
     }
 
     // default for switch(IntrinsicID)
     default: break;
     }
     // break out of case ISD::INTRINSIC_VOID in switch(Op.getOpcode())
     break;
   }
case ISD::SETEQ: {
                                LHS.getOperand(3),
                                LHSCC);
       break;
     }
     }
     return SDValue();
   }
 
-  case AMDGPUISD::EXPORT: {
+  case AMDGPUISD::R600_EXPORT: {
     SDValue Arg = N->getOperand(1);
     if (Arg.getOpcode() != ISD::BUILD_VECTOR)
       break;
 
     SDValue NewArgs[8] = {
       N->getOperand(0), // Chain
       SDValue(),
       N->getOperand(2), // ArrayBase
       N->getOperand(3), // Type
       N->getOperand(4), // SWZ_X
       N->getOperand(5), // SWZ_Y
       N->getOperand(6), // SWZ_Z
       N->getOperand(7) // SWZ_W
     };
     NewArgs[1] = OptimizeSwizzle(N->getOperand(1), &NewArgs[4], DAG, DL);
-    return DAG.getNode(AMDGPUISD::EXPORT, DL, N->getVTList(), NewArgs);
+    return DAG.getNode(AMDGPUISD::R600_EXPORT, DL, N->getVTList(), NewArgs);
   }
   case AMDGPUISD::TEXTURE_FETCH: {
     SDValue Arg = N->getOperand(1);
     if (Arg.getOpcode() != ISD::BUILD_VECTOR)
       break;
 
     SDValue NewArgs[19] = {
       N->getOperand(0),

lib/Target/AMDGPU/R600Instructions.td

 }
 
 def INTERP_LOAD_P0 : R600_1OP <0xE0, "INTERP_LOAD_P0", []>;
 
 //===----------------------------------------------------------------------===//
 // Export Instructions
 //===----------------------------------------------------------------------===//
 
-def ExportType : SDTypeProfile<0, 7, [SDTCisFP<0>, SDTCisInt<1>]>;
-
-def EXPORT: SDNode<"AMDGPUISD::EXPORT", ExportType,
-  [SDNPHasChain, SDNPSideEffect]>;
-
 class ExportWord0 {
   field bits<32> Word0;
 
   bits<13> arraybase;
   bits<2> type;
   bits<7> gpr;
   bits<2> elem_size;
 
class ExportBufWord1 {
   bits<1> eop;
   bits<8> inst;
 
   let Word1{11-0} = arraySize;
   let Word1{15-12} = compMask;
 }
 
 multiclass ExportPattern<Instruction ExportInst, bits<8> cf_inst> {
-  def : Pat<(EXPORT (v4f32 R600_Reg128:$src), (i32 imm:$base), (i32 imm:$type),
+  def : Pat<(R600_EXPORT (v4f32 R600_Reg128:$src), (i32 imm:$base), (i32 imm:$type),
     (i32 imm:$swz_x), (i32 imm:$swz_y), (i32 imm:$swz_z), (i32 imm:$swz_w)),
         (ExportInst R600_Reg128:$src, imm:$type, imm:$base,
         imm:$swz_x, imm:$swz_y, imm:$swz_z, imm:$swz_w, cf_inst, 0)
   >;
 
 }
 
 multiclass SteamOutputExportPattern<Instruction ExportInst,

lib/Target/AMDGPU/SIISelLowering.cpp

if (const ConstantFPSDNode *K = dyn_cast<ConstantFPSDNode>(Src)) {
 
       SDValue NegOne = DAG.getTargetConstant(FloatToBits(-1.0f), DL, MVT::i32);
       return DAG.getNode(AMDGPUISD::KILL, DL, MVT::Other, Chain, NegOne);
     }
 
     SDValue Cast = DAG.getNode(ISD::BITCAST, DL, MVT::i32, Src);
     return DAG.getNode(AMDGPUISD::KILL, DL, MVT::Other, Chain, Cast);
   }
+  case AMDGPUIntrinsic::SI_export: {
+    const ConstantSDNode *En = cast<ConstantSDNode>(Op.getOperand(2));
+    const ConstantSDNode *VM = cast<ConstantSDNode>(Op.getOperand(3));
+    const ConstantSDNode *Done = cast<ConstantSDNode>(Op.getOperand(4));
+    const ConstantSDNode *Tgt = cast<ConstantSDNode>(Op.getOperand(5));
+    const ConstantSDNode *Compr = cast<ConstantSDNode>(Op.getOperand(6));
+
+    const SDValue Ops[] = {
+      Chain,
+      DAG.getTargetConstant(En->getZExtValue(), DL, MVT::i8),
+      DAG.getTargetConstant(VM->getZExtValue(), DL, MVT::i1),
+      DAG.getTargetConstant(Tgt->getZExtValue(), DL, MVT::i8),
+      DAG.getTargetConstant(Compr->getZExtValue(), DL, MVT::i1),
+      Op.getOperand(7), // src0
+      Op.getOperand(8), // src1
+      Op.getOperand(9), // src2
+      Op.getOperand(10) // src3
+    };
+
+    unsigned Opc = Done->isNullValue() ?
+      AMDGPUISD::EXPORT : AMDGPUISD::EXPORT_DONE;
+    return DAG.getNode(Opc, DL, Op->getVTList(), Ops);
+  }
   default:
     return SDValue();
   }
 }
 
 SDValue SITargetLowering::LowerLOAD(SDValue Op, SelectionDAG &DAG) const {
   SDLoc DL(Op);
   LoadSDNode *Load = cast<LoadSDNode>(Op);

lib/Target/AMDGPU/SIInsertSkips.cpp

bool SIInsertSkips::skipIfDead(MachineInstr &MI, MachineBasicBlock &NextBB) {
 
   // If the exec mask is non-zero, skip the next two instructions
   BuildMI(&MBB, DL, TII->get(AMDGPU::S_CBRANCH_EXECNZ))
     .addMBB(&NextBB);
 
   MachineBasicBlock::iterator Insert = SkipBB->begin();
 
   // Exec mask is zero: Export to NULL target...
-  BuildMI(*SkipBB, Insert, DL, TII->get(AMDGPU::EXP))
-    .addImm(0)
+  BuildMI(*SkipBB, Insert, DL, TII->get(AMDGPU::EXP_DONE))
     .addImm(0x09) // V_008DFC_SQ_EXP_NULL
-    .addImm(0)
-    .addImm(1)
-    .addImm(1)
     .addReg(AMDGPU::VGPR0, RegState::Undef)
     .addReg(AMDGPU::VGPR0, RegState::Undef)
     .addReg(AMDGPU::VGPR0, RegState::Undef)
-    .addReg(AMDGPU::VGPR0, RegState::Undef);
+    .addReg(AMDGPU::VGPR0, RegState::Undef)
+    .addImm(1)  // vm
+    .addImm(0)  // compr
+    .addImm(0); // en
 
   // ... and terminate wavefront.
   BuildMI(*SkipBB, Insert, DL, TII->get(AMDGPU::S_ENDPGM));
 
   return true;
 }
 
 void SIInsertSkips::kill(MachineInstr &MI) {

lib/Target/AMDGPU/SIInsertWaits.cpp

 
 Counters SIInsertWaits::getHwCounts(MachineInstr &MI) {
   uint64_t TSFlags = MI.getDesc().TSFlags;
   Counters Result = { { 0, 0, 0 } };
 
   Result.Named.VM = !!(TSFlags & SIInstrFlags::VM_CNT);
 
   // Only consider stores or EXP for EXP_CNT
-  Result.Named.EXP = !!(TSFlags & SIInstrFlags::EXP_CNT &&
-      (MI.getOpcode() == AMDGPU::EXP || MI.getDesc().mayStore()));
+  Result.Named.EXP = !!(TSFlags & SIInstrFlags::EXP_CNT) && MI.mayStore();
 
   // LGKM may uses larger values
   if (TSFlags & SIInstrFlags::LGKM_CNT) {
 
     if (TII->isSMRD(MI)) {
 
       if (MI.getNumOperands() != 0) {
         assert(MI.getOperand(0).isReg() &&
if (ST->getGeneration() >= SISubtarget::VOLCANIC_ISLANDS) {
     if (TII->isSMRD(*I))
       LastOpcodeType = SMEM;
     else if (Increment.Named.VM)
       LastOpcodeType = VMEM;
   }
 
   // Remember which export instructions we have seen
   if (Increment.Named.EXP) {
-    ExpInstrTypesSeen |= I->getOpcode() == AMDGPU::EXP ? 1 : 2;
+    ExpInstrTypesSeen |=
+      (I->getOpcode() == AMDGPU::EXP ||  I->getOpcode() == AMDGPU::EXP_DONE) ? 1 : 2;
   }
 
   for (unsigned i = 0, e = I->getNumOperands(); i != e; ++i) {
     MachineOperand &Op = I->getOperand(i);
     if (!isOpRelevant(Op))
       continue;
 
     const TargetRegisterClass *RC = TII->getOpRegClass(*I, i);

lib/Target/AMDGPU/SIInstrFormats.td

 
 class VINTRPCommon <dag outs, dag ins, string asm, list<dag> pattern> :
     InstSI <outs, ins, asm, pattern> {
   let mayLoad = 1;
   let mayStore = 0;
   let hasSideEffects = 0;
 }
 
+class EXPCommon<dag outs, dag ins, string asm, list<dag> pattern> :
+  InstSI<outs, ins, asm, pattern> {
+  let EXP_CNT = 1;
+  let mayLoad = 0; // Set to 1 if done bit is set.
+  let mayStore = 1;
+  let UseNamedOperandTable = 1;
+  let Uses = [EXEC];
+  let SchedRW = [WriteExport];
+}
+
 } // End Uses = [EXEC]
 
 class MIMG <dag outs, dag ins, string asm, list<dag> pattern> :
     InstSI <outs, ins, asm, pattern> {
 
   let VM_CNT = 1;
   let EXP_CNT = 1;
   let MIMG = 1;
   let Uses = [EXEC];
 
   let UseNamedOperandTable = 1;
   let hasSideEffects = 0; // XXX ????
 }

lib/Target/AMDGPU/SIInstrInfo.td

class SIMCInstr <string pseudo, int subtarget> {
   string PseudoInstr = pseudo;
   int Subtarget = subtarget;
 }
 
 //===----------------------------------------------------------------------===//
 // EXP classes
 //===----------------------------------------------------------------------===//
 
-class EXPCommon : InstSI<
+class EXP_Helper<bit done, SDPatternOperator node = null_frag> : EXPCommon<
   (outs),
-  (ins i32imm:$en, i32imm:$tgt, i32imm:$compr, i32imm:$done, i32imm:$vm,
-       VGPR_32:$src0, VGPR_32:$src1, VGPR_32:$src2, VGPR_32:$src3),
-  "exp $en, $tgt, $compr, $done, $vm, $src0, $src1, $src2, $src3",
-  [] > {
-
-  let EXP_CNT = 1;
-  let Uses = [EXEC];
-  let SchedRW = [WriteExport];
-}
-
-multiclass EXP_m {
+  (ins i8imm:$tgt, VGPR_32:$src0, VGPR_32:$src1, VGPR_32:$src2, VGPR_32:$src3,
+       i1imm:$vm, i1imm:$compr, i8imm:$en),
+  "exp $en, $tgt, $compr, "#!if(done, "1", "0")#", $vm, $src0, $src1, $src2, $src3",
+  [(node (i8 timm:$en), (i1 timm:$vm), (i8 timm:$tgt), (i1 timm:$compr),
+         f32:$src0, f32:$src1, f32:$src2, f32:$src3)]
+>;
 
+// Split EXP instruction into EXP and EXP_DONE so we can set
+// mayLoad for done=1.
+multiclass EXP_m<bit done, SDPatternOperator node> {
+  let mayLoad = done in {
     let isPseudo = 1, isCodeGenOnly = 1 in {
-    def "" : EXPCommon, SIMCInstr <"exp", SIEncodingFamily.NONE> ;
+      def "" : EXP_Helper<done, node>,
+               SIMCInstr <"exp"#!if(done, "_done", ""), SIEncodingFamily.NONE>;
     }
 
-  def _si : EXPCommon, SIMCInstr <"exp", SIEncodingFamily.SI>, EXPe {
-    let DecoderNamespace="SICI";
+    let done = done in {
+      def _si : EXP_Helper<done>,
+                SIMCInstr <"exp"#!if(done, "_done", ""), SIEncodingFamily.SI>,
+                EXPe {
+        let DecoderNamespace = "SICI";
         let DisableDecoder = DisableSIDecoder;
       }
 
-  def _vi : EXPCommon, SIMCInstr <"exp", SIEncodingFamily.VI>, EXPe_vi {
-    let DecoderNamespace="VI";
+      def _vi : EXP_Helper<done>,
+                SIMCInstr <"exp"#!if(done, "_done", ""), SIEncodingFamily.VI>,
+                EXPe_vi {
+        let DecoderNamespace = "VI";
         let DisableDecoder = DisableVIDecoder;
       }
     }
+  }
+}
 
 //===----------------------------------------------------------------------===//
 // Vector ALU classes
 //===----------------------------------------------------------------------===//
 
 class getNumSrcArgs<ValueType Src0, ValueType Src1, ValueType Src2> {
   int ret =
     !if (!eq(Src0.Value, untyped.Value),      0,

lib/Target/AMDGPU/SIInstructions.td

 include "BUFInstructions.td"
 
 let SubtargetPredicate = isGCN in {
 
 //===----------------------------------------------------------------------===//
 // EXP Instructions
 //===----------------------------------------------------------------------===//
 
-defm EXP : EXP_m;
+defm EXP : EXP_m<0, AMDGPUexport>;
+defm EXP_DONE : EXP_m<1, AMDGPUexport_done>;
 
 //===----------------------------------------------------------------------===//
 // VINTRP Instructions
 //===----------------------------------------------------------------------===//
 
 let Uses = [M0, EXEC] in {
 
 // FIXME: Specify SchedRW for VINTRP insturctions.
def : Pat<
   (SI_ELSE $src, $target, 0)
 >;
 
 def : Pat <
   (int_AMDGPU_kilp),
   (SI_KILL (i32 0xbf800000))
 >;
 
-def : Pat <
-  (int_SI_export imm:$en, imm:$vm, imm:$done, imm:$tgt, imm:$compr,
-                 f32:$src0, f32:$src1, f32:$src2, f32:$src3),
-  (EXP imm:$en, imm:$tgt, imm:$compr, imm:$done, imm:$vm,
-       $src0, $src1, $src2, $src3)
->;
-
 //===----------------------------------------------------------------------===//
 // VOP1 Patterns
 //===----------------------------------------------------------------------===//
 
 let Predicates = [UnsafeFPMath] in {
 
 //def : RcpPat<V_RCP_F64_e32, f64>;
 //defm : RsqPat<V_RSQ_F64_e32, f64>;

lib/Target/AMDGPU/SIIntrinsics.td

 // Backend internal SI Intrinsic Definitions. User code should not
 // directly use these.
 //
 //===----------------------------------------------------------------------===//
 
 
 let TargetPrefix = "SI", isTarget = 1 in {
   def int_SI_packf16 : Intrinsic <[llvm_i32_ty], [llvm_float_ty, llvm_float_ty], [IntrNoMem]>;
-  def int_SI_export : Intrinsic <[], [llvm_i32_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty, llvm_float_ty, llvm_float_ty, llvm_float_ty, llvm_float_ty], []>;
+
+  def int_SI_export : Intrinsic <[],
+    [llvm_i32_ty,   // en
+    llvm_i32_ty,    // vm   (FIXME: should be i1)
+    llvm_i32_ty,    // done (FIXME: should be i1)
+    llvm_i32_ty,    // tgt
+    llvm_i32_ty,    // compr (FIXME: should be i1)
+    llvm_float_ty,  // src0
+    llvm_float_ty,  // src1
+    llvm_float_ty,  // src2
+    llvm_float_ty], // src3
+    []
+  >;
+
   def int_SI_load_const : Intrinsic <[llvm_float_ty], [llvm_anyint_ty, llvm_i32_ty], [IntrNoMem]>;
   def int_SI_vs_load_input : Intrinsic <[llvm_v4f32_ty], [llvm_anyint_ty, llvm_i16_ty, llvm_i32_ty], [IntrNoMem]> ;
 
   // Fully-flexible TBUFFER_STORE_FORMAT_* except for the ADDR64 bit, which is not exposed
   def int_SI_tbuffer_store : Intrinsic <
     [],
     [llvm_anyint_ty, // rsrc(SGPR)
      llvm_anyint_ty, // vdata(VGPR), overloaded for types i32, v2i32, v4i32

test/CodeGen/AMDGPU/ret.ll

 ; RUN: llc -march=amdgcn -mcpu=tahiti -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 ; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
 
 declare void @llvm.SI.export(i32, i32, i32, i32, i32, float, float, float, float)
 
 ; GCN-LABEL: {{^}}vgpr:
 ; GCN: v_mov_b32_e32 v1, v0
 ; GCN-DAG: v_add_f32_e32 v0, 1.0, v1
-; GCN-DAG: exp 15, 0, 1, 1, 1, v1, v1, v1, v1
+; GCN-DAG: exp 15, 0, -1, 1, -1, v1, v1, v1, v1
 ; GCN: s_waitcnt expcnt(0)
 ; GCN-NOT: s_endpgm
 define amdgpu_vs {float, float} @vgpr([9 x <16 x i8>] addrspace(2)* byval, i32 inreg, i32 inreg, float) {
   call void @llvm.SI.export(i32 15, i32 1, i32 1, i32 0, i32 1, float %3, float %3, float %3, float %3)
   %x = fadd float %3, 1.0
   %a = insertvalue {float, float} undef, float %x, 0
   %b = insertvalue {float, float} %a, float %3, 1
   ret {float, float} %b
 }
 
 ; GCN-LABEL: {{^}}vgpr_literal:
 ; GCN: v_mov_b32_e32 v4, v0
-; GCN: exp 15, 0, 1, 1, 1, v4, v4, v4, v4
+; GCN: exp 15, 0, -1, 1, -1, v4, v4, v4, v4
 ; GCN-DAG: v_mov_b32_e32 v0, 1.0
 ; GCN-DAG: v_mov_b32_e32 v1, 2.0
 ; GCN-DAG: v_mov_b32_e32 v2, 4.0
 ; GCN-DAG: v_mov_b32_e32 v3, -1.0
 ; GCN: s_waitcnt expcnt(0)
 ; GCN-NOT: s_endpgm
 define amdgpu_vs {float, float, float, float} @vgpr_literal([9 x <16 x i8>] addrspace(2)* byval, i32 inreg, i32 inreg, float) {
   call void @llvm.SI.export(i32 15, i32 1, i32 1, i32 0, i32 1, float %3, float %3, float %3, float %3)

 define amdgpu_vs {i32, i32, i32, i32} @sgpr_literal([9 x <16 x i8>] addrspace(2)* byval, i32 inreg, i32 inreg, float) {
   %x = add i32 %2, 2
   ret {i32, i32, i32, i32} {i32 5, i32 6, i32 7, i32 8}
 }
 
 
 ; GCN-LABEL: {{^}}both:
 ; GCN: v_mov_b32_e32 v1, v0
-; GCN-DAG: exp 15, 0, 1, 1, 1, v1, v1, v1, v1
+; GCN-DAG: exp 15, 0, -1, 1, -1, v1, v1, v1, v1
 ; GCN-DAG: v_add_f32_e32 v0, 1.0, v1
 ; GCN-DAG: s_add_i32 s0, s3, 2
 ; GCN-DAG: s_mov_b32 s1, s2
 ; GCN: s_mov_b32 s2, s3
 ; GCN: s_waitcnt expcnt(0)
 ; GCN-NOT: s_endpgm
 define amdgpu_vs {float, i32, float, i32, i32} @both([9 x <16 x i8>] addrspace(2)* byval, i32 inreg, i32 inreg, float) {
   call void @llvm.SI.export(i32 15, i32 1, i32 1, i32 0, i32 1, float %3, float %3, float %3, float %3)
   %v = fadd float %3, 1.0
   %s = add i32 %2, 2
   %a0 = insertvalue {float, i32, float, i32, i32} undef, float %v, 0
   %a1 = insertvalue {float, i32, float, i32, i32} %a0, i32 %s, 1
   %a2 = insertvalue {float, i32, float, i32, i32} %a1, float %3, 2
   %a3 = insertvalue {float, i32, float, i32, i32} %a2, i32 %1, 3
   %a4 = insertvalue {float, i32, float, i32, i32} %a3, i32 %2, 4
   ret {float, i32, float, i32, i32} %a4
 }
 
 
 ; GCN-LABEL: {{^}}structure_literal:
 ; GCN: v_mov_b32_e32 v3, v0
-; GCN: exp 15, 0, 1, 1, 1, v3, v3, v3, v3
+; GCN: exp 15, 0, -1, 1, -1, v3, v3, v3, v3
 ; GCN-DAG: v_mov_b32_e32 v0, 1.0
 ; GCN-DAG: s_mov_b32 s0, 2
 ; GCN-DAG: s_mov_b32 s1, 3
 ; GCN-DAG: v_mov_b32_e32 v1, 2.0
 ; GCN-DAG: v_mov_b32_e32 v2, 4.0
 ; GCN: s_waitcnt expcnt(0)
 define amdgpu_vs {{float, i32}, {i32, <2 x float>}} @structure_literal([9 x <16 x i8>] addrspace(2)* byval, i32 inreg, i32 inreg, float) {
   call void @llvm.SI.export(i32 15, i32 1, i32 1, i32 0, i32 1, float %3, float %3, float %3, float %3)
   ret {{float, i32}, {i32, <2 x float>}} {{float, i32} {float 1.0, i32 2}, {i32, <2 x float>} {i32 3, <2 x float> <float 2.0, float 4.0>}}
 }

test/CodeGen/MIR/AMDGPU/movrels-bug.mir

 tracksRegLiveness: true
 body:             |
   bb.0:
     %m0 = S_MOV_B32 undef %sgpr0
     V_MOVRELD_B32_e32 undef %vgpr2, 0, implicit %m0, implicit %exec, implicit-def %vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8, implicit undef %vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8(tied-def 4)
     %m0 = S_MOV_B32 undef %sgpr0
     %vgpr1 = V_MOVRELS_B32_e32 undef %vgpr1, implicit %m0, implicit %exec, implicit killed %vgpr1_vgpr2_vgpr3_vgpr4_vgpr5_vgpr6_vgpr7_vgpr8
     %vgpr4 = V_MAC_F32_e32 undef %vgpr0, undef %vgpr0, undef %vgpr4, implicit %exec
-    EXP 15, 12, 0, 1, 0, undef %vgpr0, killed %vgpr1, killed %vgpr4, undef %vgpr0, implicit %exec
+    EXP_DONE 15, undef %vgpr0, killed %vgpr1, killed %vgpr4, undef %vgpr0, 0, 0, 12, implicit %exec
     S_ENDPGM
 
 ...