[SystemZ] Improve foldMemoryOperandImpl: vec->FP conversions

[jonpa]

Summary

(copied from original post:)

The patch checks the other operands of the single-lane (W) vector instruction. If all are already allocated to an FP (<16) register, then a memfold pseudo for the mapped fp memory instruction is used (since the register allocator has already begun to spill, it seems less promising to try to constrain any not yet allocated register to FP registers and do the fold also in such cases).

New mappings are added from W... to FP instructions. Since any instruction that is mapped to a memory instruction in getMemOpcode() has to be correctly treated, I wanted to be sure that we only see expected opcodes by recognizing them all in foldMemoryOperandImpl() and then having a check to see that there are no stray vector instructions showing up. It seems to me now that it would probably be more reasonable to just check any instruction at that point for vector registers allocated >15 and in that case return nullptr. In other words trust the mapping in SystemZInstrFormats.td and allow any folding by it as long as there are no FP16-31 registers allocated. Waiting for some feedback before changing this...

The MemFoldCopies stat is still low and acceptable, I think: 37 on a SPEC 17 build. This is when the regalloc evicts one of the registers of the memfold pseudo and a COPY then later has to be built in SystemZPostRewrite.cpp.

Added commutation flags on WFA/WFM - I hope this is ok and also that there are no unwanted implications regarding fp semantics with this patch.

LIS->getRegUnit() will compute the LiveInterval for CC. This is now needed in two places so it was moved to the top of the function and so always called (should not be a compile time problem). Because of this, the kill flags on CC in int-cmp-56.mir are removed (which seems to always be done during regalloc).

[jonpa]

This is a patch on-top of the other part of this improvement, so the diff is not against trunk exclusively, although nearly all of it should be...

[jonpa]

Patch rebased

[jonpa]

patch rebased.

[jonpa]

• Added a new test case that shows the use of a VR128 virtual register operand. This is fun12 in foldmemop-vec-cmp.mir, where a float 1.0 is generated with a VGMF.

• As discussed, the new mapping between vector instructions and FP instructions is closely related to an already existing mapping in SystemZShortenInst.cpp, and it might be nice to merge them.

The patch currently adds this type of mapping:

WFADB -> ADB_MemFoldPseudo -> ADB
WFCDB                      -> CDB
WFSQDB                     -> SQDB

SystemZShorten uses a mapping to the FP register opcode:

WFADB  -> ADBR
WFCDB  -> CDBR
WFSQDB -> SQDBR

If we added a generated function getRegOpcode(), we could in Shortening use that to do

WFADB -> ADB_MemFoldPseudo -> ADB -> ADBR

If we instead replaced the mapping in Shortening with a generated function getFPOpcode(), foldMemoryOperandImpl() would need a new getPseudoMemOpcode() function to do:

WFADB -> ADBR -> ADB -> ADB_MemFoldPseudo

The existing getTargetMemOpcode() would still be needed in PostRewrite.

On the other hand, looking at this patch it might be possible to skip the added opcode checks and deduce that information for foldMemoryOperandImpl() looking at the register classes and number of operands.

[jonpa]

Instead of checking a lot of newly mapped opcodes, use a general heuristic that simply demands that any operand which is a vector register per the descriptor must be allocated to an FP register. The opcodes for the fused fp instructions are still checked for explicitly, even though it might be ok to count the number of operands and their register classes.

Some minor fixing as well compared to last time.

On SPEC'17, I see:

ld             :                70304                67606    -2698
adb            :                13118                14401    +1283
lghi           :               463193               464385    +1192
lay            :                71431                70314    -1117
sdb            :                 4039                 5063    +1024
sdbr           :                 7641                 6640    -1001
adbr           :                20458                19473     -985
mdb            :                17530                17927     +397
wfmdb          :                39703                39374     -329
wfadb          :                24564                24266     -298
lde            :                52890                52603     -287
vlrepg         :                19277                19187      -90
aeb            :                10104                10188      +84
meeb           :                18323                18404      +81
ddb            :                  676                  755      +79
ldy            :                 3602                 3526      -76
ddbr           :                 2736                 2661      -75
cdb            :                 3556                 3627      +71
cebr           :                15036                14966      -70
...

, with -ffp-contract=fast:

ld             :                77018                75490    -1528
adb            :                 9949                10576     +627
adbr           :                 8877                 8369     -508
madb           :                 6408                 6879     +471
madbr          :                 9262                 8895     -367
lghi           :               463190               463500     +310
lay            :                72607                72306     -301
lde            :                56246                55963     -283
mdb            :                 7795                 8037     +242
wfmdb          :                23881                23686     -195
wfadb          :                11031                10912     -119
wfmadb         :                18341                18237     -104
cdb            :                 3556                 3638      +82
ddbr           :                 2723                 2641      -82
ddb            :                  676                  758      +82
cdbr           :                12034                11957      -77
ceb            :                 8184                 8257      +73
cebr           :                15044                14971      -73
vlrepg         :                19038                18970      -68
...

[jonpa]

Remove new _r instruction classes. This did not change the resulting tablegen mappings.

[jonpa]

Maybe there should be a test that checks that a live CC value isn't clobbered by introducing an FP-mem instruction, such as ADB...

[uweigand]

In D76705#2007796, @jonpa wrote:

Maybe there should be a test that checks that a live CC value isn't clobbered by introducing an FP-mem instruction, such as ADB...

Yes, of course -- but isn't that already in your patch (line 1184)?

[uweigand]

Looks good to me now, just one cosmetic comment inline.

And also a question about this:

LIS->getRegUnit() will compute the LiveInterval for CC. This is now needed in two places so it was moved to the top of the function and so always called (should not be a compile time problem). Because of this, the kill flags on CC in int-cmp-56.mir are removed (which seems to always be done during regalloc).

I don't think I understand this -- looking at the code in question, CC does appear to be killed in that instruction. Is this not right? If it is, why does recomputing live intervals remove the flag?

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

2923

So it is a bit weird that the vec->fp renaming is done here for Unary and Compare, but is not done (and instead done in the reverse direction by the MemFold pseudos) for Binary and Ternary. It would be nicer if this were symmetric.

[jonpa]

Added test case for a live CC: foldmemop-vec-cc.mir

[jonpa]

Yes, of course -- but isn't that already in your patch (line 1184)?

Ah - I meant a test as in "test case", which I could not find in the new tests...

I don't think I understand this -- looking at the code in question, CC does appear to be killed in that instruction. Is this not right? If it is, why does recomputing live intervals remove the flag?

It will be eventually added again after regalloc (not until PostRA Machine Scheduler since it is not allocatable), but this particular test case only runs the greedy pass so that will be seen in the output.

So it is a bit weird that the vec->fp renaming is done here for Unary and Compare, but is not done (and instead done in the reverse direction by the MemFold pseudos) for Binary and Ternary. It would be nicer if this were symmetric.

It seems that (as long as we don't want to change existing mappings on the scalar instructions)

WFSQDB has to change the name since SQDBR is already mapped to SQDB:

SQDBR ->OpKey:SQDBR-> SQDB
WFSQDB ->OpKey:SQDBR-> SQDB

So in trying to do the OpKey processing on the register instruction side to make it more symmetric, I find that

ADBR is mapped to ADB, via OpKey:ADBR.

In order to map WFADB to ADB_MemFoldPseudo, I needed to remove "wf" and also use a "MemFold" string, since the plain "ADB" would map it to the target memory instruction.

The ternary instructions also similarly needed a "MemFold" key substring.

Is this better than previously?

[uweigand]

> Is this better than previously?

Well, not really :-)

Maybe the cleanest way would be to actually have MemFold pseudos for the *vector* instructions, and have those handle the remapping. So for example:

WFADB -> OpKey:WFADB -> WFADB_MemFold -> MemKey:ADB -> ADB
and then just the same for
WFSQDB ->OpKey:WFSQDB->WFSQDB_MemFold -> MemKey:SQDB -> SQDB

Does this make sense?

[jonpa]

Does this make sense?

I tried this... One drawback is that the flags of the memfold pseudo must now be carefully transferred from the target memory instruction. I found that only ADB and SDB sets CC, while MDB and DDB do not - it was nice before in a way to have all of that defined in only one place. So I had to use one more class that defines a pseudo that sets CC... I also had to cancel out the isCommutableFlag on the memfold-pseudo.

I am not sure exactly which way is the cleanest - what do you think?

[jonpa]

Reverted back to the first version of new instruction classes providing the mapping from W...-reg -> FP-mem instructions.

[uweigand]

In D76705#2020342, @jonpa wrote:

Reverted back to the first version of new instruction classes providing the mapping from W...-reg -> FP-mem instructions.

Thanks. Let my try one final attempt to simply this confusing logic :-)

Right now, we have:

• For UnaryVRRa and CompareVRRa, the OpKey holds the FP-reg instruction name (e.g. wldeb->ldebr or wfsqsb->sqebr), which the matches the "mem" OpKey for the normal FP-mem instruction.
• For BinaryVRRa and TernaryVRRe, the OpKey holds the W-reg instruction name, which matches the "mem" OpKey of the MemFold-pseudo, which has the FP-reg instruction name as MemKey.

Can we instead move the W->FP mnemonic conversion back into BinaryVRRa/TernaryVRRe instead, so that the OpKey for all these vector instructions consistently holds the FP-reg instruction name? The MemFold-pseudos would then have the same mnemonic as OpKey and MemKey.

Because that conversion is still awkward as it has to handle many not-quite-consistent name combinations, maybe we should then even make this explicit by giving UnaryVRRa and friends an optional "fp_mnemonic" operand, so that the mapping is fully explicit in the main .td file, e.g. like so:

def WLDEB : UnaryVRRa<"wldeb", 0xE7C4, any_fpextend, v64db, v32sb, 2, 8, "ldebr">;
def WFSQSB : UnaryVRRa<"wfsqsb", 0xE7CE, any_fsqrt, v32sb, v32sb, 2, 8, "sqebr">;

[uweigand]

Ah, that still runs into the same name collisions with the MemFold pattern as you pointed out earlier. Sorry for missing that!

So if we use an extra modifier in the OpKey for those case, then we're basically back to your version of Apr 28, 7:15 PM, except possibly with the explicit mnemonics in the .td file.

[uweigand]

Hmm, talking about MemFold, I'm wondering about this:
​ // Fused multiply and add/sub need to have the same dst and accumulator reg.

Given that this check is necessary, what's then the point of having a MemFold for those ternary instructions in the first place? Can't we then directly emit the target instruction?

[jonpa]

Can we instead move the W->FP mnemonic conversion back into BinaryVRRa/TernaryVRRe instead, so that the OpKey for all these vector instructions consistently holds the FP-reg instruction name? The MemFold-pseudos would then have the same mnemonic as OpKey and MemKey.
... Ah, that still runs into the same name collisions with the MemFold pattern as you pointed out earlier. Sorry for missing that!

Yeah, WFADB:ADBR would map to ADB, and not ADB_MemFoldPseudo which is what we need...

So if we use an extra modifier in the OpKey for those case, then we're basically back to your version of Apr 28, 7:15 PM, except possibly with the explicit mnemonics in the .td file.

Ok, went back to that version plus added the explicit fp_mnemonic fields, which seems nice to me as they eliminate those ugly string substitutions.

Hmm, talking about MemFold, I'm wondering about this:

​> // Fused multiply and add/sub need to have the same dst and accumulator reg.

Given that this check is necessary, what's then the point of having a MemFold for those ternary instructions in the first place? Can't we then directly emit the target instruction?

I believe this follows the same pattern as the other MemFold cases: A WFMADB has four register operands, which all may be different virtual registers at this point. In the case where we find that DstReg and AccReg has already been allocated to the same physical register, we proceed with a MemFold pseudo. The register allocator may still however evict live ranges and reallocate these operands, which is why we can't go directly to MADB.

[uweigand]

In D76705#2029327, @jonpa wrote:

Can we instead move the W->FP mnemonic conversion back into BinaryVRRa/TernaryVRRe instead, so that the OpKey for all these vector instructions consistently holds the FP-reg instruction name? The MemFold-pseudos would then have the same mnemonic as OpKey and MemKey.
... Ah, that still runs into the same name collisions with the MemFold pattern as you pointed out earlier. Sorry for missing that!

Yeah, WFADB:ADBR would map to ADB, and not ADB_MemFoldPseudo which is what we need...

So if we use an extra modifier in the OpKey for those case, then we're basically back to your version of Apr 28, 7:15 PM, except possibly with the explicit mnemonics in the .td file.

Ok, went back to that version plus added the explicit fp_mnemonic fields, which seems nice to me as they eliminate those ugly string substitutions.

Yes, this does look a lot more reasonable. Thanks for going through all those iterations!

Hmm, talking about MemFold, I'm wondering about this:

​> // Fused multiply and add/sub need to have the same dst and accumulator reg.

Given that this check is necessary, what's then the point of having a MemFold for those ternary instructions in the first place? Can't we then directly emit the target instruction?

I believe this follows the same pattern as the other MemFold cases: A WFMADB has four register operands, which all may be different virtual registers at this point. In the case where we find that DstReg and AccReg has already been allocated to the same physical register, we proceed with a MemFold pseudo. The register allocator may still however evict live ranges and reallocate these operands, which is why we can't go directly to MADB.

Ah, I see. Makes sense.

See one final inline comment, otherwise this now looks good to me.

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

4429

That first line setting OpKey seems superfluous?

[jonpa]

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

4429

oh, sorry...

Thanks for review!

llvm/lib/Target/SystemZ/SystemZInstrFP.td

 // Addition.
 let Uses = [FPC], mayRaiseFPException = 1,
     Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0xF in {
   let isCommutable = 1 in {
     def AEBR : BinaryRRE<"aebr", 0xB30A, any_fadd, FP32,  FP32>;
     def ADBR : BinaryRRE<"adbr", 0xB31A, any_fadd, FP64,  FP64>;
     def AXBR : BinaryRRE<"axbr", 0xB34A, any_fadd, FP128, FP128>;
   }
-  def AEB : BinaryRXE<"aeb", 0xED0A, any_fadd, FP32, load, 4>;
-  def ADB : BinaryRXE<"adb", 0xED1A, any_fadd, FP64, load, 8>;
+  defm AEB : BinaryRXEAndPseudo<"aeb", 0xED0A, any_fadd, FP32, load, 4>;
+  defm ADB : BinaryRXEAndPseudo<"adb", 0xED1A, any_fadd, FP64, load, 8>;
 }
 
 // Subtraction.
 let Uses = [FPC], mayRaiseFPException = 1,
     Defs = [CC], CCValues = 0xF, CompareZeroCCMask = 0xF in {
   def SEBR : BinaryRRE<"sebr", 0xB30B, any_fsub, FP32,  FP32>;
   def SDBR : BinaryRRE<"sdbr", 0xB31B, any_fsub, FP64,  FP64>;
   def SXBR : BinaryRRE<"sxbr", 0xB34B, any_fsub, FP128, FP128>;
 
-  def SEB : BinaryRXE<"seb",  0xED0B, any_fsub, FP32, load, 4>;
-  def SDB : BinaryRXE<"sdb",  0xED1B, any_fsub, FP64, load, 8>;
+  defm SEB : BinaryRXEAndPseudo<"seb",  0xED0B, any_fsub, FP32, load, 4>;
+  defm SDB : BinaryRXEAndPseudo<"sdb",  0xED1B, any_fsub, FP64, load, 8>;
 }
 
 // Multiplication.
 let Uses = [FPC], mayRaiseFPException = 1 in {
   let isCommutable = 1 in {
     def MEEBR : BinaryRRE<"meebr", 0xB317, any_fmul, FP32,  FP32>;
     def MDBR  : BinaryRRE<"mdbr",  0xB31C, any_fmul, FP64,  FP64>;
     def MXBR  : BinaryRRE<"mxbr",  0xB34C, any_fmul, FP128, FP128>;
   }
-  def MEEB : BinaryRXE<"meeb", 0xED17, any_fmul, FP32, load, 4>;
-  def MDB  : BinaryRXE<"mdb",  0xED1C, any_fmul, FP64, load, 8>;
+  defm MEEB : BinaryRXEAndPseudo<"meeb", 0xED17, any_fmul, FP32, load, 4>;
+  defm MDB  : BinaryRXEAndPseudo<"mdb",  0xED1C, any_fmul, FP64, load, 8>;
 }
 
 // f64 multiplication of two FP32 registers.
 let Uses = [FPC], mayRaiseFPException = 1 in
   def MDEBR : BinaryRRE<"mdebr", 0xB30C, null_frag, FP64, FP32>;
 def : Pat<(any_fmul (f64 (any_fpextend FP32:$src1)),
                     (f64 (any_fpextend FP32:$src2))),
           (MDEBR (INSERT_SUBREG (f64 (IMPLICIT_DEF)),
def : Pat<(any_fmul (f128 (any_fpextend FP64:$src1)),
             (MXDB (INSERT_SUBREG (f128 (IMPLICIT_DEF)), FP64:$src1, subreg_h64),
                   bdxaddr12only:$addr)>;
 
 // Fused multiply-add.
 let Uses = [FPC], mayRaiseFPException = 1 in {
   def MAEBR : TernaryRRD<"maebr", 0xB30E, z_any_fma, FP32, FP32>;
   def MADBR : TernaryRRD<"madbr", 0xB31E, z_any_fma, FP64, FP64>;
 
-  def MAEB : TernaryRXF<"maeb", 0xED0E, z_any_fma, FP32, FP32, load, 4>;
-  def MADB : TernaryRXF<"madb", 0xED1E, z_any_fma, FP64, FP64, load, 8>;
+  defm MAEB : TernaryRXFAndPseudo<"maeb", 0xED0E, z_any_fma, FP32, FP32, load, 4>;
+  defm MADB : TernaryRXFAndPseudo<"madb", 0xED1E, z_any_fma, FP64, FP64, load, 8>;
 }
 
 // Fused multiply-subtract.
 let Uses = [FPC], mayRaiseFPException = 1 in {
   def MSEBR : TernaryRRD<"msebr", 0xB30F, z_any_fms, FP32, FP32>;
   def MSDBR : TernaryRRD<"msdbr", 0xB31F, z_any_fms, FP64, FP64>;
 
-  def MSEB : TernaryRXF<"mseb", 0xED0F, z_any_fms, FP32, FP32, load, 4>;
-  def MSDB : TernaryRXF<"msdb", 0xED1F, z_any_fms, FP64, FP64, load, 8>;
+  defm MSEB : TernaryRXFAndPseudo<"mseb", 0xED0F, z_any_fms, FP32, FP32, load, 4>;
+  defm MSDB : TernaryRXFAndPseudo<"msdb", 0xED1F, z_any_fms, FP64, FP64, load, 8>;
 }
 
 // Division.
 let Uses = [FPC], mayRaiseFPException = 1 in {
   def DEBR : BinaryRRE<"debr", 0xB30D, any_fdiv, FP32,  FP32>;
   def DDBR : BinaryRRE<"ddbr", 0xB31D, any_fdiv, FP64,  FP64>;
   def DXBR : BinaryRRE<"dxbr", 0xB34D, any_fdiv, FP128, FP128>;
 
-  def DEB : BinaryRXE<"deb", 0xED0D, any_fdiv, FP32, load, 4>;
-  def DDB : BinaryRXE<"ddb", 0xED1D, any_fdiv, FP64, load, 8>;
+  defm DEB : BinaryRXEAndPseudo<"deb", 0xED0D, any_fdiv, FP32, load, 4>;
+  defm DDB : BinaryRXEAndPseudo<"ddb", 0xED1D, any_fdiv, FP64, load, 8>;
 }
 
 // Divide to integer.
 let Uses = [FPC], mayRaiseFPException = 1, Defs = [CC] in {
   def DIEBR : TernaryRRFb<"diebr", 0xB353, FP32, FP32, FP32>;
   def DIDBR : TernaryRRFb<"didbr", 0xB35B, FP64, FP64, FP64>;
 }
 

llvm/lib/Target/SystemZ/SystemZInstrFormats.td

class UnaryVRRa<string mnemonic, bits<16> opcode, SDPatternOperator operator,
                 TypedReg tr1, TypedReg tr2, bits<4> type = 0, bits<4> m4 = 0,
                 bits<4> m5 = 0>
   : InstVRRa<opcode, (outs tr1.op:$V1), (ins tr2.op:$V2),
              mnemonic#"\t$V1, $V2",
              [(set (tr1.vt tr1.op:$V1), (operator (tr2.vt tr2.op:$V2)))]> {
   let M3 = type;
   let M4 = m4;
   let M5 = m5;
+  let OpKey = !subst("w", "", !subst("wf", "", !subst("sb", "eb", mnemonic)))#"r"#
+              !subst("VR", "FP", !cast<string>(tr1.op));
+  let OpType = "reg";
 }
 
 class UnaryVRRaGeneric<string mnemonic, bits<16> opcode, bits<4> m4 = 0,
                        bits<4> m5 = 0>
   : InstVRRa<opcode, (outs VR128:$V1), (ins VR128:$V2, imm32zx4:$M3),
              mnemonic#"\t$V1, $V2, $M3", []> {
   let M4 = m4;
   let M5 = m5;
class BinaryVRRc<string mnemonic, bits<16> opcode, SDPatternOperator operator,
                  bits<4> m6 = 0>
   : InstVRRc<opcode, (outs tr1.op:$V1), (ins tr2.op:$V2, tr2.op:$V3),
              mnemonic#"\t$V1, $V2, $V3",
              [(set (tr1.vt tr1.op:$V1), (operator (tr2.vt tr2.op:$V2),
                                                   (tr2.vt tr2.op:$V3)))]> {
   let M4 = type;
   let M5 = m5;
   let M6 = m6;
+  let OpKey = !subst("meb", "meeb", !subst("wf", "",
+              !subst("sb", "eb", mnemonic)))
+              #"MemFold"#!subst("VR", "FP", !cast<string>(tr1.op));
+  let OpType = "reg";
 }
 
 class BinaryVRRcGeneric<string mnemonic, bits<16> opcode, bits<4> m5 = 0,
                         bits<4> m6 = 0>
   : InstVRRc<opcode, (outs VR128:$V1),
              (ins VR128:$V2, VR128:$V3, imm32zx4:$M4),
              mnemonic#"\t$V1, $V2, $V3, $M4", []> {
   let M5 = m5;
class CompareVRRa<string mnemonic, bits<16> opcode, SDPatternOperator operator,
                   TypedReg tr, bits<4> type>
   : InstVRRa<opcode, (outs), (ins tr.op:$V1, tr.op:$V2),
              mnemonic#"\t$V1, $V2",
              [(set CC, (operator (tr.vt tr.op:$V1), (tr.vt tr.op:$V2)))]> {
   let isCompare = 1;
   let M3 = type;
   let M4 = 0;
   let M5 = 0;
+  let OpKey = !subst("wf", "", !subst("sb", "eb", mnemonic))#"r"#
+              !subst("VR", "FP", !cast<string>(tr.op));
+  let OpType = "reg";
 }
 
 class CompareVRRaGeneric<string mnemonic, bits<16> opcode>
   : InstVRRa<opcode, (outs), (ins VR128:$V1, VR128:$V2, imm32zx4:$M3),
              mnemonic#"\t$V1, $V2, $M3", []> {
   let isCompare = 1;
   let M4 = 0;
   let M5 = 0;
class TernaryVRRe<string mnemonic, bits<16> opcode, SDPatternOperator operator,
   : InstVRRe<opcode, (outs tr1.op:$V1),
              (ins tr2.op:$V2, tr2.op:$V3, tr1.op:$V4),
              mnemonic#"\t$V1, $V2, $V3, $V4",
              [(set (tr1.vt tr1.op:$V1), (operator (tr2.vt tr2.op:$V2),
                                                   (tr2.vt tr2.op:$V3),
                                                   (tr1.vt tr1.op:$V4)))]> {
   let M5 = m5;
   let M6 = type;
+  let OpKey = mnemonic#tr1.op;
+  let OpKey = !subst("wf", "", !subst("sb", "eb", mnemonic))
+              #"MemFold"#!subst("VR", "FP", !cast<string>(tr1.op));
+  let OpType = "reg";
 }
 
 class TernaryVRReFloatGeneric<string mnemonic, bits<16> opcode>
   : InstVRRe<opcode, (outs VR128:$V1),
              (ins VR128:$V2, VR128:$V3, VR128:$V4, imm32zx4:$M5, imm32zx4:$M6),
              mnemonic#"\t$V1, $V2, $V3, $V4, $M5, $M6", []>;
 
 class TernaryVRSb<string mnemonic, bits<16> opcode, SDPatternOperator operator,
: Pseudo<(outs cls:$R1), (ins cls:$R2, mode:$XBD2), []> {
     let MemKey = mnemonic#cls;
     let MemType = "pseudo";
     let mayLoad = 1;
     let AccessBytes = bytes;
     let HasIndex = 1;
     let hasNoSchedulingInfo = 1;
 }
 
+// Same as MemFoldPseudo but for mapping a W... vector instruction
+class MemFoldPseudo_FP<string mnemonic, RegisterOperand cls, bits<5> bytes,
+                    AddressingMode mode>
+  : MemFoldPseudo<mnemonic, cls, bytes, mode> {
+    let OpKey = mnemonic#"MemFold"#cls;
+}
+
+class MemFoldPseudo_FPTern<string mnemonic, RegisterOperand cls, bits<5> bytes,
+                           AddressingMode mode>
+  : Pseudo<(outs cls:$R1), (ins cls:$R2, cls:$R3, mode:$XBD2), []> {
+    let OpKey = mnemonic#"MemFold"#cls;
+    let OpType = "mem";
+    let MemKey = mnemonic#cls;
+    let MemType = "pseudo";
+    let mayLoad = 1;
+    let AccessBytes = bytes;
+    let HasIndex = 1;
+    let hasNoSchedulingInfo = 1;
+}
+
 // Same as MemFoldPseudo but for Load On Condition with CC operands.
 class MemFoldPseudo_CondMove<string mnemonic, RegisterOperand cls, bits<5> bytes,
                              AddressingMode mode>
   : Pseudo<(outs cls:$R1),
            (ins cls:$R2, mode:$XBD2, cond4:$valid, cond4:$M3), []> {
     let OpKey = !subst("loc", "sel", mnemonic)#"r"#cls;
     let OpType = "mem";
     let MemKey = mnemonic#cls;

 //===----------------------------------------------------------------------===//
 // Multiclasses that emit both real and pseudo instructions
 //===----------------------------------------------------------------------===//
 
 multiclass BinaryRXYAndPseudo<string mnemonic, bits<16> opcode,
                               SDPatternOperator operator, RegisterOperand cls,
                               SDPatternOperator load, bits<5> bytes,
                               AddressingMode mode = bdxaddr20only> {
-
   def "" : BinaryRXY<mnemonic, opcode, operator, cls, load, bytes, mode> {
     let MemKey = mnemonic#cls;
     let MemType = "target";
   }
   let Has20BitOffset = 1 in
     def _MemFoldPseudo : MemFoldPseudo<mnemonic, cls, bytes, mode>;
 }
 
let DispKey = mnemonic # cls in {
     }
     let DispSize = "20" in
       def Y  : BinaryRXY<mnemonic#"y", rxyOpcode, operator, cls, load,
                          bytes, bdxaddr20pair>;
   }
   def _MemFoldPseudo : MemFoldPseudo<mnemonic, cls, bytes, bdxaddr12pair>;
 }
 
+multiclass BinaryRXEAndPseudo<string mnemonic, bits<16> opcode,
+                              SDPatternOperator operator, RegisterOperand cls,
+                              SDPatternOperator load, bits<5> bytes> {
+  def "" : BinaryRXE<mnemonic, opcode, operator, cls, load, bytes> {
+    let MemKey = mnemonic#cls;
+    let MemType = "target";
+  }
+  def _MemFoldPseudo : MemFoldPseudo_FP<mnemonic, cls, bytes, bdxaddr12pair>;
+}
+
+multiclass TernaryRXFAndPseudo<string mnemonic, bits<16> opcode,
+                               SDPatternOperator operator, RegisterOperand cls1,
+                               RegisterOperand cls2, SDPatternOperator load,
+                               bits<5> bytes> {
+  def "" : TernaryRXF<mnemonic, opcode, operator, cls1, cls2, load, bytes> {
+    let MemKey = mnemonic#cls1;
+    let MemType = "target";
+  }
+  def _MemFoldPseudo : MemFoldPseudo_FPTern<mnemonic, cls1, bytes, bdxaddr12pair>;
+}
+
 multiclass CondUnaryRSYPairAndMemFold<string mnemonic, bits<16> opcode,
                                       SDPatternOperator operator,
                                       RegisterOperand cls, bits<5> bytes,
                                       AddressingMode mode = bdaddr20only> {
   defm "" : CondUnaryRSYPair<mnemonic, opcode, operator, cls, bytes, mode>;
   def _MemFoldPseudo : MemFoldPseudo_CondMove<mnemonic, cls, bytes, mode>;
 }
 

llvm/lib/Target/SystemZ/SystemZInstrInfo.cpp

MachineInstr *SystemZInstrInfo::convertToThreeAddress(
   return nullptr;
 }
 
 MachineInstr *SystemZInstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
     MachineBasicBlock::iterator InsertPt, int FrameIndex,
     LiveIntervals *LIS, VirtRegMap *VRM) const {
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   unsigned Size = MFI.getObjectSize(FrameIndex);
   unsigned Opcode = MI.getOpcode();
 
-  if (Ops.size() == 2 && Ops[0] == 0 && Ops[1] == 1) {
-    if (LIS != nullptr && (Opcode == SystemZ::LA || Opcode == SystemZ::LAY) &&
-        isInt<8>(MI.getOperand(2).getImm()) && !MI.getOperand(3).getReg()) {
-
-      // Check CC liveness, since new instruction introduces a dead
-      // def of CC.
-      MCRegUnitIterator CCUnit(SystemZ::CC, TRI);
-      LiveRange &CCLiveRange = LIS->getRegUnit(*CCUnit);
-      ++CCUnit;
-      assert(!CCUnit.isValid() && "CC only has one reg unit.");
-      SlotIndex MISlot =
-          LIS->getSlotIndexes()->getInstructionIndex(MI).getRegSlot();
-      if (!CCLiveRange.liveAt(MISlot)) {
-        // LA(Y) %reg, CONST(%reg) -> AGSI %mem, CONST
-        MachineInstr *BuiltMI = BuildMI(*InsertPt->getParent(), InsertPt,
-                                        MI.getDebugLoc(), get(SystemZ::AGSI))
-                                    .addFrameIndex(FrameIndex)
-                                    .addImm(0)
-                                    .addImm(MI.getOperand(2).getImm());
-        BuiltMI->findRegisterDefOperand(SystemZ::CC)->setIsDead(true);
-        CCLiveRange.createDeadDef(MISlot, LIS->getVNInfoAllocator());
-        return BuiltMI;
-      }
-    }
+  // Check CC liveness if new instruction introduces a dead def of CC.
+  SlotIndex MISlot = SlotIndex();
+  LiveRange *CCLiveRange = nullptr;
+  bool CCLiveAtMI = true;
+  if (LIS) {
+    MISlot = LIS->getSlotIndexes()->getInstructionIndex(MI).getRegSlot();
+    CCLiveRange = &LIS->getRegUnit(*CCUnit);
+    CCLiveAtMI = CCLiveRange->liveAt(MISlot);
+  }
+
+  if (Ops.size() == 2 && Ops[0] == 0 && Ops[1] == 1) {
+    if (!CCLiveAtMI && (Opcode == SystemZ::LA || Opcode == SystemZ::LAY) &&
+        isInt<8>(MI.getOperand(2).getImm()) && !MI.getOperand(3).getReg()) {
+      CCLiveRange->createDeadDef(MISlot, LIS->getVNInfoAllocator());
     return nullptr;
   }
 
   // All other cases require a single operand.
   if (Ops.size() != 1)
     return nullptr;
 
   unsigned OpNum = Ops[0];
if (MMO->getSize() == Size && !MMO->isVolatile() && !MMO->isAtomic()) {
             .addFrameIndex(FrameIndex)
             .addImm(0)
             .addMemOperand(MMO);
       }
     }
   }
 
   // If the spilled operand is the final one or the instruction is
-  // commutable, try to change <INSN>R into <INSN>.
+  // commutable, try to change <INSN>R into <INSN>.  Don't introduce a def of
+  // CC if it is live and MI does not define it.
   unsigned NumOps = MI.getNumExplicitOperands();
   int MemOpcode = SystemZ::getMemOpcode(Opcode);
-  if (MemOpcode == -1)
+  if (MemOpcode == -1 ||
+      (CCLiveAtMI && !MI.definesRegister(SystemZ::CC) &&
+       get(MemOpcode).hasImplicitDefOfPhysReg(SystemZ::CC)))
     return nullptr;
 
+  // Check if all other vregs have a usable allocation in the case of vector
+  // to FP conversion.
+  const MCInstrDesc &MCID = MI.getDesc();
+  for (unsigned I = 0, E = MCID.getNumOperands(); I != E; ++I) {
+    const MCOperandInfo &MCOI = MCID.OpInfo[I];
+    if (MCOI.OperandType != MCOI::OPERAND_REGISTER || I == OpNum)
+      continue;
+    const TargetRegisterClass *RC = TRI->getRegClass(MCOI.RegClass);
+    if (RC == &SystemZ::VR32BitRegClass || RC == &SystemZ::VR64BitRegClass) {
+      Register Reg = MI.getOperand(I).getReg();
+      Register PhysReg = Register::isVirtualRegister(Reg)
+                             ? (VRM ? VRM->getPhys(Reg) : Register())
+                             : Reg;
+      if (!PhysReg ||
+          !(SystemZ::FP32BitRegClass.contains(PhysReg) ||
+            SystemZ::FP64BitRegClass.contains(PhysReg) ||
+            SystemZ::VF128BitRegClass.contains(PhysReg)))
+        return nullptr;
+    }
+  }
+  // Fused multiply and add/sub need to have the same dst and accumulator reg.
+  bool FusedFPOp = (Opcode == SystemZ::WFMADB || Opcode == SystemZ::WFMASB ||
+                    Opcode == SystemZ::WFMSDB || Opcode == SystemZ::WFMSSB);
+  if (FusedFPOp) {
+    Register DstReg = VRM->getPhys(MI.getOperand(0).getReg());
+    Register AccReg = VRM->getPhys(MI.getOperand(3).getReg());
+    if (OpNum == 0 || OpNum == 3 || DstReg != AccReg)
+      return nullptr;
+  }
+
   // Try to swap compare operands if possible.
   bool NeedsCommute = false;
   if ((MI.getOpcode() == SystemZ::CR || MI.getOpcode() == SystemZ::CGR ||
-       MI.getOpcode() == SystemZ::CLR || MI.getOpcode() == SystemZ::CLGR) &&
+       MI.getOpcode() == SystemZ::CLR || MI.getOpcode() == SystemZ::CLGR ||
+       MI.getOpcode() == SystemZ::WFCDB || MI.getOpcode() == SystemZ::WFCSB ||
+       MI.getOpcode() == SystemZ::WFKDB || MI.getOpcode() == SystemZ::WFKSB) &&
       OpNum == 0 && prepareCompareSwapOperands(MI))
     NeedsCommute = true;
 
   bool CCOperands = false;
   if (MI.getOpcode() == SystemZ::LOCRMux || MI.getOpcode() == SystemZ::LOCGR ||
       MI.getOpcode() == SystemZ::SELRMux || MI.getOpcode() == SystemZ::SELGR) {
     assert(MI.getNumOperands() == 6 && NumOps == 5 &&
            "LOCR/SELR instruction operands corrupt?");
else {
           Register::isVirtualRegister(SrcReg) &&
           DstPhys == VRM->getPhys(SrcReg))
         NeedsCommute = (OpNum == 1);
       else
         return nullptr;
     }
   }
 
-  if ((OpNum == NumOps - 1) || NeedsCommute) {
+  if ((OpNum == NumOps - 1) || NeedsCommute || FusedFPOp) {
     const MCInstrDesc &MemDesc = get(MemOpcode);
     uint64_t AccessBytes = SystemZII::getAccessSize(MemDesc.TSFlags);
     assert(AccessBytes != 0 && "Size of access should be known");
     assert(AccessBytes <= Size && "Access outside the frame index");
     uint64_t Offset = Size - AccessBytes;
     MachineInstrBuilder MIB = BuildMI(*InsertPt->getParent(), InsertPt,
                                       MI.getDebugLoc(), get(MemOpcode));
     if (MI.isCompare()) {
       assert(NumOps == 2 && "Expected 2 register operands for a compare.");
       MIB.add(MI.getOperand(NeedsCommute ? 1 : 0));
     }
+    else if (FusedFPOp) {
+      MIB.add(MI.getOperand(0));
+      MIB.add(MI.getOperand(3));
+      MIB.add(MI.getOperand(OpNum == 1 ? 2 : 1));
+    }
     else {
       MIB.add(MI.getOperand(0));
       if (NeedsCommute)
         MIB.add(MI.getOperand(2));
       else
         for (unsigned I = 1; I < OpNum; ++I)
           MIB.add(MI.getOperand(I));
     }
     MIB.addFrameIndex(FrameIndex).addImm(Offset);
     if (MemDesc.TSFlags & SystemZII::HasIndex)
       MIB.addReg(0);
     if (CCOperands) {
       unsigned CCValid = MI.getOperand(NumOps).getImm();
       unsigned CCMask = MI.getOperand(NumOps + 1).getImm();
       MIB.addImm(CCValid);
       MIB.addImm(NeedsCommute ? CCMask ^ CCValid : CCMask);
     }
+    if (MIB->definesRegister(SystemZ::CC) &&
+        (!MI.definesRegister(SystemZ::CC) ||
+         MI.registerDefIsDead(SystemZ::CC))) {
+      MIB->addRegisterDead(SystemZ::CC, TRI);
+      if (CCLiveRange)
+        CCLiveRange->createDeadDef(MISlot, LIS->getVNInfoAllocator());
+    }
+    // Constrain the register classes if converted from a vector opcode. The
+    // allocated regs are in an FP reg-class per previous check above.
+    for (const MachineOperand &MO : MIB->operands())
+      if (MO.isReg() && Register::isVirtualRegister(MO.getReg())) {
+        unsigned Reg = MO.getReg();
+        if (MRI.getRegClass(Reg) == &SystemZ::VR32BitRegClass)
+          MRI.setRegClass(Reg, &SystemZ::FP32BitRegClass);
+        else if (MRI.getRegClass(Reg) == &SystemZ::VR64BitRegClass)
+          MRI.setRegClass(Reg, &SystemZ::FP64BitRegClass);
+        else if (MRI.getRegClass(Reg) == &SystemZ::VR128BitRegClass)
+          MRI.setRegClass(Reg, &SystemZ::VF128BitRegClass);
+      }
+
     transferDeadCC(&MI, MIB);
     transferMIFlag(&MI, MIB, MachineInstr::NoSWrap);
+    transferMIFlag(&MI, MIB, MachineInstr::NoFPExcept);
     return MIB;
   }
 
   return nullptr;
 }
 
 MachineInstr *SystemZInstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,

llvm/test/CodeGen/SystemZ/foldmemop-vec-binops.mir

+# RUN: llc -mtriple=s390x-linux-gnu -mcpu=z14 -start-before=greedy %s -o - \
+# RUN:   | FileCheck %s
+#
+# Test folding of a memory operand into an fp memory instruction.
+
+--- |
+  define void @fun0(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun1(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun2(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun3(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun4(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun5(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun6(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun7(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun8(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun9(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun10(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun11(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun12(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun13(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun14(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun15(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun16(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun17(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun18(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun19(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun20(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun21(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun22(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun23(float %arg0, float %arg1, float* %Dst) { ret void }
+
+...
+
+# Test with both orders of operands since some operations are commutative.
+
+# CHECK-LABEL: fun0:
+# CHECK: adb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun0
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFADB %0, %1, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun1:
+# CHECK: adb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun1
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFADB %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+# ADB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun2:
+# CHECK: wfadb   %f0, %v16, %f0
+---
+name:            fun2
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFADB %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun3:
+# CHECK: aeb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun3
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFASB %0, %1, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun4:
+# CHECK: aeb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun4
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFASB %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# AEB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun5:
+# CHECK: wfasb   %f0, %v16, %f0
+---
+name:            fun5
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFASB %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun6:
+# CHECK: sdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun6
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFSDB %0, %1, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun7:
+# CHECK: wfsdb   %f0, %f1, %f0
+---
+name:            fun7
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFSDB %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+# SDB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun8:
+# CHECK: wfsdb   %f0, %f0, %v16
+---
+name:            fun8
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFSDB %0, %1, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun9:
+# CHECK: seb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun9
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFSSB %0, %1, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun10:
+# CHECK: wfssb   %f0, %f1, %f0
+---
+name:            fun10
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFSSB %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# SEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun11:
+# CHECK: wfssb   %f0, %f0, %v16
+---
+name:            fun11
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFSSB %0, %1, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+# CHECK-LABEL: fun12:
+# CHECK: ddb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun12
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFDDB %0, %1, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun13:
+# CHECK: wfddb   %f0, %f1, %f0
+---
+name:            fun13
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFDDB %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+# DDB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun14:
+# CHECK: wfddb   %f0, %f0, %v16
+---
+name:            fun14
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFDDB %0, %1, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun15:
+# CHECK: deb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun15
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFDSB %0, %1, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun16:
+# CHECK: wfdsb   %f0, %f1, %f0
+---
+name:            fun16
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFDSB %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# DEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun17:
+# CHECK: wfdsb   %f0, %f0, %v16
+---
+name:            fun17
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFDSB %0, %1, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun18:
+# CHECK: mdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun18
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMDB %0, %1, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun19:
+# CHECK: mdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun19
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMDB %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+# MDB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun20:
+# CHECK: wfmdb   %f0, %v16, %f0
+---
+name:            fun20
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMDB %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun21:
+# CHECK: meeb    %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun21
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSB %0, %1, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun22:
+# CHECK: meeb    %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun22
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSB %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...
+
+
+# MEEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun23:
+# CHECK: wfmsb   %f0, %v16, %f0
+---
+name:            fun23
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSB %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 4 into %ir.Dst)
+    Return
+
+...

llvm/test/CodeGen/SystemZ/foldmemop-vec-cc.mir

+# RUN: llc -mtriple=s390x-linux-gnu -mcpu=z14 -start-before=greedy %s -o - \
+# RUN:   | FileCheck %s
+#
+# Test that folding does not occur if it would introduce a clobbering of a live CC.
+
+--- |
+  define void @fun0(double %arg0, double %arg1, double* %Dst) { ret void }
+
+...
+
+# CHECK-LABEL: fun0:
+# CHECK: ld      %f1, 160(%r15)          # 8-byte Folded Reload
+# CHECK-NEXT: wfadb   %f0, %f0, %f1
+---
+name:            fun0
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    CDBR %0, %1, implicit-def $cc, implicit $fpc
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFADB %0, %1, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 14, 8, implicit killed $cc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+

llvm/test/CodeGen/SystemZ/foldmemop-vec-cmp.mir

+# RUN: llc -mtriple=s390x-linux-gnu -mcpu=z14 -start-before=greedy %s -o - \
+# RUN:   | FileCheck %s
+#
+# Test folding of a memory operand into an fp memory instruction.
+
+--- |
+  define void @fun0(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun1(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun2(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun3(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun4(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun5(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun6(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun7(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun8(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun9(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun10(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun11(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun12(float %arg0, float %arg1, float* %Dst) { ret void }
+
+...
+
+# CHECK-LABEL: fun0:
+# CHECK: cdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun0
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFCDB %0, %1, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun1:
+# CHECK: cdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun1
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFCDB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CDB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun2:
+# CHECK:      ld      %f0, 160(%r15)                  # 8-byte Folded Reload
+# CHECK-NEXT: wfcdb   %v16, %f0
+---
+name:            fun2
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFCDB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun3:
+# CHECK: ceb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun3
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFCSB %0, %1, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun4:
+# CHECK: ceb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun4
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFCSB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun5:
+# CHECK:      lde     %f0, 164(%r15)                  # 4-byte Folded Reload
+# CHECK-NEXT: wfcsb   %v16, %f0
+---
+name:            fun5
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFCSB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun6:
+# CHECK: kdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun6
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFKDB %0, %1, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun7:
+# CHECK: kdb     %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun7
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFKDB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# KDB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun8:
+# CHECK:      ld      %f0, 160(%r15)                  # 8-byte Folded Reload
+# CHECK-NEXT: wfkdb   %v16, %f0
+---
+name:            fun8
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFKDB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun9:
+# CHECK: keb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun9
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFKSB %0, %1, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun10:
+# CHECK: keb     %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun10
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFKSB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun11:
+# CHECK:      lde     %f0, 164(%r15)                  # 4-byte Folded Reload
+# CHECK-NEXT: wfksb   %v16, %f0
+---
+name:            fun11
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    nofpexcept WFKSB %1, %0, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# This test case involves a 128 bit operand (VGMF generating float 1.0).
+# CHECK-LABEL: fun12:
+# CHECK: vgmf    %v0, 2, 8
+# CHECK-NEXT: ceb     %f0, 164(%r15)          # 4-byte Folded Reload
+---
+name:            fun12
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr128bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: gr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s, $f2s, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr128bit = VGMF 2, 8
+    nofpexcept WFCSB %0, %1.subreg_h32:vr128bit, implicit-def $cc, implicit $fpc
+    %4:gr64bit = LGHI 0
+    %4:gr64bit = LOCGHI %4, 1, 15, 8, implicit killed $cc
+    STG %4, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...

llvm/test/CodeGen/SystemZ/foldmemop-vec-fusedfp.mir

+# RUN: llc -mtriple=s390x-linux-gnu -mcpu=z14 -start-before=greedy %s -o - \
+# RUN:   | FileCheck %s
+#
+# Test folding of a memory operand into an fp memory instruction.
+
+--- |
+  define void @fun0(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun1(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun2(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun3(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun4(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun5(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun6(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun7(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun8(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun9(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun10(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun11(double %arg0, double %arg1, double* %Dst) { ret void }
+  define void @fun12(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun13(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun14(float %arg0, float %arg1, float* %Dst) { ret void }
+  define void @fun15(float %arg0, float %arg1, float* %Dst) { ret void }
+
+...
+
+# CHECK-LABEL: fun0:
+# CHECK: madb    %f0, %f1, 160(%r15)             # 8-byte Folded Reload
+---
+name:            fun0
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMADB %0, %1, %4, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun1:
+# CHECK: madb    %f0, %f1, 160(%r15)             # 8-byte Folded Reload
+---
+name:            fun1
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMADB %1, %0, %4, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+# MADB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun2:
+# CHECK: wfmadb  %f0, %v16, %f1, %f0
+---
+name:            fun2
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMADB %1, %0, %4, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# MADB can't be used unless accumulator and result registers are the same.
+# CHECK-LABEL: fun3:
+# CHECK: wfmadb  %f0, %f0, %f1, %f2
+---
+name:            fun3
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMADB %4, %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun4:
+# CHECK: maeb    %f0, %f1, 160(%r15)             # 4-byte Folded Reload
+---
+name:            fun4
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMASB %0, %1, %4, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun5:
+# CHECK: maeb    %f0, %f1, 160(%r15)             # 4-byte Folded Reload
+---
+name:            fun5
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMASB %1, %0, %4, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# MAEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun6:
+# CHECK: wfmasb  %f0, %v16, %f1, %f0
+---
+name:            fun6
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMASB %1, %0, %4, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# MAEB can't be used unless accumulator and result registers are the same.
+# CHECK-LABEL: fun7:
+# CHECK: wfmasb  %f0, %f0, %f1, %f2
+---
+name:            fun7
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMASB %4, %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun8:
+# CHECK: msdb    %f0, %f1, 160(%r15)             # 8-byte Folded Reload
+---
+name:            fun8
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMSDB %0, %1, %4, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun9:
+# CHECK: msdb    %f0, %f1, 160(%r15)             # 8-byte Folded Reload
+---
+name:            fun9
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMSDB %1, %0, %4, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+# MSDB can't be used if one operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun10:
+# CHECK: wfmsdb  %f0, %v16, %f1, %f0
+---
+name:            fun10
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMSDB %1, %0, %4, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# MSDB can't be used unless accumulator and result registers are the same.
+# CHECK-LABEL: fun11:
+# CHECK: wfmsdb  %f0, %f0, %f1, %f2
+---
+name:            fun11
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: fp64bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr64bit }
+  - { id: 4, class: fp64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+  - { reg: '$f2d', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp64bit = COPY $f2d
+    %0:fp64bit = COPY $f0d
+    %4:fp64bit = COPY $f2d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr64bit = nofpexcept WFMSDB %4, %1, %0, implicit $fpc
+    VST64 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun12:
+# CHECK: mseb    %f0, %f1, 160(%r15)             # 4-byte Folded Reload
+---
+name:            fun12
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSSB %0, %1, %4, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# CHECK-LABEL: fun13:
+# CHECK: mseb    %f0, %f1, 160(%r15)             # 4-byte Folded Reload
+---
+name:            fun13
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSSB %1, %0, %4, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# MSEB can't be used if one operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun14:
+# CHECK: wfmssb  %f0, %v16, %f1, %f0
+---
+name:            fun14
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:vr32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSSB %1, %0, %4, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...
+
+
+# MSEB can't be used unless accumulator and result registers are the same.
+# CHECK-LABEL: fun15:
+# CHECK: wfmssb  %f0, %f0, %f1, %f2
+---
+name:            fun15
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp32bit }
+  - { id: 2, class: addr64bit }
+  - { id: 3, class: vr32bit }
+  - { id: 4, class: fp32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+  - { reg: '$f2s', virtual-reg: '%1' }
+  - { reg: '$r2d', virtual-reg: '%2' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d, $f2d, $r2d
+  
+    %2:addr64bit = COPY $r2d
+    %1:fp32bit = COPY $f2s
+    %0:fp32bit = COPY $f0s
+    %4:fp32bit = COPY $f2s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %3:vr32bit = nofpexcept WFMSSB %4, %1, %0, implicit $fpc
+    VST32 %3, %2, 0, $noreg :: (store 8 into %ir.Dst)
+    Return
+
+...

llvm/test/CodeGen/SystemZ/foldmemop-vec-unary.mir

+# RUN: llc -mtriple=s390x-linux-gnu -mcpu=z14 -start-before=greedy %s -o - \
+# RUN:   | FileCheck %s
+#
+# Test folding of a memory operand into an fp memory instruction.
+
+--- |
+  define double @fun0(float %arg0) { ret double 0.0 }
+  define double @fun1(float %arg0) { ret double 0.0 }
+  define double @fun2(float %arg0) { ret double 0.0 }
+  define double @fun3(double %arg0) { ret double 0.0 }
+  define double @fun4(double %arg0) { ret double 0.0 }
+  define float @fun5(float %arg0) { ret float 0.0 }
+  define float @fun6(float %arg0) { ret float 0.0 }
+
+...
+
+
+# CHECK-LABEL: fun0:
+# CHECK: ldeb    %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun0
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s
+  
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr64bit = nofpexcept WLDEB %0, implicit $fpc
+    $f0d = COPY %1
+    Return implicit $f0d
+
+...
+
+
+# LDEB can't be used if dst operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun1:
+# CHECK:      lde     %f0, 164(%r15)                  # 4-byte Folded Reload
+# CHECK-NEXT: wldeb   %v16, %f0
+---
+name:            fun1
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s
+  
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr64bit = nofpexcept WLDEB %0, implicit $fpc
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    $f0d = COPY %1
+    Return implicit $f0d
+
+...
+
+
+# Spilling the destination of an fp extension needs an extra store instruction.
+# CHECK-LABEL: fun2:
+# CHECK:      ldebr   %f0, %f0
+# CHECK-NEXT: std     %f0, 160(%r15)                  # 8-byte Folded Spill
+---
+name:            fun2
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: fp64bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s
+  
+    %0:fp32bit = COPY $f0s
+    %1:fp64bit = nofpexcept WLDEB %0, implicit $fpc
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    $f0d = COPY %1
+    Return implicit $f0d
+
+...
+
+
+# CHECK-LABEL: fun3:
+# CHECK: sqdb    %f0, 160(%r15)                  # 8-byte Folded Reload
+---
+name:            fun3
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d
+  
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr64bit = nofpexcept WFSQDB %0, implicit $fpc
+    $f0d = COPY %1
+    Return implicit $f0d
+
+...
+
+
+# SQDB can't be used if dst operand is a VR64 (and not FP64).
+# CHECK-LABEL: fun4:
+# CHECK:      ld      %f0, 160(%r15)                  # 8-byte Folded Reload
+# CHECK-NEXT: wfsqdb  %v16, %f0
+---
+name:            fun4
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp64bit }
+  - { id: 1, class: vr64bit }
+liveins:
+  - { reg: '$f0d', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0d
+  
+    %0:fp64bit = COPY $f0d
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr64bit = nofpexcept WFSQDB %0, implicit $fpc
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    $f0d = COPY %1
+    Return implicit $f0d
+
+...
+
+
+# CHECK-LABEL: fun5:
+# CHECK: sqeb    %f0, 164(%r15)                  # 4-byte Folded Reload
+---
+name:            fun5
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s
+  
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr32bit = nofpexcept WFSQSB %0, implicit $fpc
+    $f0s = COPY %1
+    Return implicit $f0d
+
+...
+
+
+# SQEB can't be used if dst operand is a VR32 (and not FP32).
+# CHECK-LABEL: fun6:
+# CHECK:      lde     %f0, 164(%r15)                  # 4-byte Folded Reload
+# CHECK-NEXT: wfsqsb  %v16, %f0
+---
+name:            fun6
+alignment:       16
+tracksRegLiveness: true
+registers:
+  - { id: 0, class: fp32bit }
+  - { id: 1, class: vr32bit }
+liveins:
+  - { reg: '$f0s', virtual-reg: '%0' }
+frameInfo:
+  maxAlignment:    1
+machineFunctionInfo: {}
+body:             |
+  bb.0:
+    liveins: $f0s
+  
+    %0:fp32bit = COPY $f0s
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    %1:vr32bit = nofpexcept WFSQSB %0, implicit $fpc
+    INLINEASM &"", 1, 12, implicit-def dead early-clobber $r0d, 12, implicit-def dead early-clobber $r1d, 12, implicit-def dead early-clobber $r2d, 12, implicit-def dead early-clobber $r3d, 12, implicit-def dead early-clobber $r4d, 12, implicit-def dead early-clobber $r5d, 12, implicit-def dead early-clobber $r6d, 12, implicit-def dead early-clobber $r7d, 12, implicit-def dead early-clobber $r8d, 12, implicit-def dead early-clobber $r9d, 12, implicit-def dead early-clobber $r10d, 12, implicit-def dead early-clobber $r11d, 12, implicit-def dead early-clobber $r12d, 12, implicit-def dead early-clobber $r13d, 12, implicit-def dead early-clobber $r14d, 12, implicit-def dead early-clobber $f0d, 12, implicit-def dead early-clobber $f1d, 12, implicit-def dead early-clobber $f2d, 12, implicit-def dead early-clobber $f3d, 12, implicit-def dead early-clobber $f4d, 12, implicit-def dead early-clobber $f5d, 12, implicit-def dead early-clobber $f6d, 12, implicit-def dead early-clobber $f7d, 12, implicit-def dead early-clobber $f8d, 12, implicit-def dead early-clobber $f9d, 12, implicit-def dead early-clobber $f10d, 12, implicit-def dead early-clobber $f11d, 12, implicit-def dead early-clobber $f12d, 12, implicit-def dead early-clobber $f13d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f14d, 12, implicit-def dead early-clobber $f15d
+    $f0s = COPY %1
+    Return implicit $f0s
+
+...

llvm/test/CodeGen/SystemZ/int-cmp-56.mir

--- |
   define i32 @fun3(i32* %ptr0)  { ret i32 0 }
   define i32 @fun4(i32* %ptr0)  { ret i32 0 }
 ...
 
 
 # Test CGR -> CG
 # CHECK: name:            fun1
 # CHECK: CG %10, %stack.0, 0, $noreg, implicit-def $cc :: (load 8 from %stack.0)
-# CHECK-NEXT: %12:gr64bit = LOCGHI %12, 8, 14, 12, implicit killed $cc
+# CHECK-NEXT: %12:gr64bit = LOCGHI %12, 8, 14, 12, implicit $cc
 ---
 name:            fun1
 alignment:       16
 tracksRegLiveness: true
 registers:
   - { id: 0, class: addr64bit }
   - { id: 1, class: gr64bit }
   - { id: 2, class: gr64bit }
bb.0:
     $r2d = COPY %12
     Return implicit $r2d
 ...
 
 
 # Test CLGR -> CLG
 # CHECK: name:            fun2
 # CHECK: CLG %10, %stack.0, 0, $noreg, implicit-def $cc :: (load 8 from %stack.0)
-# CHECK-NEXT: %12:gr64bit = LOCGHI %12, 8, 14, 12, implicit killed $cc
+# CHECK-NEXT: %12:gr64bit = LOCGHI %12, 8, 14, 12, implicit $cc
 ---
 name:            fun2
 alignment:       16
 tracksRegLiveness: true
 registers:
   - { id: 0, class: addr64bit }
   - { id: 1, class: gr64bit }
   - { id: 2, class: gr64bit }
bb.0:
     $r2d = COPY %12
     Return implicit $r2d
 ...
 
 
 # Test CR -> C
 # CHECK: name:            fun3
 # CHECK: C %10, %stack.0, 0, $noreg, implicit-def $cc :: (load 4 from %stack.0)
-# CHECK: %12:gr32bit = LOCHIMux %12, 8, 14, 12, implicit killed $cc
+# CHECK: %12:gr32bit = LOCHIMux %12, 8, 14, 12, implicit $cc
 ---
 name:            fun3
 alignment:       16
 tracksRegLiveness: true
 registers:
   - { id: 0, class: addr64bit }
   - { id: 1, class: gr32bit }
   - { id: 2, class: gr32bit }
bb.0:
     $r2l = COPY %12
     Return implicit $r2l
 ...
 
 
 # Test CLR -> CL
 # CHECK: name:            fun4
 # CHECK: CL %10, %stack.0, 0, $noreg, implicit-def $cc :: (load 4 from %stack.0)
-# CHECK: %12:gr32bit = LOCHIMux %12, 8, 14, 12, implicit killed $cc
+# CHECK: %12:gr32bit = LOCHIMux %12, 8, 14, 12, implicit $cc
 ---
 name:            fun4
 alignment:       16
 tracksRegLiveness: true
 registers:
   - { id: 0, class: addr64bit }
   - { id: 1, class: gr32bit }
   - { id: 2, class: gr32bit }