[mips] MIPS32R6 compact branch support

[sdardis]

Summary

Summary:
MIPSR6 introduces a class of branches called compact branches. Unlike the
traditional MIPS branches which have a delay slot, compact branches do not
have a delay slot. The instruction following the compact branch is only
executed if the branch is not taken and must not be a branch.

This patch implements support for compact branches on MIPS32R6.

Generate compact branches for MIPS32R6 when the delay slot filler cannot fill
a delay slot. Inspect the generated code for forbidden slot hazards (a compact
branch with an adjacent branch or other CTI) and insert nops to clear this
hazard.

[vkalintiris]

lib/Target/Mips/Mips.h

35

Do we really need this pass? Shouldn't we just take care so that the MipsDelaySlotFiller pass does not create any hazards in the first place?

lib/Target/Mips/MipsDelaySlotFiller.cpp

604–605

We can remove this once we add support for microMIPS in getEquivalentCompactForm().

620

The STI.hasMips64r6() will be true if STI.hasMip32r6() is true.

test/CodeGen/Mips/compact-branches.ll

1 ↗

(On Diff #45374)

Although it isn't absolutely necessary, it would be better to have a function per instruction check, ie. one function that tests bnec, another one for bgezc etc.

[vkalintiris]

lib/Target/Mips/MipsDelaySlotFiller.cpp

507–559

We should make this function a member of MipsSEInstrInfo. This is where we keep similar logic for branch instruction operations.

Can you merge the first switch statement into the second? The check for the number of explicit operands becomes unnecessary once you've identified the opcode. Also, this is where we should provide the relevant logic for branches in microMIPS mode.

509–517

The same functionality is implemented in MipsInstrInfo::genInstrWithNewOpc(). You can expand the logic of that function, for branches that use the $zero register.

557

I think that we should return 0 here. Mips::ZERO is a positive number.

[sdardis]

Addressed comments. Shifted the $zero detection for branches into MipsInstrInfo::getInstrWithNewOpc. I've elected to keep the separate hazard pass for the moment, see my reply there.

Thanks,
Simon

[sdardis]

Apologies, comment didn't get posted, reproduced here:

Shouldn't we just take care so that the MipsDelaySlotFiller pass does not create any hazards in the first place?

For this patch that approach could be taken but I think its better it introduce a fairly specific pass to handle forbidden slot hazards.

Handling hazards as a separate pass introduces a high level of safety as then compact branches can be produced anywhere in the compilation pipeline without the subtly incorrect reliance that the delay slot filler will fix those hazards. It's a separation of concerns issue in my view.

Aside: GCC for MIPS uses a very similar hazard strategy for forbidden slot, hi/lo and delayed value hazards.

Thoughts?

lib/Target/Mips/Mips.h

35

Shouldn't we just take care so that the MipsDelaySlotFiller pass does not create any hazards in the first place?

For this patch that approach could be taken but I think its better it introduce a fairly specific pass to handle forbidden slot hazards.

Handling hazards as a separate pass introduces a high level of safety as then compact branches can be produced anywhere in the compilation pipeline without the subtly incorrect reliance that the delay slot filler will fix those hazards. It's a separation of concerns issue in my view.

Aside: GCC for MIPS uses a very similar hazard strategy for forbidden slot, hi/lo and delayed value hazards.

Thoughts?

[vkalintiris]

In D16353#332475, @sdardis wrote:

Apologies, comment didn't get posted, reproduced here:

Shouldn't we just take care so that the MipsDelaySlotFiller pass does not create any hazards in the first place?

For this patch that approach could be taken but I think its better it introduce a fairly specific pass to handle forbidden slot hazards.

Handling hazards as a separate pass introduces a high level of safety as then compact branches can be produced anywhere in the compilation pipeline without the subtly incorrect reliance that the delay slot filler will fix those hazards. It's a separation of concerns issue in my view.

Aside: GCC for MIPS uses a very similar hazard strategy for forbidden slot, hi/lo and delayed value hazards.

Thoughts?

Following this approach means that the MipsDelaySlotFiller pass will no longer generate correct output code, even in the presence of correct input code. As far as I can tell, the code generator provides (or at least it should provide) correct input code to the MipsDelaySlotFiller pass as it does not generate any CTIs in any slot.

Even if we start emitting CTIs in forbidden slots for some unforeseeable reason in the future, conceptually, the MipsDelaySlotFiller pass would be the more appropriate place to remove them as it is responsbile for the handling
of slots either way.

Having said that, I wouldn't be against a dedicated hazard handling pass if we couldn't avoid generating them from the code generator, or handle them at the right pass.

[sdardis]

Shifted the hazard clearing logic into the delay slot filler.

[vkalintiris]

lib/Target/Mips/MipsDelaySlotFiller.cpp

511–515

If you update Branch with the return value of genInstrWithNewOpc() then deleting the previous branch is as simple as: std::next(Branch)->eraseFromParent();

538–580

I think that it would be useful to have these functions at MipsInstrInfo as we could call them from other passes too.

557–621

There's no need to iterate over every instruction twice. We can check whether we have to add a NOP inside replaceWithCompactBranch() and bundle it together with the compact branch.

562

The cast is redundant.

lib/Target/Mips/MipsInstrInfo.cpp

351

We don't have to check for the number of explicit operands.

352

You can use MipsABIInfo::GetZeroReg() to test with the right zero register. Also, the cast is unnecessary.

367–368

Shouldn't we set ZeroOperandBranch to false in the default case?

lib/Target/Mips/MipsSEInstrInfo.cpp

439

Redundant newline.

474–483

We will never reach this if STI.hasMips32r6() is true. We should merge the logic of opcode selection into the first switch statement.

[sdardis]

Addressed outstanding comments except for the one regarding Filler::clearFSHazards. See my comment there.

Added a FIXME to MipsISelLowering.cpp: MipsTargetLowering::emitAtomicBinary on an outstanding code gen issue and to MipsInstrInfo::genInstrWithNewOpc as they're related. Without the checking for Mips::ZERO and MipsABIInfo::GetZeroReg() we get more verbose asm than necessary.

Updated commit text to reflect that its for MIPS32R6 only.

[sdardis]

lib/Target/Mips/MipsDelaySlotFiller.cpp

557–621

We can check whether we have to add a NOP inside replaceWithCompactBranch() and bundle it together with the compact branch.

We can't. You've missed case C in the comment.

Consider the following from function l() in the test case before delay slot filling (using assembly here to keep things clear):

      move   $16, $2
      jal      j
      bne    $16, $2, $BB0_2
# BB#1:                                 # %if.then
      addiu   $4, $zero, -2
      jal     f

For the first jal, the delay slot filler will insert the 'move' into its delay slot. The delay slot filler has no instruction to put in the delay slot of bne, so it transforms it into bnec. No nop will be inserted as a) there is no CTI following bnec in that basic block, and b) the first non-debug instruction in the physically following basic block is not a CTI.

The delay filler then considers BB#1 and schedules the addiu into the delay slot of the jal. The delay filler has now created a forbidden slot hazard. Handling this in the delay filler itself requires identifying such special cases, then inserting a nop into the basic block of the compact branch (which could be the previous basic block to the one we're working on).

This is in addition to cases A and B described in the comment. My reasoning for processing all the instructions again is that it trivializes all those cases (and possibly others) into a 'is the physical successor instruction of a compact branch a CTI?' examination rather than inserting checks where ever we could create a FS hazard. Inserting handling FS hazard logic in multiple places I think could lead to correctness issues as we'd need to ensure we cover all possible cases, rather than a simple (mini-)pass decoupled from the delay slot filling logic.

Thoughts?

lib/Target/Mips/MipsInstrInfo.cpp

352

You can use MipsABIInfo::GetZeroReg() to test with the right zero register.

Strangely enough, this doesn't work.

MipsTargeLowering::emitAtomicBinary will generate a Mips::BEQ with Mips::ZERO as an operand on mips64 if the requested size is 4. For mips64 it appears we need to check for MipsABIInfo::GetZeroReg() and Mips::ZERO for the moment.

This appears to be an outstanding issue with emitAtomicBinary.

lib/Target/Mips/MipsSEInstrInfo.cpp

439

clang-format inserts the newline, leave or keep it?

[vkalintiris]

lib/Target/Mips/MipsDelaySlotFiller.cpp

557–621

The (C) case will happen when filling the delay slot of a normal branch, ie. inside the MipsDelaySlotFiller pass. We can teach the "search-backwards" part of the delay slot filler (DSF) algorithm to handle this by checking the last instruction of the BB's layout predecessor. If there are other issues that conceptually belong to the DSF pass, then we should add them too by modifying the pass accordingly.

I don't see how (B) can happen as we only emit compact branches at this pass for the time being. From my perspective, if a previous pass would like to add a compact-branch, then I can think of two options: (a) insert the compact branch only if the next instruction is not a CTI-hazard and leave the task of fixing the corner cases that happen during the filling of normal branches by the DSF, or (b) insert a NOP and just offload everything to the DSF. I can't think of any reason for a post-DSF pass to insert new/additional compact-branches or move non-CTI instructions from a forbidden slot.

Even if we find ourselves constrained by having the DSF pass handling everything, then we can add a separate/final pass that fixes every case left over from previous passes. However, with this design we would achieve the minimum number of places were our code is wrong/invalid.

If we'd want to be on the really safe side and make sure that we are aware of every possible case that we might haven't consider yet, then we can add a separate pass and enable it only for debug builds. It could assert upon finding a compact-branch that contains a CTI in its forbidden slot.

lib/Target/Mips/MipsInstrInfo.cpp

352

Ok, if I understand correctly then we should just check for Mips::ZERO and keep the FIXME comment, ie. we can remove the Subtarget.getABI().GetZeroReg() check as it's not working at the moment.

lib/Target/Mips/MipsSEInstrInfo.cpp

439

It doesn't insert one for me, so I suggest that we remove the newline.

[sdardis]

lib/Target/Mips/MipsDelaySlotFiller.cpp

557–621

Even if we find ourselves constrained by having the DSF pass handling everything, then we can add a separate/final pass that fixes every case left over from previous passes. However, with this design we would achieve the minimum number of places were our code is wrong/invalid.

Rather than wiring FS hazard handling logic into the DSF, we could split it off into a separate pass and enable it after the delay slot filler, like the first diff. Thoughts dsanders? It keeps the implementation safer and simpler.

[sdardis]

Addressed comments. move hazard schedule back into it's own pass. You ok with that change Daniel?

[sdardis]

Simplify MipsHazardSchedule pass according to style guide.

[sdardis]

Ping.

[sdardis]

Ping.

[dsanders]

Most of these are nits but there a a couple important ones.

Regarding testing this pass. Ideally we would be using MIR but I'm not going to require that since I haven't had chance to try it myself.

Could you put the new tests in a subdirectory so that we keep forbidden slot tests together and can migrate them to MIR tests later?

lib/Target/Mips/MipsDelaySlotFiller.cpp

511

Mips16 uses the delay slot filler too as far as I know. If so, this could be a Mips16InstrInfo

557–624

The separation makes sense to me given that we're only using compact branches when delay slots go unfilled.

There are a couple cases I can think of where this may not be the right thing to do:

1. If we need a nop for both the delay slot and the forbidden slot then we need to decide which path should have the bubble. Ideally, it should be on the coldest path.
2. Some machines may generally prefer compact branches.

For #1, we should allow the hazard scheduler to revert the branch to a delay slot branch if the BB probabilities suggest that the taken path is colder than the not-taken path.

For #2, I think it's reasonable to expect that such machine will exist in the future but I'm not aware of any yet. It shouldn't be difficult to deal with that if/when it arises.

563–564

This could be a Mips16InstrInfo

616–619

Could you put braces around this?

lib/Target/Mips/MipsHazardSchedule.cpp

12–44

Could you change this into doxygen documentation? (see '\file')

87

Don't repeat the function name in new code. Also, please use doxygen comments ('///').

98

We shouldn't have the '|| inMicroMipsMode()' here. microMIPSR6 has forbidden slots too.

105

Use the arrow operator instead of '(*FI).'. There's a few other cases of this below

123

We can only have one fallthrough. We should stop iterating once we find it.

126–129

Please factor out the duplicate code.

lib/Target/Mips/MipsInstrInfo.cpp

259–301

We ought to tablegen-erate these.

346

I read this as being a branch with no operands. Can you make it clearer?

352

We'll should test for both ZERO and ZERO_64 so that this is still correct when the atomics are fixed.

lib/Target/Mips/MipsInstrInfo.h

73

Please make this doxygen comment document the function

73–77

Please make the arguments references since nullptr is not permitted

lib/Target/Mips/MipsSEInstrInfo.cpp

452

This doesn't look equivalent to BEQ to me. What happens if the second operand of the comparison isn't $zero?

457

This doesn't look equivalent to BEQ to me. What happens if the second operand of the comparison isn't $zero?

lib/Target/Mips/MipsSEInstrInfo.h

69 ↗

(On Diff #46763)

The iterator should be const too.

[sdardis]

Addressed nits. Instructions are now flagged as CTIs or having forbidden slots in the .td files using target specific flags. Moved getEquivalentCompactForm to MipsInstrInfo as it can actually apply to MIPS16, MIPSR6, microMIPS, microMIPSR6.

[sdardis]

Rebased to ToT.

lib/Target/Mips/MipsDelaySlotFiller.cpp

557–624

For #1, I'm not quite following you here. We have to place the nop after the compact branch as part the same basic block. For example:

BB1:
   <no instructions to go in a delay slot>
   bne v0,L3
BB2:
   jal g
   * move a0, v0
BB3:
....

The bne here will be transformed into a compact form. Since the next instruction is unsafe in the forbidden slot, we have to put a nop in the slot. We can't put the nop in BB3 as the first instruction as that doesn't clear the hazard. The hazard is due to instruction layout, not execution.

For #2, it's somewhat trivial to block delay slot filling for branches which have corresponding delay slot forms.

lib/Target/Mips/MipsHazardSchedule.cpp

98

microMIPSR6 doesn't have forbidden slots. The MD00582-2B-microMIPS32-AFP-06.03 says that "Any instruction, including a branch or jump, may immediately follow a branch or jump, that is, delay slot restrictions do not apply in Release 6". Confusingly MIPSR6 ISA has them.

lib/Target/Mips/MipsSEInstrInfo.cpp

452

If the second operand of the comparison is not $zero canUseMicroMipsBranches is false as that explicitly checks for conditions that beqz16 can be used in. However, if we have microMIPSR6 we can generate beqc (long version). if both canUseMicroMipsBranches and hasMips32r6() is false we return 0 since there's no equivalent form.

I've renamed canUseMicroMipsBranches to canUseShortMMBranches to better reflect what it's doing.

457

See above.

[sdardis]

Re: MIR tests:

I have another ~6 patches for compact branch/jumps pending. Unfortunately every one of them modifies compact-branches.ll, I haven't submitted them upstream yet because this one is the necessary groundwork for all of them. After submitting of them all or in the last patch, I could redo test/Codegen/Mips/compactbranches/compact-branches.ll as a MIR based test.

[sdardis]

Ping.

[dsanders]

LGTM with a couple nits.

lib/Target/Mips/MipsDelaySlotFiller.cpp

557–624

For #1, I'm trying to say that, in some circumstances, delay-slot branches can be a better choice than compact branches when both are available.

For example:

BB1:
  ...
  bne $2, BB3
  nop
BB2:
  jal g
BB3:
  ...

introduces a bubble on both the taken and not-taken paths while:

BB1:
  ...
  bnec $2, BB3
  nop
BB2:
  jal g
BB3:
  ...

introduces a bubble on the taken path and possibly two bubbles on the not-taken path depending on implementation.
I'm told this kind of thing occurs between delay-slot returns and compact-branch returns but we need to verify that.

My comment was intended to be something to think about rather than something to do in this patch. Forbidden slots is a correctness issue so it's important that we get a solution in place and we can expand on performance decisions in later patches.

The hazard is due to instruction layout, not execution.

That's right, but we aren't forced to choose code that has this hazard.

lib/Target/Mips/MipsHazardSchedule.cpp

98

Ok then. I'm pretty sure it had them at one point but it doesn't now

lib/Target/Mips/MipsInstrInfo.cpp

259–260

Repeated name in comment.

lib/Target/Mips/MipsInstrInfo.td

1118–1122

Why move this above DEI_FT?

1321–1325

Could you indent this since we have nested 'let' statements

lib/Target/Mips/MipsSEInstrInfo.cpp

452

Ok, thanks.

457

Ok, thanks

[sdardis]

Patch updated to address nits. Could you commit on my behalf? Thanks.

lib/Target/Mips/MipsInstrInfo.td

1118–1122

I was grouping it syscall/break/(d)eret classes so I could mark them all with one let isCTI =1 {..}. Want me to move it back?

[dsanders]

Could you commit on my behalf?

Sure

lib/Target/Mips/MipsInstrInfo.td

1118–1122

Ah ok. I was associating the '}' with the class rather than the let so it looked like there was no reason to move it.

[sdardis]

Rebased and retested to ToT (r263428).

lib/Target/Mips/CMakeLists.txt

add_llvm_target(MipsCodeGen
   Mips16ISelLowering.cpp
   Mips16RegisterInfo.cpp
   MipsAnalyzeImmediate.cpp
   MipsAsmPrinter.cpp
   MipsCCState.cpp
   MipsConstantIslandPass.cpp
   MipsDelaySlotFiller.cpp
   MipsFastISel.cpp
+  MipsHazardSchedule.cpp
   MipsInstrInfo.cpp
   MipsISelDAGToDAG.cpp
   MipsISelLowering.cpp
   MipsFrameLowering.cpp
   MipsLongBranch.cpp
   MipsMCInstLower.cpp
   MipsMachineFunction.cpp
   MipsModuleISelDAGToDAG.cpp

lib/Target/Mips/MCTargetDesc/MipsBaseInfo.h

enum {
     FrmJ  = 3,
     /// FrmFR - This form is for instructions of the format FR.
     FrmFR = 4,
     /// FrmFI - This form is for instructions of the format FI.
     FrmFI = 5,
     /// FrmOther - This form is for instructions that have no specific format.
     FrmOther = 6,
 
-    FormMask = 15
+    FormMask = 15,
+    /// IsCTI - Instruction is a Control Transfer Instruction.
+    IsCTI = 1 << 4,
+    /// HasForbiddenSlot - Instruction has a forbidden slot.
+    HasForbiddenSlot = 1 << 5
+
   };
 }
 }
 
 #endif

lib/Target/Mips/Mips.h

namespace llvm {
   class FunctionPass;
 
   ModulePass *createMipsOs16Pass(MipsTargetMachine &TM);
   ModulePass *createMips16HardFloatPass(MipsTargetMachine &TM);
 
   FunctionPass *createMipsModuleISelDagPass(MipsTargetMachine &TM);
   FunctionPass *createMipsOptimizePICCallPass(MipsTargetMachine &TM);
   FunctionPass *createMipsDelaySlotFillerPass(MipsTargetMachine &TM);
+  FunctionPass *createMipsHazardSchedule(MipsTargetMachine &tm);
   FunctionPass *createMipsLongBranchPass(MipsTargetMachine &TM);
   FunctionPass *createMipsConstantIslandPass(MipsTargetMachine &tm);
 } // end namespace llvm;
 
 #endif

lib/Target/Mips/Mips32r6InstrInfo.td

 
 class CMP_CONDN_DESC_BASE<string CondStr, string Typestr,
                           RegisterOperand FGROpnd,
                           SDPatternOperator Op = null_frag> {
   dag OutOperandList = (outs FGRCCOpnd:$fd);
   dag InOperandList = (ins FGROpnd:$fs, FGROpnd:$ft);
   string AsmString = !strconcat("cmp.", CondStr, ".", Typestr, "\t$fd, $fs, $ft");
   list<dag> Pattern = [(set FGRCCOpnd:$fd, (Op FGROpnd:$fs, FGROpnd:$ft))];
+  bit isCTI = 1;
 }
 
 multiclass CMP_CC_M <FIELD_CMP_FORMAT Format, string Typestr,
                      RegisterOperand FGROpnd>{
   let AdditionalPredicates = [NotInMicroMips] in {
     def CMP_F_#NAME : COP1_CMP_CONDN_FM<Format, FIELD_CMP_COND_AF>,
                       CMP_CONDN_DESC_BASE<"af", Typestr, FGROpnd>,
                       ISA_MIPS32R6, HARDFLOAT;

 }
 
 class AUI_DESC : AUI_DESC_BASE<"aui", GPR32Opnd>;
 
 class BRANCH_DESC_BASE {
   bit isBranch = 1;
   bit isTerminator = 1;
   bit hasDelaySlot = 0;
+  bit isCTI = 1;
 }
 
 class BC_DESC_BASE<string instr_asm, DAGOperand opnd> : BRANCH_DESC_BASE,
     MipsR6Arch<instr_asm> {
   dag InOperandList = (ins opnd:$offset);
   dag OutOperandList = (outs);
   string AsmString = !strconcat(instr_asm, "\t$offset");
   bit isBarrier = 1;
+  bit isCTI = 1;
 }
 
 class CMP_BC_DESC_BASE<string instr_asm, DAGOperand opnd,
                        RegisterOperand GPROpnd> : BRANCH_DESC_BASE {
   dag InOperandList = (ins GPROpnd:$rs, GPROpnd:$rt, opnd:$offset);
   dag OutOperandList = (outs);
   string AsmString = !strconcat(instr_asm, "\t$rs, $rt, $offset");
   list<Register> Defs = [AT];
+  bit hasForbiddenSlot = 1;
+  bit isCTI = 1;
 }
 
 class CMP_CBR_EQNE_Z_DESC_BASE<string instr_asm, DAGOperand opnd,
                                RegisterOperand GPROpnd> : BRANCH_DESC_BASE {
   dag InOperandList = (ins GPROpnd:$rs, opnd:$offset);
   dag OutOperandList = (outs);
   string AsmString = !strconcat(instr_asm, "\t$rs, $offset");
   list<Register> Defs = [AT];
+  bit hasForbiddenSlot = 1;
+  bit isCTI = 1;
 }
 
 class CMP_CBR_RT_Z_DESC_BASE<string instr_asm, DAGOperand opnd,
                              RegisterOperand GPROpnd>
     : BRANCH_DESC_BASE, MipsR6Arch<instr_asm> {
   dag InOperandList = (ins GPROpnd:$rt, opnd:$offset);
   dag OutOperandList = (outs);
   string AsmString = !strconcat(instr_asm, "\t$rt, $offset");
   list<Register> Defs = [AT];
+  bit hasForbiddenSlot = 1;
+  bit isCTI = 1;
 }
 
 class BAL_DESC : BC_DESC_BASE<"bal", brtarget> {
   bit isCall = 1;
   bit hasDelaySlot = 1;
   list<Register> Defs = [RA];
+  let isCTI = 1;
 }
 
 class BALC_DESC : BC_DESC_BASE<"balc", brtarget26> {
   bit isCall = 1;
   list<Register> Defs = [RA];
+  bit isCTI = 1;
 }
 
 class BC_DESC : BC_DESC_BASE<"bc", brtarget26>;
 class BGEC_DESC : CMP_BC_DESC_BASE<"bgec", brtarget, GPR32Opnd>;
 class BGEUC_DESC : CMP_BC_DESC_BASE<"bgeuc", brtarget, GPR32Opnd>;
 class BEQC_DESC : CMP_BC_DESC_BASE<"beqc", brtarget, GPR32Opnd>;
 class BNEC_DESC : CMP_BC_DESC_BASE<"bnec", brtarget, GPR32Opnd>;
 

 class BEQZC_DESC : CMP_CBR_EQNE_Z_DESC_BASE<"beqzc", brtarget21, GPR32Opnd>;
 class BNEZC_DESC : CMP_CBR_EQNE_Z_DESC_BASE<"bnezc", brtarget21, GPR32Opnd>;
 
 class COP1_BCCZ_DESC_BASE<string instr_asm> : BRANCH_DESC_BASE {
   dag InOperandList = (ins FGR64Opnd:$ft, brtarget:$offset);
   dag OutOperandList = (outs);
   string AsmString = instr_asm;
   bit hasDelaySlot = 1;
+  bit isCTI = 1;
 }
 
 class BC1EQZ_DESC : COP1_BCCZ_DESC_BASE<"bc1eqz $ft, $offset">;
 class BC1NEZ_DESC : COP1_BCCZ_DESC_BASE<"bc1nez $ft, $offset">;
 
 class COP2_BCCZ_DESC_BASE<string instr_asm> : BRANCH_DESC_BASE {
   dag InOperandList = (ins COP2Opnd:$ct, brtarget:$offset);
   dag OutOperandList = (outs);
   string AsmString = instr_asm;
   bit hasDelaySlot = 1;
+  bit isCTI = 1;
 }
 
 class BC2EQZ_DESC : COP2_BCCZ_DESC_BASE<"bc2eqz $ct, $offset">;
 class BC2NEZ_DESC : COP2_BCCZ_DESC_BASE<"bc2nez $ct, $offset">;
 
 class BOVC_DESC   : CMP_BC_DESC_BASE<"bovc", brtarget, GPR32Opnd>;
 class BNVC_DESC   : CMP_BC_DESC_BASE<"bnvc", brtarget, GPR32Opnd>;
 
 class JMP_IDX_COMPACT_DESC_BASE<string opstr, DAGOperand opnd,
                                 RegisterOperand GPROpnd>
     : MipsR6Arch<opstr> {
   dag InOperandList = (ins GPROpnd:$rt, opnd:$offset);
   string AsmString = !strconcat(opstr, "\t$rt, $offset");
   list<dag> Pattern = [];
   bit isTerminator = 1;
   bit hasDelaySlot = 0;
+  bit isCTI = 1;
 }
 
 class JIALC_DESC : JMP_IDX_COMPACT_DESC_BASE<"jialc", calloffset16,
                                              GPR32Opnd> {
   bit isCall = 1;
   list<Register> Defs = [RA];
 }
 
 class JIC_DESC : JMP_IDX_COMPACT_DESC_BASE<"jic", jmpoffset16, GPR32Opnd> {
   bit isBarrier = 1;
   list<Register> Defs = [AT];
 }
 
 class JR_HB_R6_DESC : JR_HB_DESC_BASE<"jr.hb", GPR32Opnd> {
   bit isBranch = 1;
   bit isIndirectBranch = 1;
   bit hasDelaySlot = 1;
   bit isTerminator=1;
   bit isBarrier=1;
+  bit isCTI = 1;
 }
 
 class BITSWAP_DESC_BASE<string instr_asm, RegisterOperand GPROpnd>
     : MipsR6Arch<instr_asm> {
   dag OutOperandList = (outs GPROpnd:$rd);
   dag InOperandList = (ins GPROpnd:$rt);
   string AsmString = !strconcat(instr_asm, "\t$rd, $rt");
   list<dag> Pattern = [];

 class CLO_R6_DESC : CLO_R6_DESC_BASE<"clo", GPR32Opnd>;
 class CLZ_R6_DESC : CLZ_R6_DESC_BASE<"clz", GPR32Opnd>;
 
 class SDBBP_R6_DESC {
   dag OutOperandList = (outs);
   dag InOperandList = (ins uimm20:$code_);
   string AsmString = "sdbbp\t$code_";
   list<dag> Pattern = [];
+  bit isCTI = 1;
 }
 
 //===----------------------------------------------------------------------===//
 //
 // Instruction Definitions
 //
 //===----------------------------------------------------------------------===//
 

lib/Target/Mips/MipsDelaySlotFiller.cpp

if (!isIdentifiedObject(V))
       return false;
 
     Objects.push_back(*I);
   }
 
   return true;
 }
 
 // Replace Branch with the compact branch instruction.
 Iter Filler::replaceWithCompactBranch(MachineBasicBlock &MBB,
                                       Iter Branch, DebugLoc DL) {
-  const MipsInstrInfo *TII =
-      MBB.getParent()->getSubtarget<MipsSubtarget>().getInstrInfo();
-
-  unsigned NewOpcode =
-    (((unsigned) Branch->getOpcode()) == Mips::BEQ) ? Mips::BEQZC_MM
-                                                    : Mips::BNEZC_MM;
-
-  const MCInstrDesc &NewDesc = TII->get(NewOpcode);
-  MachineInstrBuilder MIB = BuildMI(MBB, Branch, DL, NewDesc);
-
-  MIB.addReg(Branch->getOperand(0).getReg());
-  MIB.addMBB(Branch->getOperand(2).getMBB());
+  const MipsSubtarget &STI = MBB.getParent()->getSubtarget<MipsSubtarget>();
+  const MipsInstrInfo *TII = STI.getInstrInfo();
 
-  Iter tmpIter = Branch;
-  Branch = std::prev(Branch);
-  MBB.erase(tmpIter);
+  unsigned NewOpcode = TII->getEquivalentCompactForm(Branch);
+  Branch = TII->genInstrWithNewOpc(NewOpcode, Branch);
 
+  std::next(Branch)->eraseFromParent();
   return Branch;
 }
 
 // Replace Jumps with the compact jump instruction.
 Iter Filler::replaceWithCompactJump(MachineBasicBlock &MBB,
                                     Iter Jump, DebugLoc DL) {
   const MipsInstrInfo *TII =
       MBB.getParent()->getSubtarget<MipsSubtarget>().getInstrInfo();
 
   const MCInstrDesc &NewDesc = TII->get(Mips::JRC16_MM);
   MachineInstrBuilder MIB = BuildMI(MBB, Jump, DL, NewDesc);
 
   MIB.addReg(Jump->getOperand(0).getReg());
 
   Iter tmpIter = Jump;
   Jump = std::prev(Jump);
   MBB.erase(tmpIter);
 
   return Jump;
 }
 
 // For given opcode returns opcode of corresponding instruction with short
 // delay slot.
 static int getEquivalentCallShort(int Opcode) {
   switch (Opcode) {
   case Mips::BGEZAL:
     return Mips::BGEZALS_MM;
   case Mips::BLTZAL:
     return Mips::BLTZALS_MM;
   case Mips::JAL:
     return Mips::JALS_MM;
   case Mips::JALR:
     return Mips::JALRS_MM;
   case Mips::JALR16_MM:
     return Mips::JALRS16_MM;
   default:
     llvm_unreachable("Unexpected call instruction for microMIPS.");
   }
 }
 
 /// runOnMachineBasicBlock - Fill in delay slots for the given basic block.
 /// We assume there is only one delay slot per delayed instruction.
 bool Filler::runOnMachineBasicBlock(MachineBasicBlock &MBB) {
   bool Changed = false;
   const MipsSubtarget &STI = MBB.getParent()->getSubtarget<MipsSubtarget>();
   bool InMicroMipsMode = STI.inMicroMipsMode();
   const MipsInstrInfo *TII = STI.getInstrInfo();
 
   for (Iter I = MBB.begin(); I != MBB.end(); ++I) {
     if (!hasUnoccupiedSlot(&*I))
       continue;
 
     ++FilledSlots;
     Changed = true;
 
     // Delay slot filling is disabled at -O0.
     if (!DisableDelaySlotFiller && (TM.getOptLevel() != CodeGenOpt::None)) {
       bool Filled = false;
 
       if (searchBackward(MBB, I)) {
         Filled = true;
       } else if (I->isTerminator()) {
         if (searchSuccBBs(MBB, I)) {
           Filled = true;
         }
       } else if (searchForward(MBB, I)) {
         Filled = true;
       }
 
       if (Filled) {
         // Get instruction with delay slot.
         MachineBasicBlock::instr_iterator DSI(I);
 
         if (InMicroMipsMode && TII->GetInstSizeInBytes(&*std::next(DSI)) == 2 &&
             DSI->isCall()) {
           // If instruction in delay slot is 16b change opcode to
           // corresponding instruction with short delay slot.
           DSI->setDesc(TII->get(getEquivalentCallShort(DSI->getOpcode())));
         }
 
         continue;
       }
     }
 
     // If instruction is BEQ or BNE with one ZERO register, then instead of
     // adding NOP replace this instruction with the corresponding compact
     // branch instruction, i.e. BEQZC or BNEZC.
-    unsigned Opcode = I->getOpcode();
     if (InMicroMipsMode) {
-      switch (Opcode) {
-        case Mips::BEQ:
-        case Mips::BNE:
-          if (((unsigned) I->getOperand(1).getReg()) == Mips::ZERO) {
-            I = replaceWithCompactBranch(MBB, I, I->getDebugLoc());
-            continue;
-          }
-          break;
-        case Mips::JR:
-        case Mips::PseudoReturn:
-        case Mips::PseudoIndirectBranch:
-          // For microMIPS the PseudoReturn and PseudoIndirectBranch are allways
-          // expanded to JR_MM, so they can be replaced with JRC16_MM.
-          I = replaceWithCompactJump(MBB, I, I->getDebugLoc());
-          continue;
-        default:
-          break;
+      if (TII->getEquivalentCompactForm(I)) {
+
+      if (I->isIndirectBranch() || I->isReturn()) {
+        // For microMIPS the PseudoReturn and PseudoIndirectBranch are always
       }
     }
+
+    // For MIPSR6 attempt to produce the corresponding compact (no delay slot)
+    // form of the branch. This should save putting in a NOP.
+    if ((STI.hasMips32r6()) && TII->getEquivalentCompactForm(I)) {
+      I = replaceWithCompactBranch(MBB, I, I->getDebugLoc());
+      continue;
+    }
+
     // Bundle the NOP to the instruction with the delay slot.
     BuildMI(MBB, std::next(I), I->getDebugLoc(), TII->get(Mips::NOP));
     MIBundleBuilder(MBB, I, std::next(I, 2));
   }
 
   return Changed;
 }
 

lib/Target/Mips/MipsHazardSchedule.cpp

+//===-- MipsHazardSchedule.cpp - Workaround pipeline hazards---------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+/// \file
+/// This pass is used to workaround certain pipeline hazards. For now, this covers
+/// compact branch hazards. In future this pass can be extended to other pipeline
+/// hazards, such as various MIPS1 hazards, processor errata that require
+/// instruction reorganization, etc.
+///
+/// This pass has to run after the delay slot filler as that pass can introduce
+/// pipeline hazards, hence the existing hazard recognizer is not suitable.
+///
+/// Hazards handled: forbidden slots for MIPSR6.
+///
+/// A forbidden slot hazard occurs when a compact branch instruction is executed
+/// and the adjacent instruction in memory is a control transfer instruction such
+/// as a branch or jump, ERET, ERETNC, DERET, WAIT and PAUSE.
+///
+/// For example:
+///
+/// 0x8004  bnec  a1,v0,<P+0x18>
+/// 0x8008  beqc  a1,a2,<P+0x54>
+///
+/// In such cases, the processor is required to signal a Reserved Instruction
+/// exception.
+///
+/// Here, if the instruction at 0x8004 is executed, the processor will raise an
+/// exception as there is a control transfer instruction at 0x8008.
+///
+/// There are two sources of forbidden slot hazards:
+///
+/// A) A previous pass has created a compact branch directly.
+/// B) Transforming a delay slot branch into compact branch. This case can be
+///    difficult to process as lookahead for hazards is insufficent, as
+///    backwards delay slot fillling can also produce hazards in previously
+///    processed instuctions.
+///
+//===----------------------------------------------------------------------===//
+
+#include "Mips.h"
+#include "MipsInstrInfo.h"
+#include "MipsSEInstrInfo.h"
+#include "MipsTargetMachine.h"
+#include "llvm/IR/Function.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Target/TargetInstrInfo.h"
+#include "llvm/Target/TargetMachine.h"
+#include "llvm/Target/TargetRegisterInfo.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "mips-hazard-schedule"
+
+STATISTIC(NumInsertedNops, "Number of nops inserted");
+
+namespace {
+
+typedef MachineBasicBlock::iterator Iter;
+typedef MachineBasicBlock::reverse_iterator ReverseIter;
+
+class MipsHazardSchedule : public MachineFunctionPass {
+
+public:
+  MipsHazardSchedule(TargetMachine &tm) : MachineFunctionPass(ID), TM(tm) {}
+
+  const char *getPassName() const override { return "Mips Hazard Schedule"; }
+
+  bool runOnMachineFunction(MachineFunction &F) override;
+
+private:
+  static char ID;
+  const TargetMachine &TM;
+};
+
+char MipsHazardSchedule::ID = 0;
+} // end of anonymous namespace
+
+/// Returns a pass that clears pipeline hazards.
+FunctionPass *llvm::createMipsHazardSchedule(MipsTargetMachine &tm) {
+  return new MipsHazardSchedule(tm);
+}
+
+bool MipsHazardSchedule::runOnMachineFunction(MachineFunction &MF) {
+
+  const MipsSubtarget *STI =
+      &static_cast<const MipsSubtarget &>(MF.getSubtarget());
+
+  // Forbidden slot hazards are only defined for MIPSR6.
+  if (!STI->hasMips32r6() || STI->inMicroMipsMode())
+    return false;
+
+  bool Changed = false;
+  const MipsInstrInfo *TII = STI->getInstrInfo();
+
+  for (MachineFunction::iterator FI = MF.begin(); FI != MF.end(); ++FI) {
+    for (Iter I = FI->begin(); I != FI->end(); ++I) {
+
+      // Forbidden slot hazard handling. Use lookahead over state.
+      if (!TII->HasForbiddenSlot(*I))
+        continue;
+
+      bool InsertNop = false;
+      // Next instruction in the basic block.
+      if (std::next(I) != FI->end() &&
+          !TII->SafeInForbiddenSlot(*std::next(I))) {
+        InsertNop = true;
+      } else {
+        // Next instruction in the physical successor basic block.
+        for (auto *Succ : FI->successors()) {
+          if (FI->isLayoutSuccessor(Succ) &&
+              Succ->getFirstNonDebugInstr() != Succ->end() &&
+              !TII->SafeInForbiddenSlot(*Succ->getFirstNonDebugInstr())) {
+            InsertNop = true;
+            break;
+          }
+        }
+      }
+
+      if (InsertNop) {
+        Changed = true;
+        MIBundleBuilder(I)
+            .append(BuildMI(MF, I->getDebugLoc(), TII->get(Mips::NOP)));
+        NumInsertedNops++;
+      }
+    }
+  }
+  return Changed;
+}

lib/Target/Mips/MipsISelLowering.cpp

MipsTargetLowering::emitAtomicBinary(MachineInstr *MI, MachineBasicBlock *BB,
 
   MachineFunction *MF = BB->getParent();
   MachineRegisterInfo &RegInfo = MF->getRegInfo();
   const TargetRegisterClass *RC = getRegClassFor(MVT::getIntegerVT(Size * 8));
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   DebugLoc DL = MI->getDebugLoc();
   unsigned LL, SC, AND, NOR, ZERO, BEQ;
 
+  // FIXME: The below code should check for the ISA to emit the correct 64bit
+  // operations when the size is 4.
   if (Size == 4) {
     if (isMicroMips) {
       LL = Mips::LL_MM;
       SC = Mips::SC_MM;
     } else {
       LL = Subtarget.hasMips32r6() ? Mips::LL_R6 : Mips::LL;
       SC = Subtarget.hasMips32r6() ? Mips::SC_R6 : Mips::SC;
     }

lib/Target/Mips/MipsInstrFormats.td

class MipsInst<dag outs, dag ins, string asmstr, list<dag> pattern,
 
   let AsmString   = asmstr;
   let Pattern     = pattern;
   let Itinerary   = itin;
 
   //
   // Attributes specific to Mips instructions...
   //
-  bits<4> FormBits = Form.Value;
+  bits<4> FormBits     = Form.Value;
+  bit isCTI            = 0; // Any form of Control Transfer Instruction.
+                            // Required for MIPSR6
+  bit hasForbiddenSlot = 0; // Instruction has a forbidden slot.
 
   // TSFlags layout should be kept in sync with MipsInstrInfo.h.
   let TSFlags{3-0}   = FormBits;
+  let TSFlags{4}     = isCTI;
+  let TSFlags{5}     = hasForbiddenSlot;
 
   let DecoderNamespace = "Mips";
 
   field bits<32> SoftFail = 0;
 }
 
 // Mips32/64 Instruction Format
 class InstSE<dag outs, dag ins, string asmstr, list<dag> pattern,

lib/Target/Mips/MipsInstrInfo.h

public:
   ReverseBranchCondition(SmallVectorImpl<MachineOperand> &Cond) const override;
 
   BranchType AnalyzeBranch(MachineBasicBlock &MBB, MachineBasicBlock *&TBB,
                            MachineBasicBlock *&FBB,
                            SmallVectorImpl<MachineOperand> &Cond,
                            bool AllowModify,
                            SmallVectorImpl<MachineInstr*> &BranchInstrs) const;
 
+  /// Determine the opcode of a non-delay slot form for a branch if one exists.
+  unsigned getEquivalentCompactForm(const MachineBasicBlock::iterator I) const;
+
+  /// Predicate to determine if an instruction can go in a forbidden slot.
+  bool SafeInForbiddenSlot(const MachineInstr &MI) const;
+
+  /// Predicate to determine if an instruction has a forbidden slot.
+  bool HasForbiddenSlot(const MachineInstr &MI) const;
+
   /// Insert nop instruction when hazard condition is found
   void insertNoop(MachineBasicBlock &MBB,
                   MachineBasicBlock::iterator MI) const override;
 
   /// getRegisterInfo - TargetInstrInfo is a superset of MRegister info.  As
   /// such, whenever a client has an instance of instruction info, it should
   /// always be able to get register info as well (through this method).
   ///

lib/Target/Mips/MipsInstrInfo.cpp

if (LastOpc != UncondBrOpc)
     return BT_None;
 
   AnalyzeCondBr(SecondLastInst, SecondLastOpc, TBB, Cond);
   FBB = LastInst->getOperand(0).getMBB();
 
   return BT_CondUncond;
 }
 
+/// getEquivalentCompactForm - Return the corresponding compact (no delay slot)
+/// form of a branch.
+unsigned MipsInstrInfo::getEquivalentCompactForm(
+    const MachineBasicBlock::iterator I) const {
+  unsigned Opcode = I->getOpcode();
+  bool canUseShortMMBranches =
+      Subtarget.inMicroMipsMode() &&
+      (Opcode == Mips::BNE || Opcode == Mips::BEQ) &&
+      I->getOperand(1).getReg() == Subtarget.getABI().GetZeroReg();
+
+  if (Subtarget.hasMips32r6() || canUseShortMMBranches) {
+    switch (Opcode) {
+    case Mips::B:
+      return Mips::BC;
+    case Mips::BAL:
+      return Mips::BALC;
+    case Mips::BEQ:
+      if (canUseShortMMBranches)
+        return Mips::BEQZC_MM;
+      else
+        return Mips::BEQC;
+    case Mips::BNE:
+      if (canUseShortMMBranches)
+        return Mips::BNEZC_MM;
+      else
+        return Mips::BNEC;
+    case Mips::BGE:
+      return Mips::BGEC;
+    case Mips::BGEU:
+      return Mips::BGEUC;
+    case Mips::BGEZ:
+      return Mips::BGEZC;
+    case Mips::BGTZ:
+      return Mips::BGTZC;
+    case Mips::BLEZ:
+      return Mips::BLEZC;
+    case Mips::BLT:
+      return Mips::BLTC;
+    case Mips::BLTU:
+      return Mips::BLTUC;
+    case Mips::BLTZ:
+      return Mips::BLTZC;
+    default:
+      return 0;
+    }
+  }
+
+  return 0;
+}
+
+/// Predicate for distingushing between control transfer instructions and all
+/// other instructions for handling forbidden slots. Consider inline assembly
+/// as unsafe as well.
+bool MipsInstrInfo::SafeInForbiddenSlot(const MachineInstr &MI) const {
+  if (MI.isInlineAsm())
+    return false;
+
+  return (MI.getDesc().TSFlags & MipsII::IsCTI) == 0;
+
+}
+
+/// Predicate for distingushing instructions that have forbidden slots.
+bool MipsInstrInfo::HasForbiddenSlot(const MachineInstr &MI) const {
+  return (MI.getDesc().TSFlags & MipsII::HasForbiddenSlot) != 0;
+}
+
 /// Return the number of bytes of code the specified instruction may be.
 unsigned MipsInstrInfo::GetInstSizeInBytes(const MachineInstr *MI) const {
   switch (MI->getOpcode()) {
   default:
     return MI->getDesc().getSize();
   case  TargetOpcode::INLINEASM: {       // Inline Asm: Variable size.
     const MachineFunction *MF = MI->getParent()->getParent();
     const char *AsmStr = MI->getOperand(0).getSymbolName();
     return getInlineAsmLength(AsmStr, *MF->getTarget().getMCAsmInfo());
   }
   case Mips::CONSTPOOL_ENTRY:
     // If this machine instr is a constant pool entry, its size is recorded as
     // operand #2.
     return MI->getOperand(2).getImm();
   }
 }
 
 MachineInstrBuilder
 MipsInstrInfo::genInstrWithNewOpc(unsigned NewOpc,
                                   MachineBasicBlock::iterator I) const {
   MachineInstrBuilder MIB;
+  bool BranchWithZeroOperand = false;
+
+  // Certain branches have two forms: e.g beq $1, $zero, dst vs beqz $1, dest
+  // Pick the zero form of the branch for readable assembly and for greater
+  // branch distance in non-microMIPS mode.
+  if (I->isBranch() && I->getOperand(1).isReg() &&
+      // FIXME: Certain atomic sequences on mips64 generate 32bit references to
+      // Mips::ZERO, which is incorrect. This test should be updated to use
+      // Subtarget.getABI().GetZeroReg() when those atomic sequences and others
+      // are fixed.
+      (I->getOperand(1).getReg() == Mips::ZERO ||
+       I->getOperand(1).getReg() == Mips::ZERO_64)) {
+    BranchWithZeroOperand = true;
+    switch (NewOpc) {
+    case Mips::BEQC:
+      NewOpc = Mips::BEQZC;
+      break;
+    case Mips::BNEC:
+      NewOpc = Mips::BNEZC;
+      break;
+    case Mips::BGEC:
+      NewOpc = Mips::BGEZC;
+      break;
+    case Mips::BLTC:
+      NewOpc = Mips::BLTZC;
+      break;
+    case Mips::BNEZC_MM:
+    case Mips::BEQZC_MM:
+      break;
+    default:
+      BranchWithZeroOperand = false;
+      break;
+    }
+  }
+
   MIB = BuildMI(*I->getParent(), I, I->getDebugLoc(), get(NewOpc));
 
   for (unsigned J = 0, E = I->getDesc().getNumOperands(); J < E; ++J)
+    if (!(BranchWithZeroOperand && (J == 1)))
       MIB.addOperand(I->getOperand(J));
 
   MIB.setMemRefs(I->memoperands_begin(), I->memoperands_end());
   return MIB;
 }

lib/Target/Mips/MipsInstrInfo.td

class MipsInstAlias<string Asm, dag Result, bit Emit = 0b1> :
   InstAlias<Asm, Result, Emit>, PredicateControl;
 
 class IsCommutable {
   bit isCommutable = 1;
 }
 
 class IsBranch {
   bit isBranch = 1;
+  bit isCTI = 1;
 }
 
 class IsReturn {
   bit isReturn = 1;
+  bit isCTI = 1;
 }
 
 class IsCall {
   bit isCall = 1;
+  bit isCTI = 1;
 }
 
 class IsTailCall {
   bit isCall = 1;
   bit isTerminator = 1;
   bit isReturn = 1;
   bit isBarrier = 1;
   bit hasExtraSrcRegAllocReq = 1;
   bit isCodeGenOnly = 1;
+  bit isCTI = 1;
 }
 
 class IsAsCheapAsAMove {
   bit isAsCheapAsAMove = 1;
 }
 
 class NeverHasSideEffects {
   bit hasSideEffects = 0;
class CBranch<string opstr, DAGOperand opnd, PatFrag cond_op,
   InstSE<(outs), (ins RO:$rs, RO:$rt, opnd:$offset),
          !strconcat(opstr, "\t$rs, $rt, $offset"),
          [(brcond (i32 (cond_op RO:$rs, RO:$rt)), bb:$offset)], II_BCC,
          FrmI, opstr> {
   let isBranch = 1;
   let isTerminator = 1;
   let hasDelaySlot = DelaySlot;
   let Defs = [AT];
+  bit isCTI = 1;
 }
 
 class CBranchZero<string opstr, DAGOperand opnd, PatFrag cond_op,
                   RegisterOperand RO, bit DelaySlot = 1> :
   InstSE<(outs), (ins RO:$rs, opnd:$offset),
          !strconcat(opstr, "\t$rs, $offset"),
          [(brcond (i32 (cond_op RO:$rs, 0)), bb:$offset)], II_BCCZ,
          FrmI, opstr> {
   let isBranch = 1;
   let isTerminator = 1;
   let hasDelaySlot = DelaySlot;
   let Defs = [AT];
+  bit isCTI = 1;
 }
 
 // SetCC
 class SetCC_R<string opstr, PatFrag cond_op, RegisterOperand RO> :
   InstSE<(outs GPR32Opnd:$rd), (ins RO:$rs, RO:$rt),
          !strconcat(opstr, "\t$rd, $rs, $rt"),
          [(set GPR32Opnd:$rd, (cond_op RO:$rs, RO:$rt))],
          II_SLT_SLTU, FrmR, opstr>;
class JumpFJ<DAGOperand opnd, string opstr, SDPatternOperator operator,
              SDPatternOperator targetoperator, string bopstr> :
   InstSE<(outs), (ins opnd:$target), !strconcat(opstr, "\t$target"),
          [(operator targetoperator:$target)], II_J, FrmJ, bopstr> {
   let isTerminator=1;
   let isBarrier=1;
   let hasDelaySlot = 1;
   let DecoderMethod = "DecodeJumpTarget";
   let Defs = [AT];
+  bit isCTI = 1;
 }
 
 // Unconditional branch
 class UncondBranch<Instruction BEQInst> :
   PseudoSE<(outs), (ins brtarget:$offset), [(br bb:$offset)], II_B>,
   PseudoInstExpansion<(BEQInst ZERO, ZERO, brtarget:$offset)> {
   let isBranch = 1;
   let isTerminator = 1;
   let isBarrier = 1;
   let hasDelaySlot = 1;
   let AdditionalPredicates = [RelocPIC];
   let Defs = [AT];
+  bit isCTI = 1;
 }
 
 // Base class for indirect branch and return instruction classes.
-let isTerminator=1, isBarrier=1, hasDelaySlot = 1 in
+let isTerminator=1, isBarrier=1, hasDelaySlot = 1, isCTI = 1 in
 class JumpFR<string opstr, RegisterOperand RO,
              SDPatternOperator operator = null_frag>:
   InstSE<(outs), (ins RO:$rs), "jr\t$rs", [(operator RO:$rs)], II_JR,
          FrmR, opstr>;
 
 // Indirect branch
 class IndirectBranch<string opstr, RegisterOperand RO> : JumpFR<opstr, RO> {
   let isBranch = 1;
   let isIndirectBranch = 1;
 }
 
 // Jump and Link (Call)
-let isCall=1, hasDelaySlot=1, Defs = [RA] in {
+let isCall=1, hasDelaySlot=1, isCTI=1, Defs = [RA] in {
   class JumpLink<string opstr, DAGOperand opnd> :
     InstSE<(outs), (ins opnd:$target), !strconcat(opstr, "\t$target"),
            [(MipsJmpLink tglobaladdr:$target)], II_JAL, FrmJ, opstr> {
     let DecoderMethod = "DecodeJumpTarget";
   }
 
   class JumpLinkRegPseudo<RegisterOperand RO, Instruction JALRInst,
                           Register RetReg, RegisterOperand ResRO = RO>:
class BGEZAL_FT<string opstr, DAGOperand opnd,
     InstSE<(outs), (ins RO:$rs, opnd:$offset),
            !strconcat(opstr, "\t$rs, $offset"), [], II_BCCZAL, FrmI, opstr> {
     let hasDelaySlot = DelaySlot;
   }
 
 }
 
 let isCall = 1, isTerminator = 1, isReturn = 1, isBarrier = 1, hasDelaySlot = 1,
-    hasExtraSrcRegAllocReq = 1, Defs = [AT] in {
+    hasExtraSrcRegAllocReq = 1, isCTI = 1, Defs = [AT] in {
   class TailCall<Instruction JumpInst> :
     PseudoSE<(outs), (ins calltarget:$target), [], II_J>,
     PseudoInstExpansion<(JumpInst jmptarget:$target)>;
 
   class TailCallReg<RegisterOperand RO, Instruction JRInst,
                     RegisterOperand ResRO = RO> :
     PseudoSE<(outs), (ins RO:$rs), [(MipsTailCall RO:$rs)], II_JR>,
     PseudoInstExpansion<(JRInst ResRO:$rs)>;
 }
 
 class BAL_BR_Pseudo<Instruction RealInst> :
   PseudoSE<(outs), (ins brtarget:$offset), [], II_BCCZAL>,
   PseudoInstExpansion<(RealInst ZERO, brtarget:$offset)> {
   let isBranch = 1;
   let isTerminator = 1;
   let isBarrier = 1;
   let hasDelaySlot = 1;
   let Defs = [RA];
+  bit isCTI = 1;
 }
 
+let isCTI = 1 in {
 // Syscall
 class SYS_FT<string opstr> :
   InstSE<(outs), (ins uimm20:$code_),
          !strconcat(opstr, "\t$code_"), [], NoItinerary, FrmI, opstr>;
 // Break
 class BRK_FT<string opstr> :
   InstSE<(outs), (ins uimm10:$code_1, uimm10:$code_2),
          !strconcat(opstr, "\t$code_1, $code_2"), [], NoItinerary,
          FrmOther, opstr>;
 
 // (D)Eret
 class ER_FT<string opstr> :
   InstSE<(outs), (ins),
          opstr, [], NoItinerary, FrmOther, opstr>;
 
+// Wait
+class WAIT_FT<string opstr> :
+  InstSE<(outs), (ins), opstr, [], NoItinerary, FrmOther, opstr>;
+}
+
 // Interrupts
 class DEI_FT<string opstr, RegisterOperand RO> :
   InstSE<(outs RO:$rt), (ins),
          !strconcat(opstr, "\t$rt"), [], NoItinerary, FrmOther, opstr>;
 
-// Wait
-class WAIT_FT<string opstr> :
-  InstSE<(outs), (ins), opstr, [], NoItinerary, FrmOther, opstr>;
-
 // Sync
 let hasSideEffects = 1 in
 class SYNC_FT<string opstr> :
   InstSE<(outs), (ins i32imm:$stype), "sync $stype", [(MipsSync imm:$stype)],
          NoItinerary, FrmOther, opstr>;
 
 class SYNCI_FT<string opstr> :
   InstSE<(outs), (ins mem_simm16:$addr), !strconcat(opstr, "\t$addr"), [],
          NoItinerary, FrmOther, opstr> {
   let hasSideEffects = 1;
   let DecoderMethod = "DecodeSyncI";
 }
 
+let hasSideEffects = 1, isCTI = 1 in {
 class TEQ_FT<string opstr, RegisterOperand RO, Operand ImmOp> :
   InstSE<(outs), (ins RO:$rs, RO:$rt, ImmOp:$code_),
          !strconcat(opstr, "\t$rs, $rt, $code_"), [], NoItinerary,
-         FrmI, opstr> {
-  let hasSideEffects = 1;
-}
+         FrmI, opstr>;
 
 class TEQI_FT<string opstr, RegisterOperand RO> :
   InstSE<(outs), (ins RO:$rs, simm16:$imm16),
-         !strconcat(opstr, "\t$rs, $imm16"), [], NoItinerary, FrmOther, opstr> {
-  let hasSideEffects = 1;
+         !strconcat(opstr, "\t$rs, $imm16"), [], NoItinerary, FrmOther, opstr>;
 }
 
 // Mul, Div
 class Mult<string opstr, InstrItinClass itin, RegisterOperand RO,
            list<Register> DefRegs> :
   InstSE<(outs), (ins RO:$rs, RO:$rt), !strconcat(opstr, "\t$rs, $rt"), [],
          itin, FrmR, opstr> {
   let isCommutable = 1;
InstSE<(outs RO:$rd), (ins RD:$rt, uimm16:$sel),
          !strconcat(asmstr, "\t$rt, $rd, $sel"), [], NoItinerary, FrmFR>;
 
 class TrapBase<Instruction RealInst>
   : PseudoSE<(outs), (ins), [(trap)], NoItinerary>,
     PseudoInstExpansion<(RealInst 0, 0)> {
   let isBarrier = 1;
   let isTerminator = 1;
   let isCodeGenOnly = 1;
+  let isCTI = 1;
 }
 
 //===----------------------------------------------------------------------===//
 // Pseudo instructions
 //===----------------------------------------------------------------------===//
 
 // Return RA.
-let isReturn=1, isTerminator=1, hasDelaySlot=1, isBarrier=1, hasCtrlDep=1 in
+let isReturn=1, isTerminator=1, isBarrier=1, hasCtrlDep=1, isCTI=1 in {
+let hasDelaySlot=1 in
 def RetRA : PseudoSE<(outs), (ins), [(MipsRet)]>;
 
-let isReturn=1, isTerminator=1, isBarrier=1, hasCtrlDep=1, hasSideEffects=1 in
+let hasSideEffects=1 in
 def ERet : PseudoSE<(outs), (ins), [(MipsERet)]>;
+}
 
 let Defs = [SP], Uses = [SP], hasSideEffects = 1 in {
 def ADJCALLSTACKDOWN : MipsPseudo<(outs), (ins i32imm:$amt),
                                   [(callseq_start timm:$amt)]>;
 def ADJCALLSTACKUP   : MipsPseudo<(outs), (ins i32imm:$amt1, i32imm:$amt2),
                                   [(callseq_end timm:$amt1, timm:$amt2)]>;
 }
 

 class PseudoIndirectBranchBase<RegisterOperand RO> :
     MipsPseudo<(outs), (ins RO:$rs), [(brind RO:$rs)],
                II_IndirectBranchPseudo> {
   let isTerminator=1;
   let isBarrier=1;
   let hasDelaySlot = 1;
   let isBranch = 1;
   let isIndirectBranch = 1;
+  bit isCTI = 1;
 }
 
 def PseudoIndirectBranch : PseudoIndirectBranchBase<GPR32Opnd>;
 
 // Return instructions are matched as a RetRA instruction, then are expanded
 // into PseudoReturn/PseudoReturn64 after register allocation. Finally,
 // MipsAsmPrinter expands this into JR, JR64, JALR, or JALR64 depending on the
 // ISA.
 class PseudoReturnBase<RegisterOperand RO> : MipsPseudo<(outs), (ins RO:$rs),
                                                         [], II_ReturnPseudo> {
   let isTerminator = 1;
   let isBarrier = 1;
   let hasDelaySlot = 1;
   let isReturn = 1;
   let isCodeGenOnly = 1;
   let hasCtrlDep = 1;
   let hasExtraSrcRegAllocReq = 1;
+  bit isCTI = 1;
 }
 
 def PseudoReturn : PseudoReturnBase<GPR32Opnd>;
 
 // Exception handling related node and instructions.
 // The conversion sequence is:
 // ISD::EH_RETURN -> MipsISD::EH_RETURN ->
 // MIPSeh_return -> (stack change + indirect branch)
 //
 // MIPSeh_return takes the place of regular return instruction
 // but takes two arguments (V1, V0) which are used for storing
 // the offset and return address respectively.
 def SDT_MipsEHRET : SDTypeProfile<0, 2, [SDTCisInt<0>, SDTCisPtrTy<1>]>;
 
 def MIPSehret : SDNode<"MipsISD::EH_RETURN", SDT_MipsEHRET,
                       [SDNPHasChain, SDNPOptInGlue, SDNPVariadic]>;
 
-let Uses = [V0, V1], isTerminator = 1, isReturn = 1, isBarrier = 1 in {
+let Uses = [V0, V1], isTerminator = 1, isReturn = 1, isBarrier = 1, isCTI = 1 in {
   def MIPSeh_return32 : MipsPseudo<(outs), (ins GPR32:$spoff, GPR32:$dst),
                                 [(MIPSehret GPR32:$spoff, GPR32:$dst)]>;
   def MIPSeh_return64 : MipsPseudo<(outs), (ins GPR64:$spoff,
                                                 GPR64:$dst),
                                 [(MIPSehret GPR64:$spoff, GPR64:$dst)]>;
 }
 
 /// Multiply and Divide Instructions.

 def MTC0 : MTC3OP<"mtc0", COP0Opnd, GPR32Opnd>, MFC3OP_FM<0x10, 4>, ISA_MIPS32;
 def MFC2 : MFC3OP<"mfc2", GPR32Opnd, COP2Opnd>, MFC3OP_FM<0x12, 0>;
 def MTC2 : MTC3OP<"mtc2", COP2Opnd, GPR32Opnd>, MFC3OP_FM<0x12, 4>;
 
 class Barrier<string asmstr> : InstSE<(outs), (ins), asmstr, [], NoItinerary,
                                       FrmOther, asmstr>;
 def SSNOP : MMRel, StdMMR6Rel, Barrier<"ssnop">, BARRIER_FM<1>;
 def EHB : MMRel, Barrier<"ehb">, BARRIER_FM<3>;
+
+let isCTI = 1 in
 def PAUSE : MMRel, StdMMR6Rel, Barrier<"pause">, BARRIER_FM<5>, ISA_MIPS32R2;
 
 // JR_HB and JALR_HB are defined here using the new style naming
 // scheme because some of this code is shared with Mips32r6InstrInfo.td
 // and because of that it doesn't follow the naming convention of the
 // rest of the file. To avoid a mixture of old vs new style, the new
 // style was chosen.
 class JR_HB_DESC_BASE<string instr_asm, RegisterOperand GPROpnd> {

 
 class JR_HB_DESC : InstSE<(outs), (ins), "", [], NoItinerary, FrmJ>,
                    JR_HB_DESC_BASE<"jr.hb", GPR32Opnd> {
   let isBranch=1;
   let isIndirectBranch=1;
   let hasDelaySlot=1;
   let isTerminator=1;
   let isBarrier=1;
+  bit isCTI = 1;
 }
 
 class JALR_HB_DESC : InstSE<(outs), (ins), "", [], NoItinerary, FrmJ>,
                      JALR_HB_DESC_BASE<"jalr.hb", GPR32Opnd> {
   let isIndirectBranch=1;
   let hasDelaySlot=1;
+  bit isCTI = 1;
 }
 
 class JR_HB_ENC : JR_HB_FM<8>;
 class JALR_HB_ENC : JALR_HB_FM<9>;
 
 def JR_HB : JR_HB_DESC, JR_HB_ENC, ISA_MIPS32_NOT_32R6_64R6;
 def JALR_HB : JALR_HB_DESC, JALR_HB_ENC, ISA_MIPS32;
 

 def JalTwoReg : MipsAsmPseudoInst<(outs GPR32Opnd:$rd), (ins GPR32Opnd:$rs),
                       "jal\t$rd, $rs"> ;
 def JalOneReg : MipsAsmPseudoInst<(outs), (ins GPR32Opnd:$rs),
                       "jal\t$rs"> ;
 
 def NORImm : MipsAsmPseudoInst<(outs), (ins GPR32Opnd:$rs, GPR32Opnd:$rt, simm16:$imm),
                                "nor\t$rs, $rt, $imm"> ;
 
-let hasDelaySlot = 1 in {
+let hasDelaySlot = 1, isCTI = 1 in {
 def BneImm : MipsAsmPseudoInst<(outs GPR32Opnd:$rt),
                                (ins imm64:$imm64, brtarget:$offset),
                                "bne\t$rt, $imm64, $offset">;
 def BeqImm : MipsAsmPseudoInst<(outs GPR32Opnd:$rt),
                                (ins imm64:$imm64, brtarget:$offset),
                                "beq\t$rt, $imm64, $offset">;
 
 class CondBranchPseudo<string instr_asm> :

 def BLEL : CondBranchPseudo<"blel">, ISA_MIPS2_NOT_32R6_64R6;
 def BGEL : CondBranchPseudo<"bgel">, ISA_MIPS2_NOT_32R6_64R6;
 def BGTL : CondBranchPseudo<"bgtl">, ISA_MIPS2_NOT_32R6_64R6;
 def BLTUL: CondBranchPseudo<"bltul">, ISA_MIPS2_NOT_32R6_64R6;
 def BLEUL: CondBranchPseudo<"bleul">, ISA_MIPS2_NOT_32R6_64R6;
 def BGEUL: CondBranchPseudo<"bgeul">, ISA_MIPS2_NOT_32R6_64R6;
 def BGTUL: CondBranchPseudo<"bgtul">, ISA_MIPS2_NOT_32R6_64R6;
 
+let isCTI = 1 in
 class CondBranchImmPseudo<string instr_asm> :
   MipsAsmPseudoInst<(outs), (ins GPR32Opnd:$rs, imm64:$imm, brtarget:$offset),
                     !strconcat(instr_asm, "\t$rs, $imm, $offset")>;
 
 def BLTImmMacro  : CondBranchImmPseudo<"blt">;
 def BLEImmMacro  : CondBranchImmPseudo<"ble">;
 def BGEImmMacro  : CondBranchImmPseudo<"bge">;
 def BGTImmMacro  : CondBranchImmPseudo<"bgt">;

lib/Target/Mips/MipsSEInstrInfo.cpp

unsigned MipsSEInstrInfo::getOppositeBranchOpc(unsigned Opc) const {
   case Mips::BGTZ64: return Mips::BLEZ64;
   case Mips::BGEZ64: return Mips::BLTZ64;
   case Mips::BLTZ64: return Mips::BGEZ64;
   case Mips::BLEZ64: return Mips::BGTZ64;
   case Mips::BC1T:   return Mips::BC1F;
   case Mips::BC1F:   return Mips::BC1T;
   case Mips::BEQZC_MM: return Mips::BNEZC_MM;
   case Mips::BNEZC_MM: return Mips::BEQZC_MM;
+  case Mips::BEQZC:  return Mips::BNEZC;
+  case Mips::BNEZC:  return Mips::BEQZC;
+  case Mips::BEQC:   return Mips::BNEC;
+  case Mips::BNEC:   return Mips::BEQC;
+  case Mips::BGTZC:  return Mips::BLEZC;
+  case Mips::BGEZC:  return Mips::BLTZC;
+  case Mips::BLTZC:  return Mips::BGEZC;
+  case Mips::BLEZC:  return Mips::BGTZC;
   }
 }
 
 /// Adjust SP by Amount bytes.
 void MipsSEInstrInfo::adjustStackPtr(unsigned SP, int64_t Amount,
                                      MachineBasicBlock &MBB,
                                      MachineBasicBlock::iterator I) const {
   MipsABIInfo ABI = Subtarget.getABI();
   DebugLoc DL;
   unsigned ADDu = ABI.GetPtrAdduOp();
   unsigned ADDiu = ABI.GetPtrAddiuOp();
 
   if (Amount == 0)
     return;
 
   if (isInt<16>(Amount))// addi sp, sp, amount
     BuildMI(MBB, I, DL, get(ADDiu), SP).addReg(SP).addImm(Amount);
   else { // Expand immediate that doesn't fit in 16-bit.
     unsigned Reg = loadImmediate(Amount, MBB, I, DL, nullptr);
     BuildMI(MBB, I, DL, get(ADDu), SP).addReg(SP).addReg(Reg, RegState::Kill);
   }
 }
 
 /// This function generates the sequence of instructions needed to get the
 /// result of adding register REG and immediate IMM.
 unsigned
 MipsSEInstrInfo::loadImmediate(int64_t Imm, MachineBasicBlock &MBB,
                                MachineBasicBlock::iterator II, DebugLoc DL,
                                unsigned *NewImm) const {
   MipsAnalyzeImmediate AnalyzeImm;
   const MipsSubtarget &STI = Subtarget;
   MachineRegisterInfo &RegInfo = MBB.getParent()->getRegInfo();
   unsigned Size = STI.isABI_N64() ? 64 : 32;
   unsigned LUi = STI.isABI_N64() ? Mips::LUi64 : Mips::LUi;
   unsigned ZEROReg = STI.isABI_N64() ? Mips::ZERO_64 : Mips::ZERO;
   const TargetRegisterClass *RC = STI.isABI_N64() ?
     &Mips::GPR64RegClass : &Mips::GPR32RegClass;
   bool LastInstrIsADDiu = NewImm;
 
   const MipsAnalyzeImmediate::InstSeq &Seq =
     AnalyzeImm.Analyze(Imm, Size, LastInstrIsADDiu);
   MipsAnalyzeImmediate::InstSeq::const_iterator Inst = Seq.begin();
 
   assert(Seq.size() && (!LastInstrIsADDiu || (Seq.size() > 1)));
 
   // The first instruction can be a LUi, which is different from other
   // instructions (ADDiu, ORI and SLL) in that it does not have a register
   // operand.
   unsigned Reg = RegInfo.createVirtualRegister(RC);
 
   if (Inst->Opc == LUi)
     BuildMI(MBB, II, DL, get(LUi), Reg).addImm(SignExtend64<16>(Inst->ImmOpnd));
   else
     BuildMI(MBB, II, DL, get(Inst->Opc), Reg).addReg(ZEROReg)
       .addImm(SignExtend64<16>(Inst->ImmOpnd));
 
   // Build the remaining instructions in Seq.
   for (++Inst; Inst != Seq.end() - LastInstrIsADDiu; ++Inst)
     BuildMI(MBB, II, DL, get(Inst->Opc), Reg).addReg(Reg, RegState::Kill)
       .addImm(SignExtend64<16>(Inst->ImmOpnd));
 
   if (LastInstrIsADDiu)
     *NewImm = Inst->ImmOpnd;
 
   return Reg;
 }
 
 unsigned MipsSEInstrInfo::getAnalyzableBrOpc(unsigned Opc) const {
   return (Opc == Mips::BEQ    || Opc == Mips::BNE    || Opc == Mips::BGTZ   ||
           Opc == Mips::BGEZ   || Opc == Mips::BLTZ   || Opc == Mips::BLEZ   ||
           Opc == Mips::BEQ64  || Opc == Mips::BNE64  || Opc == Mips::BGTZ64 ||
           Opc == Mips::BGEZ64 || Opc == Mips::BLTZ64 || Opc == Mips::BLEZ64 ||
           Opc == Mips::BC1T   || Opc == Mips::BC1F   || Opc == Mips::B      ||
-          Opc == Mips::J || Opc == Mips::BEQZC_MM || Opc == Mips::BNEZC_MM) ?
-         Opc : 0;
+          Opc == Mips::J  || Opc == Mips::BEQZC_MM || Opc == Mips::BNEZC_MM ||
+          Opc == Mips::BEQC   || Opc == Mips::BNEC   || Opc == Mips::BLTC   ||
+          Opc == Mips::BGEC   || Opc == Mips::BLTUC  || Opc == Mips::BGEUC  ||
+          Opc == Mips::BGTZC  || Opc == Mips::BLEZC  || Opc == Mips::BGEZC  ||
+          Opc == Mips::BGTZC  || Opc == Mips::BEQZC  || Opc == Mips::BNEZC  ||
+          Opc == Mips::BC) ? Opc : 0;
 }
 
 void MipsSEInstrInfo::expandRetRA(MachineBasicBlock &MBB,
                                   MachineBasicBlock::iterator I) const {
   if (Subtarget.isGP64bit())
     BuildMI(MBB, I, I->getDebugLoc(), get(Mips::PseudoReturn64))
         .addReg(Mips::RA_64);
   else

lib/Target/Mips/MipsTargetMachine.cpp

TargetIRAnalysis MipsTargetMachine::getTargetIRAnalysis() {
   });
 }
 
 // Implemented by targets that want to run passes immediately before
 // machine code is emitted. return true if -print-machineinstrs should
 // print out the code after the passes.
 void MipsPassConfig::addPreEmitPass() {
   MipsTargetMachine &TM = getMipsTargetMachine();
+
+  // The delay slot filler pass can potientially create forbidden slot (FS)
+  // hazards for MIPSR6 which the hazard schedule pass (HSP) will fix. Any
+  // (new) pass that creates compact branches after the HSP must handle FS
+  // hazards itself or be pipelined before the HSP.
   addPass(createMipsDelaySlotFillerPass(TM));
+  addPass(createMipsHazardSchedule(TM));
   addPass(createMipsLongBranchPass(TM));
   addPass(createMipsConstantIslandPass(TM));
 }

test/CodeGen/Mips/analyzebranch.ll

 ; FCC:           nop
 
 ; 32-GPR:        mtc1      $zero, $[[Z:f[0-9]]]
 ; 32-GPR:        mthc1     $zero, $[[Z:f[0-9]]]
 ; 64-GPR:        dmtc1     $zero, $[[Z:f[0-9]]]
 ; GPR:           cmp.lt.d  $[[FGRCC:f[0-9]+]], $[[Z]], $f12
 ; GPR:           mfc1      $[[GPRCC:[0-9]+]], $[[FGRCC]]
 ; GPR-NOT:       not       $[[GPRCC]], $[[GPRCC]]
-; GPR:           bnez      $[[GPRCC]], $BB
+; GPR:           bnezc     $[[GPRCC]], $BB
 
   %cmp = fcmp ogt double %a, 0.000000e+00
   br i1 %cmp, label %if.end6, label %if.else
 
 if.else:                                          ; preds = %entry
   %cmp3 = fcmp ogt double %b, 0.000000e+00
   br i1 %cmp3, label %if.end6, label %return
 

 
 ; FCC:           bc1f $BB
 ; FCC:           nop
 
 ; GPR:           mtc1     $zero, $[[Z:f[0-9]]]
 ; GPR:           cmp.eq.s $[[FGRCC:f[0-9]+]], $f12, $[[Z]]
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC]]
 ; GPR-NOT:       not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           beqz     $[[GPRCC]], $BB
+; 64-GPR         beqzc    $[[GPRCC]], $BB
+; 32-GPR         beqz     $[[GPRCC]], $BB
 
   %cmp = fcmp une float %f, 0.000000e+00
   br i1 %cmp, label %if.then, label %if.end
 
 if.then:                                          ; preds = %entry
   tail call void @abort() noreturn
   unreachable
 
 if.end:                                           ; preds = %entry
   tail call void (...) @f2() nounwind
   ret void
 }
 
 declare void @abort() noreturn nounwind
 
 declare void @f2(...)

test/CodeGen/Mips/atomic.ll

 ; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32   < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS32-ANY -check-prefix=NO-SEB-SEH  -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
 ; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r2 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS32-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
-; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r6 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS32-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
+; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r6 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS32-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=MIPSR6
 ; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips4    < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS64-ANY -check-prefix=NO-SEB-SEH  -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
 ; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64   < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS64-ANY -check-prefix=NO-SEB-SEH  -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
 ; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64r2 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS64-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
-; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64r6 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS64-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=NOT-MICROMIPS
+; RUN: llc -march=mips64el --disable-machine-licm -mcpu=mips64r6 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS64-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=MIPSR6
 ; RUN: llc -march=mipsel --disable-machine-licm -mcpu=mips32r2 -mattr=micromips < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS32-ANY -check-prefix=HAS-SEB-SEH -check-prefix=CHECK-EL -check-prefix=MICROMIPS
 
 ; Keep one big-endian check so that we don't reduce testing, but don't add more
 ; since endianness doesn't affect the body of the atomic operations.
 ; RUN: llc -march=mips   --disable-machine-licm -mcpu=mips32 < %s | FileCheck %s -check-prefix=ALL -check-prefix=MIPS32-ANY -check-prefix=NO-SEB-SEH -check-prefix=CHECK-EB -check-prefix=NOT-MICROMIPS
 
 @x = common global i32 0, align 4
 
 define i32 @AtomicLoadAdd32(i32 signext %incr) nounwind {
 entry:
   %0 = atomicrmw add i32* @x, i32 %incr monotonic
   ret i32 %0
 
 ; ALL-LABEL: AtomicLoadAdd32:
 
 ; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
 ; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
 
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $[[R1:[0-9]+]], 0($[[R0]])
 ; ALL:           addu    $[[R2:[0-9]+]], $[[R1]], $4
 ; ALL:           sc      $[[R2]], 0($[[R0]])
 ; NOT-MICROMIPS: beqz    $[[R2]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R2]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R2]], $[[BB0]]
 }
 
 define i32 @AtomicLoadNand32(i32 signext %incr) nounwind {
 entry:
   %0 = atomicrmw nand i32* @x, i32 %incr monotonic
   ret i32 %0
 
 ; ALL-LABEL: AtomicLoadNand32:
 
 ; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
 ; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
 
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $[[R1:[0-9]+]], 0($[[R0]])
 ; ALL:           and     $[[R3:[0-9]+]], $[[R1]], $4
 ; ALL:           nor     $[[R2:[0-9]+]], $zero, $[[R3]]
 ; ALL:           sc      $[[R2]], 0($[[R0]])
 ; NOT-MICROMIPS: beqz    $[[R2]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R2]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R2]], $[[BB0]]
 }
 
 define i32 @AtomicSwap32(i32 signext %newval) nounwind {
 entry:
   %newval.addr = alloca i32, align 4
   store i32 %newval, i32* %newval.addr, align 4
   %tmp = load i32, i32* %newval.addr, align 4
   %0 = atomicrmw xchg i32* @x, i32 %tmp monotonic
   ret i32 %0
 
 ; ALL-LABEL: AtomicSwap32:
 
 ; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
 ; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)
 
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      ${{[0-9]+}}, 0($[[R0]])
 ; ALL:           sc      $[[R2:[0-9]+]], 0($[[R0]])
 ; NOT-MICROMIPS: beqz    $[[R2]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R2]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R2]], $[[BB0]]
 }
 
 define i32 @AtomicCmpSwap32(i32 signext %oldval, i32 signext %newval) nounwind {
 entry:
   %newval.addr = alloca i32, align 4
   store i32 %newval, i32* %newval.addr, align 4
   %tmp = load i32, i32* %newval.addr, align 4
   %0 = cmpxchg i32* @x, i32 %oldval, i32 %tmp monotonic monotonic
   %1 = extractvalue { i32, i1 } %0, 0
   ret i32 %1
 
 ; ALL-LABEL: AtomicCmpSwap32:
 
 ; MIPS32-ANY:    lw      $[[R0:[0-9]+]], %got(x)
 ; MIPS64-ANY:    ld      $[[R0:[0-9]+]], %got_disp(x)(
 
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $2, 0($[[R0]])
-; ALL:           bne     $2, $4, $[[BB1:[A-Z_0-9]+]]
+; NOT-MICROMIPS: bne     $2, $4, $[[BB1:[A-Z_0-9]+]]
+; MICROMIPS:     bne     $2, $4, $[[BB1:[A-Z_0-9]+]]
+; MIPSR6:        bnec    $2, $4, $[[BB1:[A-Z_0-9]+]]
 ; ALL:           sc      $[[R2:[0-9]+]], 0($[[R0]])
 ; NOT-MICROMIPS: beqz    $[[R2]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R2]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R2]], $[[BB0]]
 ; ALL:       $[[BB1]]:
 }
 
 
 
 @y = common global i8 0, align 1
 
 define signext i8 @AtomicLoadAdd8(i8 signext %incr) nounwind {

 ; ALL:           ll      $[[R10:[0-9]+]], 0($[[R2]])
 ; ALL:           addu    $[[R11:[0-9]+]], $[[R10]], $[[R9]]
 ; ALL:           and     $[[R12:[0-9]+]], $[[R11]], $[[R7]]
 ; ALL:           and     $[[R13:[0-9]+]], $[[R10]], $[[R8]]
 ; ALL:           or      $[[R14:[0-9]+]], $[[R13]], $[[R12]]
 ; ALL:           sc      $[[R14]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R14]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R14]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R14]], $[[BB0]]
 
 ; ALL:           and     $[[R15:[0-9]+]], $[[R10]], $[[R7]]
 ; ALL:           srlv    $[[R16:[0-9]+]], $[[R15]], $[[R5]]
 
 ; NO-SEB-SEH:    sll     $[[R17:[0-9]+]], $[[R16]], 24
 ; NO-SEB-SEH:    sra     $2, $[[R17]], 24
 
 ; HAS-SEB-SEH:   seb     $2, $[[R16]]

 ; ALL:        ll      $[[R10:[0-9]+]], 0($[[R2]])
 ; ALL:        subu    $[[R11:[0-9]+]], $[[R10]], $[[R9]]
 ; ALL:        and     $[[R12:[0-9]+]], $[[R11]], $[[R7]]
 ; ALL:        and     $[[R13:[0-9]+]], $[[R10]], $[[R8]]
 ; ALL:        or      $[[R14:[0-9]+]], $[[R13]], $[[R12]]
 ; ALL:        sc      $[[R14]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R14]], $[[BB0]]
 ; MICROMIPS:  beqzc   $[[R14]], $[[BB0]]
+; MIPSR6:     beqzc   $[[R14]], $[[BB0]]
 
 ; ALL:        and     $[[R15:[0-9]+]], $[[R10]], $[[R7]]
 ; ALL:        srlv    $[[R16:[0-9]+]], $[[R15]], $[[R5]]
 
 ; NO-SEB-SEH: sll     $[[R17:[0-9]+]], $[[R16]], 24
 ; NO-SEB-SEH: sra     $2, $[[R17]], 24
 
 ; HAS-SEB-SEH:seb     $2, $[[R16]]

 ; ALL:           and     $[[R18:[0-9]+]], $[[R10]], $[[R9]]
 ; ALL:           nor     $[[R11:[0-9]+]], $zero, $[[R18]]
 ; ALL:           and     $[[R12:[0-9]+]], $[[R11]], $[[R7]]
 ; ALL:           and     $[[R13:[0-9]+]], $[[R10]], $[[R8]]
 ; ALL:           or      $[[R14:[0-9]+]], $[[R13]], $[[R12]]
 ; ALL:           sc      $[[R14]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R14]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R14]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R14]], $[[BB0]]
 
 ; ALL:           and     $[[R15:[0-9]+]], $[[R10]], $[[R7]]
 ; ALL:           srlv    $[[R16:[0-9]+]], $[[R15]], $[[R5]]
 
 ; NO-SEB-SEH:    sll     $[[R17:[0-9]+]], $[[R16]], 24
 ; NO-SEB-SEH:    sra     $2, $[[R17]], 24
 
 ; HAS-SEB-SEH:   seb     $2, $[[R16]]

 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $[[R10:[0-9]+]], 0($[[R2]])
 ; ALL:           and     $[[R18:[0-9]+]], $[[R9]], $[[R7]]
 ; ALL:           and     $[[R13:[0-9]+]], $[[R10]], $[[R8]]
 ; ALL:           or      $[[R14:[0-9]+]], $[[R13]], $[[R18]]
 ; ALL:           sc      $[[R14]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R14]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R14]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R14]], $[[BB0]]
 
 ; ALL:           and     $[[R15:[0-9]+]], $[[R10]], $[[R7]]
 ; ALL:           srlv    $[[R16:[0-9]+]], $[[R15]], $[[R5]]
 
 ; NO-SEB-SEH:    sll     $[[R17:[0-9]+]], $[[R16]], 24
 ; NO-SEB-SEH:    sra     $2, $[[R17]], 24
 
 ; HAS-SEB-SEH:   seb     $2, $[[R16]]

 ; ALL:           andi    $[[R9:[0-9]+]], $4, 255
 ; ALL:           sllv    $[[R10:[0-9]+]], $[[R9]], $[[R5]]
 ; ALL:           andi    $[[R11:[0-9]+]], $5, 255
 ; ALL:           sllv    $[[R12:[0-9]+]], $[[R11]], $[[R5]]
 
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $[[R13:[0-9]+]], 0($[[R2]])
 ; ALL:           and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; ALL:           bne     $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
+; NOT-MICROMIPS: bne     $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
+; MICROMIPS:     bne     $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
+; MIPSR6:        bnec    $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
 
 ; ALL:           and     $[[R15:[0-9]+]], $[[R13]], $[[R8]]
 ; ALL:           or      $[[R16:[0-9]+]], $[[R15]], $[[R12]]
 ; ALL:           sc      $[[R16]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R16]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R16]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R16]], $[[BB0]]
 
 ; ALL:       $[[BB1]]:
 ; ALL:           srlv    $[[R17:[0-9]+]], $[[R14]], $[[R5]]
 
 ; NO-SEB-SEH:    sll     $[[R18:[0-9]+]], $[[R17]], 24
 ; NO-SEB-SEH:    sra     $2, $[[R18]], 24
 
 ; HAS-SEB-SEH:   seb     $2, $[[R17]]

 ; ALL:           andi    $[[R9:[0-9]+]], $5, 255
 ; ALL:           sllv    $[[R10:[0-9]+]], $[[R9]], $[[R5]]
 ; ALL:           andi    $[[R11:[0-9]+]], $6, 255
 ; ALL:           sllv    $[[R12:[0-9]+]], $[[R11]], $[[R5]]
 
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $[[R13:[0-9]+]], 0($[[R2]])
 ; ALL:           and     $[[R14:[0-9]+]], $[[R13]], $[[R7]]
-; ALL:           bne     $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
+; NOT-MICROMIPS: bne     $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
+; MICROMIPS:     bne     $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
+; MIPSR6:        bnec    $[[R14]], $[[R10]], $[[BB1:[A-Z_0-9]+]]
 
 ; ALL:           and     $[[R15:[0-9]+]], $[[R13]], $[[R8]]
 ; ALL:           or      $[[R16:[0-9]+]], $[[R15]], $[[R12]]
 ; ALL:           sc      $[[R16]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R16]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R16]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R16]], $[[BB0]]
 
 ; ALL:       $[[BB1]]:
 ; ALL:           srlv    $[[R17:[0-9]+]], $[[R14]], $[[R5]]
 
 ; NO-SEB-SEH:    sll     $[[R18:[0-9]+]], $[[R17]], 24
 ; NO-SEB-SEH:    sra     $[[R19:[0-9]+]], $[[R18]], 24
 
 ; HAS-SEB-SEH:   seb     $[[R19:[0-9]+]], $[[R17]]

 ; ALL:           ll      $[[R10:[0-9]+]], 0($[[R2]])
 ; ALL:           addu    $[[R11:[0-9]+]], $[[R10]], $[[R9]]
 ; ALL:           and     $[[R12:[0-9]+]], $[[R11]], $[[R7]]
 ; ALL:           and     $[[R13:[0-9]+]], $[[R10]], $[[R8]]
 ; ALL:           or      $[[R14:[0-9]+]], $[[R13]], $[[R12]]
 ; ALL:           sc      $[[R14]], 0($[[R2]])
 ; NOT-MICROMIPS: beqz    $[[R14]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R14]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R14]], $[[BB0]]
 
 ; ALL:           and     $[[R15:[0-9]+]], $[[R10]], $[[R7]]
 ; ALL:           srlv    $[[R16:[0-9]+]], $[[R15]], $[[R5]]
 
 ; NO-SEB-SEH:    sll     $[[R17:[0-9]+]], $[[R16]], 16
 ; NO-SEB-SEH:    sra     $2, $[[R17]], 16
 
 ; MIPS32R2:      seh     $2, $[[R16]]

 
 ; ALL:           addiu   $[[PTR:[0-9]+]], $[[R0]], 1024
 ; ALL:       $[[BB0:[A-Z_0-9]+]]:
 ; ALL:           ll      $[[R1:[0-9]+]], 0($[[PTR]])
 ; ALL:           addu    $[[R2:[0-9]+]], $[[R1]], $4
 ; ALL:           sc      $[[R2]], 0($[[PTR]])
 ; NOT-MICROMIPS: beqz    $[[R2]], $[[BB0]]
 ; MICROMIPS:     beqzc   $[[R2]], $[[BB0]]
+; MIPSR6:        beqzc   $[[R2]], $[[BB0]]
 }

test/CodeGen/Mips/compactbranches/compact-branches.ll

+; RUN: llc -march=mipsel -mcpu=mips32r6 -relocation-model=static < %s | FileCheck %s
+
+; Function Attrs: nounwind
+define void @l()  {
+entry:
+  %call = tail call i32 @k()
+  %call1 = tail call i32 @j()
+  %cmp = icmp eq i32 %call, %call1
+; CHECK: bnec
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext -2)
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+declare i32 @k()
+
+declare i32 @j()
+
+declare void @f(i32 signext) 
+
+; Function Attrs: define void @l2()  {
+define void @l2()  {
+entry:
+  %call = tail call i32 @k()
+  %call1 = tail call i32 @i()
+  %cmp = icmp eq i32 %call, %call1
+; CHECK beqc
+  br i1 %cmp, label %if.end, label %if.then
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext -1)
+  br label %if.end
+
+if.end:                                           ; preds = %entry, %if.then
+  ret void
+}
+
+declare i32 @i()
+
+; Function Attrs: nounwind
+define void @l3()  {
+entry:
+  %call = tail call i32 @k()
+  %cmp = icmp slt i32 %call, 0
+; CHECK : bgez
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext 0)
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+; Function Attrs: nounwind
+define void @l4()  {
+entry:
+  %call = tail call i32 @k()
+  %cmp = icmp slt i32 %call, 1
+; CHECK: bgtzc
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext 1)
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+; Function Attrs: nounwind
+define void @l5()  {
+entry:
+  %call = tail call i32 @k()
+  %cmp = icmp sgt i32 %call, 0
+; CHECK: blezc
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext 2) 
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+; Function Attrs: nounwind
+define void @l6()  {
+entry:
+  %call = tail call i32 @k()
+  %cmp = icmp sgt i32 %call, -1
+; CHECK: bltzc
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext 3)
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+; Function Attrs: nounwind
+define void @l7()  {
+entry:
+  %call = tail call i32 @k()
+  %cmp = icmp eq i32 %call, 0
+; CHECK: bnezc
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext 4)
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+; Function Attrs: nounwind
+define void @l8()  {
+entry:
+  %call = tail call i32 @k()
+  %cmp = icmp eq i32 %call, 0
+; CHECK: beqzc
+  br i1 %cmp, label %if.end, label %if.then
+
+if.then:                                          ; preds = %entry:
+; CHECK: nop
+; CHECK: jal
+  tail call void @f(i32 signext 5)
+  br label %if.end
+
+if.end:                                           ; preds = %entry, %if.then
+  ret void
+}

test/CodeGen/Mips/fcmp.ll

 ; 32-C-DAG:      bc1t
 
 ; 32-CMP-DAG:    add.s    $[[T0:f[0-9]+]], $f14, $f12
 ; 32-CMP-DAG:    lwc1     $[[T1:f[0-9]+]], %lo($CPI32_0)(
 ; 32-CMP-DAG:    cmp.le.s $[[T2:f[0-9]+]], $[[T0]], $[[T1]]
 ; 32-CMP-DAG:    mfc1     $[[T3:[0-9]+]], $[[T2]]
 ; FIXME: This instruction is redundant.
 ; 32-CMP-DAG:    andi     $[[T4:[0-9]+]], $[[T3]], 1
-; 32-CMP-DAG:    bnez     $[[T4]],
+; 32-CMP-DAG:    bnezc    $[[T4]],
 
 ; 64-C-DAG:      add.s    $[[T0:f[0-9]+]], $f13, $f12
 ; 64-C-DAG:      lwc1     $[[T1:f[0-9]+]], %got_ofst($CPI32_0)(
 ; 64-C-DAG:      c.ole.s  $[[T0]], $[[T1]]
 ; 64-C-DAG:      bc1t
 
 ; 64-CMP-DAG:    add.s    $[[T0:f[0-9]+]], $f13, $f12
 ; 64-CMP-DAG:    lwc1     $[[T1:f[0-9]+]], %got_ofst($CPI32_0)(
 ; 64-CMP-DAG:    cmp.le.s $[[T2:f[0-9]+]], $[[T0]], $[[T1]]
 ; 64-CMP-DAG:    mfc1     $[[T3:[0-9]+]], $[[T2]]
 ; FIXME: This instruction is redundant.
 ; 64-CMP-DAG:    andi     $[[T4:[0-9]+]], $[[T3]], 1
-; 64-CMP-DAG:    bnez     $[[T4]],
+; 64-CMP-DAG:    bnezc    $[[T4]],
 
   %add = fadd fast float %at, %angle
   %cmp = fcmp ogt float %add, 1.000000e+00
   br i1 %cmp, label %if.then, label %if.end
 
 if.then:
   %sub = fadd fast float %add, -1.000000e+00
   br label %if.end

 ; 32-C-DAG:      bc1t
 
 ; 32-CMP-DAG:    add.d    $[[T0:f[0-9]+]], $f14, $f12
 ; 32-CMP-DAG:    ldc1     $[[T1:f[0-9]+]], %lo($CPI33_0)(
 ; 32-CMP-DAG:    cmp.le.d $[[T2:f[0-9]+]], $[[T0]], $[[T1]]
 ; 32-CMP-DAG:    mfc1     $[[T3:[0-9]+]], $[[T2]]
 ; FIXME: This instruction is redundant.
 ; 32-CMP-DAG:    andi     $[[T4:[0-9]+]], $[[T3]], 1
-; 32-CMP-DAG:    bnez     $[[T4]],
+; 32-CMP-DAG:    bnezc    $[[T4]],
 
 ; 64-C-DAG:      add.d    $[[T0:f[0-9]+]], $f13, $f12
 ; 64-C-DAG:      ldc1     $[[T1:f[0-9]+]], %got_ofst($CPI33_0)(
 ; 64-C-DAG:      c.ole.d  $[[T0]], $[[T1]]
 ; 64-C-DAG:      bc1t
 
 ; 64-CMP-DAG:    add.d    $[[T0:f[0-9]+]], $f13, $f12
 ; 64-CMP-DAG:    ldc1     $[[T1:f[0-9]+]], %got_ofst($CPI33_0)(
 ; 64-CMP-DAG:    cmp.le.d $[[T2:f[0-9]+]], $[[T0]], $[[T1]]
 ; 64-CMP-DAG:    mfc1     $[[T3:[0-9]+]], $[[T2]]
 ; FIXME: This instruction is redundant.
 ; 64-CMP-DAG:    andi     $[[T4:[0-9]+]], $[[T3]], 1
-; 64-CMP-DAG:    bnez     $[[T4]],
+; 64-CMP-DAG:    bnezc    $[[T4]],
 
   %add = fadd fast double %at, %angle
   %cmp = fcmp ogt double %add, 1.000000e+00
   br i1 %cmp, label %if.then, label %if.end
 
 if.then:
   %sub = fadd fast double %add, -1.000000e+00
   br label %if.end
 
 if.end:
   %theta.0 = phi double [ %sub, %if.then ], [ %add, %entry ]
   ret double %theta.0
 }
 
 attributes #0 = { nounwind readnone "no-nans-fp-math"="true" }

test/CodeGen/Mips/fpbr.ll

 ; 64-FCC:        c.eq.s $f12, $f13
 ; FCC:           bc1f   $BB0_2
 
 ; 32-GPR:        cmp.eq.s $[[FGRCC:f[0-9]+]], $f12, $f14
 ; 64-GPR:        cmp.eq.s $[[FGRCC:f[0-9]+]], $f12, $f13
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC:f[0-9]+]]
 ; FIXME: We ought to be able to transform not+bnez -> beqz
 ; GPR:           not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           bnez     $[[GPRCC]], $BB0_2
+; 32-GPR:        bnez     $[[GPRCC]], $BB0_2
+; 64-GPR:        bnezc    $[[GPRCC]], $BB0_2
 
   %cmp = fcmp oeq float %f2, %f3
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:                                          ; preds = %entry
   tail call void (...) @g0() nounwind
   br label %if.end
 

 ; 32-FCC:        c.olt.s $f12, $f14
 ; 64-FCC:        c.olt.s $f12, $f13
 ; FCC:           bc1f    $BB1_2
 
 ; 32-GPR:        cmp.ule.s $[[FGRCC:f[0-9]+]], $f14, $f12
 ; 64-GPR:        cmp.ule.s $[[FGRCC:f[0-9]+]], $f13, $f12
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC:f[0-9]+]]
 ; GPR-NOT:       not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           bnez     $[[GPRCC]], $BB1_2
+; 32-GPR:        bnez     $[[GPRCC]], $BB1_2
+; 64-GPR:        bnezc    $[[GPRCC]], $BB1_2
 
   %cmp = fcmp olt float %f2, %f3
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:                                          ; preds = %entry
   tail call void (...) @g0() nounwind
   br label %if.end
 

 ; 32-FCC:        c.ole.s $f12, $f14
 ; 64-FCC:        c.ole.s $f12, $f13
 ; FCC:           bc1t    $BB2_2
 
 ; 32-GPR:        cmp.ult.s $[[FGRCC:f[0-9]+]], $f14, $f12
 ; 64-GPR:        cmp.ult.s $[[FGRCC:f[0-9]+]], $f13, $f12
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC:f[0-9]+]]
 ; GPR-NOT:       not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           beqz     $[[GPRCC]], $BB2_2
+; 32-GPR:        beqz     $[[GPRCC]], $BB2_2
+; 64-GPR:        beqzc    $[[GPRCC]], $BB2_2
 
   %cmp = fcmp ugt float %f2, %f3
   br i1 %cmp, label %if.else, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void (...) @g0() nounwind
   br label %if.end
 

 ; 64-FCC:        c.eq.d $f12, $f13
 ; FCC:           bc1f $BB3_2
 
 ; 32-GPR:        cmp.eq.d $[[FGRCC:f[0-9]+]], $f12, $f14
 ; 64-GPR:        cmp.eq.d $[[FGRCC:f[0-9]+]], $f12, $f13
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC:f[0-9]+]]
 ; FIXME: We ought to be able to transform not+bnez -> beqz
 ; GPR:           not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           bnez     $[[GPRCC]], $BB3_2
+; 32-GPR:        bnez     $[[GPRCC]], $BB3_2
+; 64-GPR:        bnezc    $[[GPRCC]], $BB3_2
 
   %cmp = fcmp oeq double %f2, %f3
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:                                          ; preds = %entry
   tail call void (...) @g0() nounwind
   br label %if.end
 

 ; 32-FCC:        c.olt.d $f12, $f14
 ; 64-FCC:        c.olt.d $f12, $f13
 ; FCC:           bc1f $BB4_2
 
 ; 32-GPR:        cmp.ule.d $[[FGRCC:f[0-9]+]], $f14, $f12
 ; 64-GPR:        cmp.ule.d $[[FGRCC:f[0-9]+]], $f13, $f12
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC:f[0-9]+]]
 ; GPR-NOT:       not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           bnez     $[[GPRCC]], $BB4_2
+; 32-GPR:        bnez     $[[GPRCC]], $BB4_2
+; 64-GPR:        bnezc    $[[GPRCC]], $BB4_2
 
   %cmp = fcmp olt double %f2, %f3
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:                                          ; preds = %entry
   tail call void (...) @g0() nounwind
   br label %if.end
 

 ; 32-FCC:        c.ole.d $f12, $f14
 ; 64-FCC:        c.ole.d $f12, $f13
 ; FCC:           bc1t $BB5_2
 
 ; 32-GPR:        cmp.ult.d $[[FGRCC:f[0-9]+]], $f14, $f12
 ; 64-GPR:        cmp.ult.d $[[FGRCC:f[0-9]+]], $f13, $f12
 ; GPR:           mfc1     $[[GPRCC:[0-9]+]], $[[FGRCC:f[0-9]+]]
 ; GPR-NOT:       not      $[[GPRCC]], $[[GPRCC]]
-; GPR:           beqz     $[[GPRCC]], $BB5_2
+; 32-GPR:        beqz     $[[GPRCC]], $BB5_2
+; 64-GPR:        beqzc    $[[GPRCC]], $BB5_2
 
   %cmp = fcmp ugt double %f2, %f3
   br i1 %cmp, label %if.else, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void (...) @g0() nounwind
   br label %if.end
 
 if.else:                                          ; preds = %entry
   tail call void (...) @g1() nounwind
   br label %if.end
 
 if.end:                                           ; preds = %if.else, %if.then
   ret void
 }