CodeGen: Disable isCopyInstrImpl if there are implicit operands

[arsenm]

Summary

This is a conservative workaround for broken liveness tracking of
SUBREG_TO_REG to speculatively fix all targets. The current reported
failures are on X86 only, but this issue should appear for all targets
that use SUBREG_TO_REG. The next minimally correct refinement would be
to disallow only implicit defs.

The coalescer now introduces implicit-defs of the super register to
track the dependency on other subregisters. If we see such an implicit
operand, we cannot simply treat the subregister def as the result
operand in case downstream users depend on the implicitly defined
parts. Really target implementations should be considering the
implicit defs and trying to interpret them appropriately (maybe with
some generic helpers). The full implicit def could possibly be
reported as the move result, rather than the subregister def but that
requires additional work.

Hopefully fixes #64060 as well.

This needs to be applied to the release branch.

[vitalybuka]

Just tried, this one does not solve the issue. Or it probably introduce a different miscompile. UBSan report is gone, but the test fails text expectations. With stable ( before D150388 ) compiler the test consistently green.

D156164 also makes the test consistently green.

[arsenm]

In D156346#4536935, @vitalybuka wrote:

Just tried, this one does not solve the issue. Or it probably introduce a different miscompile. UBSan report is gone, but the test fails text expectations. With stable ( before D150388 ) compiler the test consistently green.

D156164 also makes the test consistently green.

To clarify did you only apply this patch? You need the whole stack. This one in isolation won’t fix it

[vitalybuka]

In D156346#4537085, @arsenm wrote:

In D156346#4536935, @vitalybuka wrote:

Just tried, this one does not solve the issue. Or it probably introduce a different miscompile. UBSan report is gone, but the test fails text expectations. With stable ( before D150388 ) compiler the test consistently green.

D156164 also makes the test consistently green.

To clarify did you only apply this patch? You need the whole stack. This one in isolation won’t fix it

yes, only the last one
arc patch does not apply entire stack, probably patches need to be rebased/uploaded

my 2c: if it's entire stack then revert D156381 is more appropriate solution

[arsenm]

In D156346#4537108, @vitalybuka wrote:

In D156346#4537085, @arsenm wrote:

In D156346#4536935, @vitalybuka wrote:

Just tried, this one does not solve the issue. Or it probably introduce a different miscompile. UBSan report is gone, but the test fails text expectations. With stable ( before D150388 ) compiler the test consistently green.

D156164 also makes the test consistently green.

To clarify did you only apply this patch? You need the whole stack. This one in isolation won’t fix it

yes, only the last one
arc patch does not apply entire stack, probably patches need to be rebased/uploaded

my 2c: if it's entire stack then revert D156381 is more appropriate solution

D156164 also works as the quick fix, I'm less horrified by it now that I fully understand what's happening. The full stack is a more refined variant

[vitalybuka]

In D156346#4537128, @arsenm wrote:

In D156346#4537108, @vitalybuka wrote:

In D156346#4537085, @arsenm wrote:

In D156346#4536935, @vitalybuka wrote:

Just tried, this one does not solve the issue. Or it probably introduce a different miscompile. UBSan report is gone, but the test fails text expectations. With stable ( before D150388 ) compiler the test consistently green.

D156164 also makes the test consistently green.

To clarify did you only apply this patch? You need the whole stack. This one in isolation won’t fix it

yes, only the last one
arc patch does not apply entire stack, probably patches need to be rebased/uploaded

my 2c: if it's entire stack then revert D156381 is more appropriate solution

D156164 also works as the quick fix, I'm less horrified by it now that I fully understand what's happening. The full stack is a more refined variant

I don't know details of this code, but from my experience:

1. revert
2. wait a little
3. land fixes
4. wait a little
5. revert the revert (better land patches one by one)

will be easier bisect/investigate if fixes introduce other issues later

[alexfh]

Vitaly is right. Let's revert the original commit with the dependent patches and work on a solution after that. I'm happy to test any suggested fixes, but we should first return the tree to a good state.

[qcolombet]

llvm/include/llvm/CodeGen/TargetInstrInfo.h

1044

Maybe that should be part of the verifier?

1053

Like I said in https://reviews.llvm.org/D156345, there was already a distinction between pure copies (isCopy) and copies with additional stuff that may require careful handling (isCopyLike).

Following this distinction, I feel that either:

• this is the right fix, or
• we need to make all the places that deals with isCopyInstr robust w.r.t. isCopyLike semantic, which may not be worth it to pull

[arsenm]

llvm/include/llvm/CodeGen/TargetInstrInfo.h

1044

Hypothetically you can still end up with the same implicit-def case as the mov, if you were to coalesce a copy into SUBREG_TO_REG. I believe all the instances which use SUBREG_TO_REG are fed by non-copy sources, so not sure if this happens in practice. The verifier probably should verify there are no implicit-defs at least

1053

worth it to pull

pull what?

[qcolombet]

llvm/include/llvm/CodeGen/TargetInstrInfo.h

1044

Hypothetically you can still end up with the same implicit-def case as the mov

True.

The scary thing to me is if regular copies have implicit defs, it would be good to distinguish if it is to make the liveness accurate or if it is something that went south (e.g., a target specific operation that got merged with a regular generic copy, in that case, codegen may be broken.)

1053

pull what?

Pull off :P.

I mean it may not be worth spending a lot of effort to support non pure copy everywhere.
I.e., Heroically try to implement the full support everywhere for a negligible gain.

[arsenm]

llvm/include/llvm/CodeGen/TargetInstrInfo.h

1053

I kind of think we should just get rid of SUBREG_TO_REG entirely. If you wanted to depend on those bits, you should have had to define them in the first place. However doing that is going to be a lot of work. I still haven't found anywhere making use of the "assert 0" property, if you really needed that it would look like AssertZext (which I would hope would be all taken care of before selection is complete)

[qcolombet]

llvm/include/llvm/CodeGen/TargetInstrInfo.h

1053

+1
Also AFAICT, the lowering in ExpandPostRAPseudo is also broken with respect to preserving the bits when the source and target registers are not the same (super) register.

[arsenm]

bc7d88faf1a595ab59952a2054418cdd0d9eeee8

[smeenai]

Were the changes to arm-pcsections.ll in https://reviews.llvm.org/rGbc7d88faf1a595ab59952a2054418cdd0d9eeee8#change-knDnIXP6oBYw intended? We're seeing basically every single one of those fail when running the test locally. I'm also seeing bots failing, e.g. https://lab.llvm.org/buildbot/#/builders/119/builds/15375

llvm/include/llvm/CodeGen/TargetInstrInfo.h

 
 public:
   /// If the specific machine instruction is a instruction that moves/copies
   /// value from one register to another register return destination and source
   /// registers as machine operands.
   /// For COPY-instruction the method naturally returns destination and source
   /// registers as machine operands, for all other instructions the method calls
   /// target-dependent implementation.
-  std::optional<DestSourcePair> isCopyInstr(const MachineInstr &MI) const {
+  std::optional<DestSourcePair>
+  isCopyInstr(const MachineInstr &MI,
+              bool ForbidImplicitOperands = true) const {
     if (MI.isCopy()) {
+      // TODO: Should validate implicit operands here?
       return DestSourcePair{MI.getOperand(0), MI.getOperand(1)};
     }
+
+    // TODO: This is a conservative hack to ensure correctness when extra
+    // operands are added for special liveness tracking, while also not changing
+    // debug info. In particular SUBREG_TO_REG may introduce an implicit-def of
+    // a super register after coalescing. This may manifest as a copy-like
+    // instruction with an undef subregister def, and a full register
+    // implicit-def appended to the operand list.
+
+    // Really, implementations of this should be considering extra implicit
+    // operands. A more sophisticated implementation would recognize an
+    // implicit-def of the full register, and report that as the
+    // destination. This should be removed when all targets are validated for
+    // correct SUBREG_TO_REG liveness handling.
+    if (ForbidImplicitOperands && MI.getNumImplicitOperands() != 0)
+      return std::nullopt;
+
     return isCopyInstrImpl(MI);
   }
 
   bool isFullCopyInstr(const MachineInstr &MI) const {
     auto DestSrc = isCopyInstr(MI);
     if (!DestSrc)
       return false;
 

llvm/lib/CodeGen/LiveDebugValues/InstrRefBasedImpl.cpp

if (isLocationSpill(MI, MF, Reg)) {
     unsigned Size = TRI->getRegSizeInBits(Reg, *MRI);
     unsigned SpillID = MTracker->getLocID(*Loc, {Size, 0});
     DoTransfer(Reg, SpillID);
   }
   return true;
 }
 
 bool InstrRefBasedLDV::transferRegisterCopy(MachineInstr &MI) {
-  auto DestSrc = TII->isCopyInstr(MI);
+  auto DestSrc = TII->isCopyInstr(MI, false);
   if (!DestSrc)
     return false;
 
   const MachineOperand *DestRegOp = DestSrc->Destination;
   const MachineOperand *SrcRegOp = DestSrc->Source;
 
   Register SrcReg = SrcRegOp->getReg();
   Register DestReg = DestRegOp->getReg();

llvm/lib/CodeGen/LiveDebugValues/VarLocBasedImpl.cpp

void VarLocBasedLDV::removeEntryValue(const MachineInstr &MI,
   // entry value as well.
   if (MI.getDebugExpression()->getNumElements() == 0 && TransferInst) {
     // If the DBG_VALUE comes from a copy instruction that copies the entry
     // value, it means the parameter's value has not changed and we should be
     // able to use its entry value.
     // TODO: Try to keep tracking of an entry value if we encounter a propagated
     // DBG_VALUE describing the copy of the entry value. (Propagated entry value
     // does not indicate the parameter modification.)
-    auto DestSrc = TII->isCopyInstr(*TransferInst);
+    auto DestSrc = TII->isCopyInstr(*TransferInst, false);
     if (DestSrc) {
       const MachineOperand *SrcRegOp, *DestRegOp;
       SrcRegOp = DestSrc->Source;
       DestRegOp = DestSrc->Destination;
       if (Reg == DestRegOp->getReg()) {
         for (uint64_t ID : OpenRanges.getEntryValueBackupVarLocs()) {
           const VarLoc &VL = VarLocIDs[LocIndex::fromRawInteger(ID)];
           if (VL.isEntryValueCopyBackupReg(Reg) &&

 
 /// If \p MI is a register copy instruction, that copies a previously tracked
 /// value from one register to another register that is callee saved, we
 /// create new DBG_VALUE instruction  described with copy destination register.
 void VarLocBasedLDV::transferRegisterCopy(MachineInstr &MI,
                                            OpenRangesSet &OpenRanges,
                                            VarLocMap &VarLocIDs,
                                            TransferMap &Transfers) {
-  auto DestSrc = TII->isCopyInstr(MI);
+  auto DestSrc = TII->isCopyInstr(MI, false);
   if (!DestSrc)
     return;
 
   const MachineOperand *DestRegOp = DestSrc->Destination;
   const MachineOperand *SrcRegOp = DestSrc->Source;
 
   if (!DestRegOp->isDef())
     return;

llvm/lib/Target/AArch64/AArch64InstrInfo.cpp

 
 /// If the given ORR instruction is a copy, and \p DescribedReg overlaps with
 /// the destination register then, if possible, describe the value in terms of
 /// the source register.
 static std::optional<ParamLoadedValue>
 describeORRLoadedValue(const MachineInstr &MI, Register DescribedReg,
                        const TargetInstrInfo *TII,
                        const TargetRegisterInfo *TRI) {
-  auto DestSrc = TII->isCopyInstr(MI);
+  auto DestSrc = TII->isCopyInstr(MI, false);
   if (!DestSrc)
     return std::nullopt;
 
   Register DestReg = DestSrc->Destination->getReg();
   Register SrcReg = DestSrc->Source->getReg();
 
   auto Expr = DIExpression::get(MI.getMF()->getFunction().getContext(), {});
 

llvm/test/CodeGen/X86/GlobalISel/add-ext.ll

 ; CHECK-NEXT:    imulq $4, %rax, %rax
 ; CHECK-NEXT:    addq %rdi, %rax
 ; CHECK-NEXT:    leal 2(%rsi), %ecx
 ; CHECK-NEXT:    movl %ecx, %ecx
 ; CHECK-NEXT:    imulq $4, %rcx, %rcx
 ; CHECK-NEXT:    addq %rdi, %rcx
 ; CHECK-NEXT:    movl (%rcx), %ecx
 ; CHECK-NEXT:    addl (%rax), %ecx
-; CHECK-NEXT:    movl %esi, %esi
-; CHECK-NEXT:    imulq $4, %rsi, %rax
+; CHECK-NEXT:    movl %esi, %eax
+; CHECK-NEXT:    imulq $4, %rax, %rax
 ; CHECK-NEXT:    addq %rdi, %rax
 ; CHECK-NEXT:    movl %ecx, (%rax)
 ; CHECK-NEXT:    retq
 
   %add1 = add nuw i32 %i, 1
   %idx1 = zext i32 %add1 to i64
   %gep1 = getelementptr i32, ptr %a, i64 %idx1
   %load1 = load i32, ptr %gep1, align 4

llvm/test/CodeGen/X86/coalescer-breaks-subreg-to-reg-liveness-reduced.ll

 ; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
 ; CHECK-NEXT:    .cfi_def_cfa_register %rbp
 ; CHECK-NEXT:    pushq %r15
 ; CHECK-NEXT:    pushq %r14
 ; CHECK-NEXT:    pushq %r13
 ; CHECK-NEXT:    pushq %r12
 ; CHECK-NEXT:    pushq %rbx
-; CHECK-NEXT:    subq $24, %rsp
+; CHECK-NEXT:    pushq %rax
 ; CHECK-NEXT:    .cfi_offset %rbx, -56
 ; CHECK-NEXT:    .cfi_offset %r12, -48
 ; CHECK-NEXT:    .cfi_offset %r13, -40
 ; CHECK-NEXT:    .cfi_offset %r14, -32
 ; CHECK-NEXT:    .cfi_offset %r15, -24
 ; CHECK-NEXT:    movl %esi, %ebx
 ; CHECK-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-NEXT:    xorl %r15d, %r15d
-; CHECK-NEXT:    xorl %r13d, %r13d
 ; CHECK-NEXT:    xorl %r12d, %r12d
-; CHECK-NEXT:    # implicit-def: $rax
-; CHECK-NEXT:    # kill: killed $rax
+; CHECK-NEXT:    # implicit-def: $r13
 ; CHECK-NEXT:    jmp .LBB0_2
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_1: # %bb5
 ; CHECK-NEXT:    # in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    orl $1, %r12d
-; CHECK-NEXT:    movq %r14, %r13
+; CHECK-NEXT:    movq %r14, %r15
 ; CHECK-NEXT:  .LBB0_2: # %bb7
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    callq *%r15
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    callq *%rax
+; CHECK-NEXT:    movl %r13d, %r13d
 ; CHECK-NEXT:    testb $1, %bl
 ; CHECK-NEXT:    movl $0, %r14d
 ; CHECK-NEXT:    jne .LBB0_1
 ; CHECK-NEXT:  # %bb.3: # %bb17
 ; CHECK-NEXT:    # in Loop: Header=BB0_2 Depth=1
 ; CHECK-NEXT:    xorl %r14d, %r14d
-; CHECK-NEXT:    testq %r13, %r13
+; CHECK-NEXT:    testq %r15, %r15
 ; CHECK-NEXT:    sete %r14b
 ; CHECK-NEXT:    xorl %edi, %edi
-; CHECK-NEXT:    callq *%r15
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    callq *%rax
 ; CHECK-NEXT:    shlq $4, %r14
 ; CHECK-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r14 # 8-byte Folded Reload
-; CHECK-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
-; CHECK-NEXT:    movl %eax, 0
+; CHECK-NEXT:    movl %r13d, 0
 ; CHECK-NEXT:    movb $0, 4
 ; CHECK-NEXT:    jmp .LBB0_1
 bb:
   br label %bb7
 
 bb5:                                              ; preds = %bb17, %bb7
   %phi6 = phi ptr [ %getelementptr, %bb17 ], [ null, %bb7 ]
   %add = or i32 %phi9, 1

llvm/test/CodeGen/X86/coalescer-breaks-subreg-to-reg-liveness.ll

 ; CHECK-NEXT:    .cfi_offset %rbx, -56
 ; CHECK-NEXT:    .cfi_offset %r12, -48
 ; CHECK-NEXT:    .cfi_offset %r13, -40
 ; CHECK-NEXT:    .cfi_offset %r14, -32
 ; CHECK-NEXT:    .cfi_offset %r15, -24
 ; CHECK-NEXT:    movl %r8d, %r14d
 ; CHECK-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
-; CHECK-NEXT:    movq %rsi, %r12
+; CHECK-NEXT:    movq %rsi, %r13
 ; CHECK-NEXT:    movq %rdi, %r15
 ; CHECK-NEXT:    incl %r14d
 ; CHECK-NEXT:    xorl %ebx, %ebx
-; CHECK-NEXT:    # implicit-def: $rax
-; CHECK-NEXT:    # kill: killed $rax
+; CHECK-NEXT:    # implicit-def: $r12
+; CHECK-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; CHECK-NEXT:    jmp .LBB0_3
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_1: # %bb17
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    movq %r15, %r13
 ; CHECK-NEXT:    xorl %r15d, %r15d
 ; CHECK-NEXT:    testq %rbx, %rbx
 ; CHECK-NEXT:    sete %r15b
 ; CHECK-NEXT:    xorl %edi, %edi
 ; CHECK-NEXT:    callq _Znwm@PLT
 ; CHECK-NEXT:    shlq $4, %r15
 ; CHECK-NEXT:    addq {{[-0-9]+}}(%r{{[sb]}}p), %r15 # 8-byte Folded Reload
-; CHECK-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
-; CHECK-NEXT:    movq %rdx, %rcx
+; CHECK-NEXT:    movq %r12, %rcx
 ; CHECK-NEXT:    shrq $32, %rcx
 ; CHECK-NEXT:    movb %cl, 12(%rax)
-; CHECK-NEXT:    movl %edx, 8(%rax)
+; CHECK-NEXT:    movl %r12d, 8(%rax)
 ; CHECK-NEXT:    movq %r15, %rbx
 ; CHECK-NEXT:    movq %r13, %r15
+; CHECK-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r13 # 8-byte Reload
 ; CHECK-NEXT:    decl %r14d
-; CHECK-NEXT:    je .LBB0_7
+; CHECK-NEXT:    je .LBB0_8
 ; CHECK-NEXT:  .LBB0_3: # %bb7
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    callq widget@PLT
-; CHECK-NEXT:    cmpb $-5, (%r12)
-; CHECK-NEXT:    jb .LBB0_5
-; CHECK-NEXT:  # %bb.4: # %bb12
+; CHECK-NEXT:    cmpb $-5, (%r13)
+; CHECK-NEXT:    jae .LBB0_5
+; CHECK-NEXT:  # %bb.4: # in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT:    movl %r12d, %r12d
+; CHECK-NEXT:    cmpq %r15, %rbx
+; CHECK-NEXT:    jbe .LBB0_1
+; CHECK-NEXT:    jmp .LBB0_7
+; CHECK-NEXT:    .p2align 4, 0x90
+; CHECK-NEXT:  .LBB0_5: # %bb12
 ; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    movq 0, %rax
 ; CHECK-NEXT:    movq 8, %rax
-; CHECK-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
-; CHECK-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
-; CHECK-NEXT:  .LBB0_5: # %bb14
-; CHECK-NEXT:    # in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r12 # 8-byte Reload
 ; CHECK-NEXT:    cmpq %r15, %rbx
 ; CHECK-NEXT:    jbe .LBB0_1
-; CHECK-NEXT:  # %bb.6: # in Loop: Header=BB0_3 Depth=1
+; CHECK-NEXT:  .LBB0_7: # in Loop: Header=BB0_3 Depth=1
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    xorl %ebx, %ebx
 ; CHECK-NEXT:    decl %r14d
 ; CHECK-NEXT:    jne .LBB0_3
-; CHECK-NEXT:  .LBB0_7: # %bb21
+; CHECK-NEXT:  .LBB0_8: # %bb21
 ; CHECK-NEXT:    cmpb $0, 12(%rax)
-; CHECK-NEXT:    jne .LBB0_9
-; CHECK-NEXT:  # %bb.8: # %bb26
+; CHECK-NEXT:    jne .LBB0_10
+; CHECK-NEXT:  # %bb.9: # %bb26
 ; CHECK-NEXT:    addq $24, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r12
 ; CHECK-NEXT:    popq %r13
 ; CHECK-NEXT:    popq %r14
 ; CHECK-NEXT:    popq %r15
 ; CHECK-NEXT:    popq %rbp
 ; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
-; CHECK-NEXT:  .LBB0_9: # %bb25
+; CHECK-NEXT:  .LBB0_10: # %bb25
 ; CHECK-NEXT:    .cfi_def_cfa %rbp, 16
 ; CHECK-NEXT:    movq %r15, %rdi
 ; CHECK-NEXT:    callq pluto@PLT
 bb:
   br label %bb7
 
 bb5:                                              ; preds = %bb17, %bb14
   %phi = phi ptr [ %call19, %bb17 ], [ null, %bb14 ]

llvm/test/CodeGen/X86/dagcombine-cse.ll

 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    popl %ebp
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: square_high:
 ; X64:       ## %bb.0: ## %entry
-; X64-NEXT:    movl %esi, %esi
-; X64-NEXT:    movq %rsi, %rax
+; X64-NEXT:    movl %esi, %ecx
+; X64-NEXT:    movq %rcx, %rax
 ; X64-NEXT:    mulq %rdi
-; X64-NEXT:    movq %rdx, %rcx
+; X64-NEXT:    movq %rdx, %rsi
 ; X64-NEXT:    movq %rax, %r8
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    mulq %rdi
 ; X64-NEXT:    addq %r8, %rdx
-; X64-NEXT:    movq %rcx, %rax
+; X64-NEXT:    movq %rsi, %rax
 ; X64-NEXT:    adcq $0, %rax
 ; X64-NEXT:    addq %rdx, %r8
-; X64-NEXT:    adcq %rcx, %rax
-; X64-NEXT:    imulq %rsi, %rsi
-; X64-NEXT:    addq %rax, %rsi
-; X64-NEXT:    shrdq $32, %rsi, %r8
-; X64-NEXT:    shrq $32, %rsi
+; X64-NEXT:    adcq %rsi, %rax
+; X64-NEXT:    imulq %rcx, %rcx
+; X64-NEXT:    addq %rax, %rcx
+; X64-NEXT:    shrdq $32, %rcx, %r8
+; X64-NEXT:    shrq $32, %rcx
 ; X64-NEXT:    movq %r8, %rax
-; X64-NEXT:    movq %rsi, %rdx
+; X64-NEXT:    movq %rcx, %rdx
 ; X64-NEXT:    retq
 entry:
   %conv = zext i96 %x to i192
   %mul = mul nuw i192 %conv, %conv
   %shr = lshr i192 %mul, 96
   %conv2 = trunc i192 %shr to i96
   ret i96 %conv2
 }

llvm/test/CodeGen/X86/fold-and-shift-x86_64.ll

entry:
   %tmp7 = getelementptr i8, ptr %X, i64 %tmp4
   %tmp9 = load i8, ptr %tmp7
   ret i8 %tmp9
 }
 
 define i8 @t3(ptr %X, i64 %i) {
 ; CHECK-LABEL: t3:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movl %esi, %esi
-; CHECK-NEXT:    movzbl (%rdi,%rsi,4), %eax
+; CHECK-NEXT:    movl %esi, %eax
+; CHECK-NEXT:    movzbl (%rdi,%rax,4), %eax
 ; CHECK-NEXT:    retq
 
 entry:
   %tmp2 = shl i64 %i, 2
   %tmp4 = and i64 %tmp2, 17179869180
   %tmp7 = getelementptr i8, ptr %X, i64 %tmp4
   %tmp9 = load i8, ptr %tmp7
   ret i8 %tmp9

llvm/test/CodeGen/X86/unfold-masked-merge-scalar-constmask-lowhigh.ll

; CHECK-BMI-NEXT:    retq
   %my = and i32 %y, -65536
   %r = or i32 %mx, %my
   ret i32 %r
 }
 
 define i64 @out64_constmask(i64 %x, i64 %y) {
 ; CHECK-NOBMI-LABEL: out64_constmask:
 ; CHECK-NOBMI:       # %bb.0:
-; CHECK-NOBMI-NEXT:    movl %edi, %edi
+; CHECK-NOBMI-NEXT:    movl %edi, %ecx
 ; CHECK-NOBMI-NEXT:    movabsq $-4294967296, %rax # imm = 0xFFFFFFFF00000000
 ; CHECK-NOBMI-NEXT:    andq %rsi, %rax
-; CHECK-NOBMI-NEXT:    orq %rdi, %rax
+; CHECK-NOBMI-NEXT:    orq %rcx, %rax
 ; CHECK-NOBMI-NEXT:    retq
 ;
 ; CHECK-BMI-LABEL: out64_constmask:
 ; CHECK-BMI:       # %bb.0:
-; CHECK-BMI-NEXT:    movl %edi, %edi
+; CHECK-BMI-NEXT:    movl %edi, %ecx
 ; CHECK-BMI-NEXT:    movabsq $-4294967296, %rax # imm = 0xFFFFFFFF00000000
 ; CHECK-BMI-NEXT:    andq %rsi, %rax
-; CHECK-BMI-NEXT:    orq %rdi, %rax
+; CHECK-BMI-NEXT:    orq %rcx, %rax
 ; CHECK-BMI-NEXT:    retq
   %mx = and i64 %x, 4294967295
   %my = and i64 %y, -4294967296
   %r = or i64 %mx, %my
   ret i64 %r
 }
 
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;