[X86] Reduce Store Forward Block issues in HW

[lsaba]

Summary

If a load follows a store and reloads data that the store has written to memory, Intel microarchitectures can in many cases forward the data directly from the store to the load, This "store forwarding" saves cycles by enabling the load to directly obtain the data instead of accessing the data from cache or memory.
A "store forward block" occurs in cases that a store cannot be forwarded to the load. The most typical case of store forward block on Intel Core microarchiticutre that a small store cannot be forwarded to a large load.
The estimated penalty for a store forward block is ~13 cycles.

This pass tries to recognize and handle cases where "store forward block" is created by the compiler when lowering memcpy calls to a sequence
of a load and a store.

The pass currently only handles cases where memcpy is lowered to XMM/YMM registers, it tries to break the memcpy into smaller copies.
breaking the memcpy should be possible since there is no atomicity guarantee for loads and stores to XMM/YMM.

[craig.topper]

Few style comments. I didn't look at the algorithm closely yet.

lib/Target/X86/X86FixupSFB.cpp

120

What about the EVEX versions of these instructions?

221

Remove the parentheses here.

244

Use a "static const unsigned " variable instead of a #define.

458

Capitalize variable names.

[lsaba]

Addressed Craig's comments

lib/Target/X86/X86FixupSFB.cpp

120

More cases can still be added in the future

[hfinkel]

lib/Target/X86/X86FixupSFB.cpp

33

AA is too conservative.

Did you try turning it on? :-)

If you overload useAA() in X86Subtarget, then you'll actually get non-trivial AA in the backend. Given all of the recent work on adding scheduling models, we should probably experiment with this again.

That having been said, it relies on having good MMOs. If we don't, then you'll have trouble using it for this purpose until that's fixed.

225

Can you elaborate on what you're seeing? Missing MMOs are something we should fix. If the MMO size is too small, that's a bug (i.e., can cause miscompiles from invalid instruction scheduling). MMO sizes that are too big should also be fixed.

[lsaba]

lib/Target/X86/X86FixupSFB.cpp

33

Perhaps it's better to say that the solution approach suggested in the comment is more conservative, and not the AA, I will try to explain what I meant with an example:

%struct.S = type { i32, i32, i32, i32 }
; Function Attrs: nounwind uwtable 
define void @test_conditional_block(%struct.S* nocapture %s1, 
%struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3, %struct.S* 
nocapture readonly %s4) local_unnamed_addr #0 {
entry:
  %b = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 1
  store i32 %x, i32* %b, align 4
  %0 = bitcast %struct.S* %s3 to i8*
  %1 = bitcast %struct.S* %s4 to i8*
  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
  %2 = bitcast %struct.S* %s2 to i8*
  %3 = bitcast %struct.S* %s1 to i8*
  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 4, i1 false)
  ret void
}

the currently generated code:

movl %edx, 4(%rdi) // blocking store
movups (%r8), %xmm0
movups %xmm0, (%rcx)
movups (%rdi), %xmm0 // blocked load
movups %xmm0, (%rsi)

If we were to solve the problem by loading the old value, then storing like the comment suggests:

movups (%rdi), %xmm0 // blocked load
vinsert %edx, xmm0, 1 //store b to xmm0 at index 1 
movups %xmm0, (%rsi)

We would have to prove that the previous copy from s3 to s4:

movups (%r8), %xmm0
movups %xmm0, (%rcx)

does not alias or must alias (and in that case do a similar handling for it) the copy from s1 to s2, while the AA would naturally assume they may alias in this case.
On the other hand, the proposed solution in this patch (breaking the memcpy) will be correct regardless to the memory relation between s3, s4 and s1,s2.

I am not too familiar with the capabilities of the backend AA, I did not try turning it on :) correct me if i'm wrong, but relying on it and using the approach in the comment will yield a more conservative optimization than the one implemented here, at least for the trivial cases.

In general, the solution in the comment gets more complicated if the blocking store is in a predecessing block, in that case we'll need to prove safety in all predecessing blocks, or hoist the memcpy for conditional blocks

225

You are right. I saw missing MMOs, I will try to fix the ones I encountered, but it doesn't guarantee there won't be more in code I did not reach so i'm not sure it would be safe to remove those defines

[lsaba]

add MMO info

[hfinkel]

lib/Target/X86/X86FixupSFB.cpp

33

I think that this makes sense, please explain this better in the comment (i.e., that you're speculatively assuming aliasing, as the transformation is correct either way, and you often won't be able to prove a sufficient number of the interesting cases).

225

Okay, thanks (although you should use something other than #defines (constants or a function with a switch statement, etc.).

[lsaba]

addressed @hfinkel comments.
ping to reviewers.

[lsaba]

Ping.

[lsaba]

Pinging again,,

[craig.topper]

lib/Target/X86/X86FixupSFB.cpp

110

I think you need to add VMOVDQU64Z128rm/VMOVDQU32Z128rm/VMOVDQA64Z128rm/VMOVDQA32Z128rm. Those are the EVEX integer equivalents.

117

Add the EVEX integer instructions.

247

Add a comment about what's considered a relevant addressing mode?

255

Why can't these be MachineOperand references? Why are they assigned to nullptr and then immediately overwritten?

test/CodeGen/X86/fixup-sfb.ll

5

Add avx512vl command line?

[RKSimon]

All of the load/store switch statements can be moved to more compact lookup tables, similar to what we do for domain switching in X86InstrInfo.cpp?

lib/Target/X86/X86FixupSFB.cpp

18

microarchitecture

[lsaba]

addressed comments from @RKSimon and @craig.topper

[lsaba]

This comment was removed by lsaba.

[craig.topper]

lib/Target/X86/X86FixupSFB.cpp

128

Should these be DenseMaps? @RKSimon, what do you think?

Can you use a std::array or std::pair instead of std::vector for the second type since the size is fixed. Each std::vector will be a separate heap allocation.

255

I don't think this was addressed. Why aren't these references instead of pointers?

[lsaba]

addressed @craig.topper comments

[lsaba]

ping to reviewers

[craig.topper]

lib/Target/X86/X86FixupSFB.cpp

128

I don't think the DenseMap question here was answered.

233

Return a reference?

238

Return a reference here too.

426

Potentially is mispelled.

450

Can you clang-format at least this function? That 8 looks really out of place. So does the splitting of the getRegClass call.

455

Pass the map by reference. Probably const reference if you're not modifying it.

[lsaba]

addressed @craig.topper 's comments

lib/Target/X86/X86FixupSFB.cpp

128

I think DenseMap is less suitable since it doesn't have a static initializer constructor

450

This weird format is actually clang-format's output, I changed it to make it more readable

[craig.topper]

lib/Target/X86/X86FixupSFB.cpp

448

Is MRI->getTargetRegisterInfo() different than the TRI you already have?

[lsaba]

fixed TRI @craig.topper

[craig.topper]

LGTM

[lsaba]

committed in https://reviews.llvm.org/rL324835

[chandlerc]

Sorry for showing up late, but I was looking at this code because it turns out it is miscompiling code for us. We're working on a test case, but there are really a number of basic LLVM coding convention problems here.

Craig, I hate to say this, but I don't think this should have gotten approved in its current form. It violates several clear guidelines in the LLVM coding standards, and pretty frequently. Maybe try to help folks ramping up on LLVM get their patches up to a higher code quality bar before approving?

lib/Target/X86/X86FixupSFB.cpp

2

I find using *only* the acronym "SFB" everywhere really confusing. It makes it hard to discover things.

Also, the word Fixup doesn't really communicate much. Maybe X86AvoidStoreForwardingBlocks?

105

Why can't these predicates be generated by tablegen from the instruction definitions. I really don't like have lots of tables in every pass as it creates a maintenance nightmare when adding new instructions and other issues.

128

But this still isn't going to be a constant. It isn't marked constexpr. This will require building a map every time the binary starts.

Please don't create globals with complex initializers in LLVM:
http://llvm.org/docs/CodingStandards.html#do-not-use-static-constructors

Also, std::map is *really* slow...

Why not just build a function with a switch to do this mapping?

Even better, could this be generated by tablegen?

249–253

It seems really weird to have getters above for some of these and then to not use them here.

If we want a better API for extracting the memory operands, we should build one that can be used everywhere (likely in X86InstrInfo.h) rather than a couple of ad-hoc ones that we end up not using in places like this. =[

273

We don't use all caps constants typically in LLVM. And we almost always expose debug flags for this kind of constant. It should also be at the top of the file and the function comment actually attached to the function rather than the constant.

429–431

Please use early-continue or early-exit to reduce indentation when writing LLVM code:
http://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code

445

Missing vertical space here.

465

Please follow LLVM's variable naming conventions rather than inst.

Also Inst isn't a very idiomatic variable name. But worse, it is actively confusing. What is it? I would assume an instruction, perhaps a MachineInstr, but it isn't. It is a pair of int and and unsigned?? I have no idea what this variable means.

[lsaba]

@chandlerc Thanks for your comments.
still checking the option of using tablegen for some of the tables.
Any luck with the reproducer?

lib/Target/X86/X86FixupSFB.cpp

128

back to using switch functions for this.

[lsaba]

Fix the reported bug (added case to bug 36346, based on code provided by Richard Smith )
and limit the transformation to 64bit only

[lsaba]

fix comments from @niravd and @craig.topper in review https://reviews.llvm.org/D43619#inline-381377 (closing that one)

[lsaba]

ping @ reviewers, if there are no more comments i'd like to re-submit this

[niravd]

The changes from just D43619 look correct but I haven't looked deeply at this part of the patch. From the comments it looks like the patch is a bit out of date as std::map should have been converted back to switch statements.

Minor comment: I got tripped up on remembering what StoreForwardingBlock the first time I saw it. I think it Store-Forward Blocking is a more intuitive name than StoreForwardingBlock.

[lsaba]

In D41330#1049332, @niravd wrote:

The changes from just D43619 look correct but I haven't looked deeply at this part of the patch. From the comments it looks like the patch is a bit out of date as std::map should have been converted back to switch statements.

The switch statements are back, are you referring to the DisplacementSizeMap? this one needs to be sorted, that's why i'm using std::map

[craig.topper]

lib/Target/X86/X86AvoidStoreForwardingBlocks.cpp

60 ↗

(On Diff #139434)

Prefix command line option strings with "x86-"

344 ↗

(On Diff #139434)

Should this be called BlockCnt instead of BlockLimit. It itself is not a limit.

345 ↗

(On Diff #139434)

Why do we need InspectionLimit? Can't we use x86AvoidSFBInspectionLimit everywhere? Or at least make InspectionLimit const. Naively it looks like the limit might be changed in the function.

346 ↗

(On Diff #139434)

LI isn't a great name for this iterator. The name seems to have been chosen because it starts at LoadInst, but that's not meaningful once you walk away from LoadInst.

347 ↗

(On Diff #139434)

This won't visit the first instruction in the block. And it visits LoadInst. Is that intended?

368 ↗

(On Diff #139434)

PredCnt?

544 ↗

(On Diff #139434)

Use std::make_pair so you don't need to be explicit with the types.

578 ↗

(On Diff #139434)

Capitalize

638 ↗

(On Diff #139434)

Variables should be capitalized.

697 ↗

(On Diff #139434)

BlockingStoresDispSizeMap.empty()

[lsaba]

lib/Target/X86/X86AvoidStoreForwardingBlocks.cpp

345 ↗

(On Diff #139434)

I did it for readability purposes only

347 ↗

(On Diff #139434)

Nope, fixed.

[lsaba]

Updated with @craig.topper comments

[craig.topper]

lib/Target/X86/X86AvoidStoreForwardingBlocks.cpp

347 ↗

(On Diff #140236)

I don't think you want std::next here. reverse_iterator already makes it point to the instruction before LoadInst. So now I think you skip that instruction.

[lsaba]

lib/Target/X86/X86AvoidStoreForwardingBlocks.cpp

347 ↗

(On Diff #140236)

are you sure? I've double checked, without std::next the iterator starts from the load instruction

[craig.topper]

LGTM

[thegameg]

This seems to break the Machine Verifier. I filed PR37153. Do you mind taking a look please? Thanks!

[lsaba]

In D41330#1069883, @thegameg wrote:

This seems to break the Machine Verifier. I filed PR37153. Do you mind taking a look please? Thanks!

Sure, taking a look.

[lsaba]

In D41330#1069883, @thegameg wrote:

This seems to break the Machine Verifier. I filed PR37153. Do you mind taking a look please? Thanks!

Please see https://reviews.llvm.org/D45823
This should hopefully solve it

lib/Target/X86/CMakeLists.txt

set(sources
   X86CallFrameOptimization.cpp
   X86CallLowering.cpp
   X86CmovConversion.cpp
   X86DomainReassignment.cpp
   X86ExpandPseudo.cpp
   X86FastISel.cpp
   X86FixupBWInsts.cpp
   X86FixupLEAs.cpp
+  X86FixupSFB.cpp
   X86FixupSetCC.cpp
   X86FloatingPoint.cpp
   X86FrameLowering.cpp
   X86InstructionSelector.cpp
   X86ISelDAGToDAG.cpp
   X86ISelLowering.cpp
   X86InterleavedAccess.cpp
   X86InstrFMA3Info.cpp

lib/Target/X86/X86.h

 
 /// Return a pass that removes redundant LEA instructions and redundant address
 /// recalculations.
 FunctionPass *createX86OptimizeLEAs();
 
 /// Return a pass that transforms setcc + movzx pairs into xor + setcc.
 FunctionPass *createX86FixupSetCC();
 
+/// Return a pass that avoids creating store forward block issues in the hardware.
+FunctionPass *createX86FixupSFB();
+
 /// Return a pass that expands WinAlloca pseudo-instructions.
 FunctionPass *createX86WinAllocaExpander();
 
 /// Return a pass that optimizes the code-size of x86 call sequences. This is
 /// done by replacing esp-relative movs with pushes.
 FunctionPass *createX86CallFrameOptimization();
 
 /// Return an IR pass that inserts EH registration stack objects and explicit

lib/Target/X86/X86FixupSFB.cpp

+//===- X86FixupSFB.cpp - Avoid HW Store Forward Block issues -----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// If a load follows a store and reloads data that the store has written to
+// memory, Intel microarchitectures can in many cases forward the data directly
+// from the store to the load, This "store forwarding" saves cycles by enabling
+// the load to directly obtain the data instead of accessing the data from
+// cache or memory.
+// A "store forward block" occurs in cases that a store cannot be forwarded to
+// the load. The most typical case of store forward block on Intel Core
+// microarchiticutre that a small store cannot be forwarded to a large load.
+// The estimated penalty for a store forward block is ~13 cycles.
+//
+// This pass tries to recognize and handle cases where "store forward block"
+// is created by the compiler when lowering memcpy calls to a sequence
+// of a load and a store.
+//
+// The pass currently only handles cases where memcpy is lowered to
+// XMM/YMM registers, it tries to break the memcpy into smaller copies.
+// breaking the memcpy should be possible since there is no atomicity
+// guarantee for loads and stores to XMM/YMM.
+//
+// It could be better for performance to solve the problem by loading
+// to XMM/YMM then inserting the partial store before storing back from XMM/YMM
+// to memory, but this requires we know all stores that alias
+// the loaded location in memory, AA is too conservative.
+
+//===----------------------------------------------------------------------===//
+
+#include "X86InstrInfo.h"
+#include "X86Subtarget.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstr.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/IR/DebugInfoMetadata.h"
+#include "llvm/IR/DebugLoc.h"
+#include "llvm/IR/Function.h"
+#include "llvm/MC/MCInstrDesc.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "x86-fixup-SFB"
+
+static cl::opt<bool> DisableX86FixupSFB("disable-fixup-SFB", cl::Hidden,
+                                        cl::desc("X86: Disable SFB fixup."),
+                                        cl::init(false));
+namespace {
+
+class FixupSFBPass : public MachineFunctionPass {
+public:
+  FixupSFBPass() : MachineFunctionPass(ID) {}
+
+  StringRef getPassName() const override {
+    return "X86 Fixup Store Forward Block";
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+private:
+  MachineRegisterInfo *MRI;
+  const X86InstrInfo *TII;
+  const X86RegisterInfo *TRI;
+  SmallVector<std::pair<MachineInstr *, MachineInstr *>, 2> BlockedLoadsStores;
+  SmallVector<MachineInstr *, 2> ForRemoval;
+
+  /// \brief Returns couples of Load then Store to memory which look
+  ///  like a memcpy.
+  void findPotentialylBlockedCopies(MachineFunction &MF);
+  /// \brief Break the memcpy's load and store into smaller copies
+  /// such that each memory load that was blocked by a smaller store
+  /// would now be copied separately.
+  void breakBlockedCopies(MachineInstr *LoadInst, MachineInstr *StoreInst,
+                          std::map<int64_t, unsigned> *BlockingStoresDisp);
+  /// \brief Break a copy of size Size to smaller copies.
+  void buildCopies(int Size, MachineInstr *LoadInst, MachineOperand *LoadBase,
+                   int64_t LdDispImm, MachineInstr *StoreInst,
+                   MachineOperand *StoreBase, int64_t StDispImm);
+
+  void buildCopy(MachineInstr *LoadInst, unsigned NLoadOpcode,
+                 MachineOperand *LoadBase, int64_t LoadDisp,
+                 MachineInstr *StoreInst, unsigned NStoreOpcode,
+                 MachineOperand *StoreBaseReg, int64_t StoreDisp);
+
+  unsigned getRegSizeInBytes(MachineInstr *Inst);
+  static char ID;
+};
+
+} // end anonymous namespace
+
+char FixupSFBPass::ID = 0;
+
+FunctionPass *llvm::createX86FixupSFB() { return new FixupSFBPass(); }
+
+static bool isYMMLoadOpcode(unsigned LdOpcode) {
+  return LdOpcode == X86::VMOVUPSYrm || LdOpcode == X86::VMOVAPSYrm ||
+         LdOpcode == X86::VMOVUPDYrm || LdOpcode == X86::VMOVAPDYrm ||
+         LdOpcode == X86::VMOVDQUYrm || LdOpcode == X86::VMOVDQAYrm;
+}
+
+static bool isPotentialBlockedMemCpyLd(int Opcode) {
+  return Opcode == X86::MOVUPSrm || Opcode == X86::MOVAPSrm ||
+         Opcode == X86::VMOVUPSrm || Opcode == X86::VMOVAPSrm ||
+         Opcode == X86::VMOVUPDrm || Opcode == X86::VMOVAPDrm ||
+         Opcode == X86::VMOVDQUrm || Opcode == X86::VMOVDQArm ||
+         isYMMLoadOpcode(Opcode);
+}
+
+static bool isPotentialBlockedMemCpySt(int LdOpcode, int StOpcode) {
+  switch (LdOpcode) {
+  case X86::MOVUPSrm:
+  case X86::MOVAPSrm:
+    return StOpcode == X86::MOVUPSmr || StOpcode == X86::MOVAPSmr;
+  case X86::VMOVUPSrm:
+  case X86::VMOVAPSrm:
+    return StOpcode == X86::VMOVUPSmr || StOpcode == X86::VMOVAPSmr;
+  case X86::VMOVUPDrm:
+  case X86::VMOVAPDrm:
+    return StOpcode == X86::VMOVUPDmr || StOpcode == X86::VMOVAPDmr;
+  case X86::VMOVDQUrm:
+  case X86::VMOVDQArm:
+    return StOpcode == X86::VMOVDQUmr || StOpcode == X86::VMOVDQAmr;
+  case X86::VMOVUPSYrm:
+  case X86::VMOVAPSYrm:
+    return StOpcode == X86::VMOVUPSYmr || StOpcode == X86::VMOVAPSYmr;
+  case X86::VMOVUPDYrm:
+  case X86::VMOVAPDYrm:
+    return StOpcode == X86::VMOVUPDYmr || StOpcode == X86::VMOVAPDYmr;
+  case X86::VMOVDQUYrm:
+  case X86::VMOVDQAYrm:
+    return StOpcode == X86::VMOVDQUYmr || StOpcode == X86::VMOVDQAYmr;
+  default:
+    return false;
+  }
+}
+
+static bool isPotentialBlockingStoreInst(int Opcode, int LoadOpcode) {
+  bool PBlock = false;
+  PBlock |= Opcode == X86::MOV64mr || Opcode == X86::MOV64mi32 ||
+            Opcode == X86::MOV32mr || Opcode == X86::MOV32mi ||
+            Opcode == X86::MOV16mr || Opcode == X86::MOV16mi ||
+            Opcode == X86::MOV8mr || Opcode == X86::MOV8mi;
+  if (isYMMLoadOpcode(LoadOpcode))
+    PBlock |= Opcode == X86::VMOVUPSmr || Opcode == X86::VMOVAPSmr ||
+              Opcode == X86::VMOVUPDmr || Opcode == X86::VMOVAPDmr ||
+              Opcode == X86::VMOVDQUmr || Opcode == X86::VMOVDQAmr;
+  return PBlock;
+}
+
+static int getAddrOffset(MachineInstr *MI) {
+  const MCInstrDesc &Descl = MI->getDesc();
+  int AddrOffset = X86II::getMemoryOperandNo(Descl.TSFlags);
+  assert(AddrOffset != -1 && "Expected Memory Operand");
+  AddrOffset += X86II::getOperandBias(Descl);
+  return AddrOffset;
+}
+
+static MachineOperand *getBaseOperand(MachineInstr *MI) {
+  int AddrOffset = getAddrOffset(MI);
+  return &(MI->getOperand(AddrOffset + X86::AddrBaseReg));
+}
+
+static MachineOperand *getDispOperand(MachineInstr *MI) {
+  int AddrOffset = getAddrOffset(MI);
+  return &(MI->getOperand(AddrOffset + X86::AddrDisp));
+}
+
+// TODO: Consider expanding to other addressing modes in the future
+static bool isRelevantAddressingMode(MachineInstr *MI) {
+  MachineOperand *Base = nullptr;
+  MachineOperand *Disp = nullptr;
+  MachineOperand *Scale = nullptr;
+  MachineOperand *Index = nullptr;
+  MachineOperand *Segment = nullptr;
+  int AddrOffset = getAddrOffset(MI);
+  Base = &(MI->getOperand(AddrOffset + X86::AddrBaseReg));
+  Disp = &(MI->getOperand(AddrOffset + X86::AddrDisp));
+  Scale = &(MI->getOperand(AddrOffset + X86::AddrScaleAmt));
+  Index = &(MI->getOperand(AddrOffset + X86::AddrIndexReg));
+  Segment = &(MI->getOperand(AddrOffset + X86::AddrSegmentReg));
+
+  if (!((Base->isReg() && Base->getReg() != X86::NoRegister) || Base->isFI()))
+    return false;
+  if (!Disp->isImm())
+    return false;
+  if (Scale->getImm() != 1)
+    return false;
+  if (!(Index->isReg() && Index->getReg() == X86::NoRegister))
+    return false;
+  if (!(Segment->isReg() && Segment->getReg() == X86::NoRegister))
+    return false;
+  return true;
+}
+
+// TODO: Not sure this is the best way to get these sizes
+// but relying on MachineMemOperand does not always work.
+#define MOV128SZ 16
+#define MOV64SZ 8
+#define MOV32SZ 4
+#define MOV16SZ 2
+#define MOV8SZ 1
+
+static unsigned getMemSizeInBytes(int Opcode) {
+  switch (Opcode) {
+  case X86::VMOVUPSmr:
+  case X86::VMOVAPSmr:
+  case X86::VMOVUPDmr:
+  case X86::VMOVAPDmr:
+  case X86::VMOVDQUmr:
+  case X86::VMOVDQAmr:
+    return MOV128SZ;
+  case (X86::MOV64mr):
+  case (X86::MOV64mi32):
+    return MOV64SZ;
+  case (X86::MOV32mr):
+  case (X86::MOV32mi):
+    return MOV32SZ;
+  case (X86::MOV16mr):
+  case (X86::MOV16mi):
+    return MOV16SZ;
+  case (X86::MOV8mr):
+  case (X86::MOV8mi):
+    return MOV8SZ;
+  default:
+    llvm_unreachable("Unexpected Memory Opcode");
+  }
+  return 0;
+}
+
+// Collect potentially blocking stores.
+// Limit the number of instructions backwards we want to inspect
+// since the effect of store block won't be visible if the store
+// and load instructions have enough instructions in between to
+// keep the core busy.
+#define LIMIT 20
+static SmallVector<MachineInstr *, 2>
+findPotentialBlockers(MachineInstr *LoadInst) {
+  SmallVector<MachineInstr *, 2> PotentialBlockers;
+  int BlockLimit = 0;
+  for (MachineBasicBlock::iterator LI = LoadInst,
+                                   BB = LoadInst->getParent()->begin();
+       LI != BB; --LI) {
+    BlockLimit++;
+    if (BlockLimit >= LIMIT)
+      break;
+    MachineInstr &MI = *LI;
+    if (MI.getDesc().isCall())
+      break;
+    PotentialBlockers.push_back(&MI);
+  }
+  // If we didn't get to the instructions limit try predecessing blocks.
+  // Ideally we should traverse the predecessor blocks in depth with some
+  // coloring algorithm, but for now let's just look at the first order
+  // predecessors.
+  if (BlockLimit < LIMIT) {
+    MachineBasicBlock *MBB = LoadInst->getParent();
+    int LimitLeft = LIMIT - BlockLimit;
+    for (MachineBasicBlock::pred_iterator PB = MBB->pred_begin(),
+                                          PE = MBB->pred_end();
+         PB != PE; ++PB) {
+      MachineBasicBlock *PMBB = *PB;
+      int PredLimit = 0;
+      for (MachineBasicBlock::reverse_iterator PMI = PMBB->rbegin(),
+                                               PME = PMBB->rend();
+           PMI != PME; ++PMI) {
+        PredLimit++;
+        if (PredLimit >= LimitLeft)
+          break;
+        if (PMI->getDesc().isCall())
+          break;
+        PotentialBlockers.push_back(&*PMI);
+      }
+    }
+  }
+  return PotentialBlockers;
+}
+
+void FixupSFBPass::buildCopy(MachineInstr *LoadInst, unsigned NLoadOpcode,
+                             MachineOperand *LoadBase, int64_t LoadDisp,
+                             MachineInstr *StoreInst, unsigned NStoreOpcode,
+                             MachineOperand *StoreBase, int64_t StoreDisp) {
+  MachineBasicBlock *MBB = LoadInst->getParent();
+  unsigned Reg1 = MRI->createVirtualRegister(
+      TII->getRegClass(TII->get(NLoadOpcode), 0, MRI->getTargetRegisterInfo(),
+                       *(MBB->getParent())));
+  BuildMI(*MBB, LoadInst, LoadInst->getDebugLoc(), TII->get(NLoadOpcode), Reg1)
+      .add(*LoadBase)
+      .addImm(1)
+      .addReg(X86::NoRegister)
+      .addImm(LoadDisp)
+      .addReg(X86::NoRegister);
+  DEBUG(LoadInst->getPrevNode()->dump());
+  // If the load and store to are consecutive, use the loadInst location to
+  // reduce register pressure.
+  MachineInstr *StInst = StoreInst;
+  if (StoreInst->getPrevNode() == LoadInst)
+    StInst = LoadInst;
+  BuildMI(*MBB, StInst, StInst->getDebugLoc(), TII->get(NStoreOpcode))
+      .add(*StoreBase)
+      .addImm(1)
+      .addReg(X86::NoRegister)
+      .addImm(StoreDisp)
+      .addReg(X86::NoRegister)
+      .addReg(Reg1);
+  DEBUG(StInst->getPrevNode()->dump());
+}
+
+static unsigned getXMMLoadOpcode(unsigned LoadOpcode) {
+  switch (LoadOpcode) {
+  case X86::VMOVUPSYrm:
+  case X86::VMOVAPSYrm:
+    return X86::VMOVUPSrm;
+  case X86::VMOVUPDYrm:
+  case X86::VMOVAPDYrm:
+    return X86::VMOVUPDrm;
+  case X86::VMOVDQUYrm:
+  case X86::VMOVDQAYrm:
+    return X86::VMOVDQUrm;
+  default:
+    llvm_unreachable("Unexpected Load Instruction Opcode");
+  }
+  return 0;
+}
+static unsigned getXMMStoreOpcode(unsigned StoreOpcode) {
+  switch (StoreOpcode) {
+  case X86::VMOVUPSYmr:
+  case X86::VMOVAPSYmr:
+    return X86::VMOVUPSmr;
+  case X86::VMOVUPDYmr:
+  case X86::VMOVAPDYmr:
+    return X86::VMOVUPDmr;
+  case X86::VMOVDQUYmr:
+  case X86::VMOVDQAYmr:
+    return X86::VMOVDQUmr;
+  default:
+    llvm_unreachable("Unexpected Store Instruction Opcode");
+  }
+  return 0;
+}
+
+void FixupSFBPass::buildCopies(int Size, MachineInstr *LoadInst,
+                               MachineOperand *LdBase, int64_t LdDispImm,
+                               MachineInstr *StoreInst, MachineOperand *StBase,
+                               int64_t StDispImm) {
+  int LdDisp = LdDispImm;
+  int StDisp = StDispImm;
+  while (Size > 0) {
+    if ((Size - MOV128SZ >= 0) && isYMMLoadOpcode(LoadInst->getOpcode())) {
+      Size = Size - MOV128SZ;
+      buildCopy(LoadInst, getXMMLoadOpcode(LoadInst->getOpcode()), LdBase,
+                LdDisp, StoreInst, getXMMStoreOpcode(StoreInst->getOpcode()),
+                StBase, StDisp);
+      LdDisp += MOV128SZ;
+      StDisp += MOV128SZ;
+      continue;
+    }
+    if (Size - MOV64SZ >= 0) {
+      Size = Size - MOV64SZ;
+      buildCopy(LoadInst, X86::MOV64rm, LdBase, LdDisp, StoreInst, X86::MOV64mr,
+                StBase, StDisp);
+      LdDisp += MOV64SZ;
+      StDisp += MOV64SZ;
+      continue;
+    }
+    if (Size - MOV32SZ >= 0) {
+      Size = Size - MOV32SZ;
+      buildCopy(LoadInst, X86::MOV32rm, LdBase, LdDisp, StoreInst, X86::MOV32mr,
+                StBase, StDisp);
+      LdDisp += MOV32SZ;
+      StDisp += MOV32SZ;
+      continue;
+    }
+    if (Size - MOV16SZ >= 0) {
+      Size = Size - MOV16SZ;
+      buildCopy(LoadInst, X86::MOV16rm, LdBase, LdDisp, StoreInst, X86::MOV16mr,
+                StBase, StDisp);
+      LdDisp += MOV16SZ;
+      StDisp += MOV16SZ;
+      continue;
+    }
+    if (Size - MOV8SZ >= 0) {
+      Size = Size - MOV8SZ;
+      buildCopy(LoadInst, X86::MOV8rm, LdBase, LdDisp, StoreInst, X86::MOV8mr,
+                StBase, StDisp);
+      LdDisp += MOV8SZ;
+      StDisp += MOV8SZ;
+      continue;
+    }
+  }
+  assert(Size == 0 && "Wrong size division");
+}
+
+static void updateKillStatus(MachineInstr *LoadInst, MachineInstr *StoreInst) {
+  MachineOperand *LoadBase = getBaseOperand(LoadInst);
+  MachineOperand *StoreBase = getBaseOperand(StoreInst);
+  if (LoadBase->isReg()) {
+    MachineInstr *LastLoad = LoadInst->getPrevNode();
+    // If the original load/store to xmm/ymm were consecutive
+    // then the partial copies were also created in
+    // a consecutive order to reduce register pressure
+    // and the location of the last load is before the last store.
+    if (StoreInst->getPrevNode() == LoadInst)
+      LastLoad = LoadInst->getPrevNode()->getPrevNode();
+    getBaseOperand(LastLoad)->setIsKill(LoadBase->isKill());
+  }
+  if (StoreBase->isReg()) {
+    MachineInstr *StInst = StoreInst;
+    if (StoreInst->getPrevNode() == LoadInst)
+      StInst = LoadInst;
+    getBaseOperand(StInst->getPrevNode())->setIsKill(StoreBase->isKill());
+  }
+}
+
+void FixupSFBPass::findPotentialylBlockedCopies(MachineFunction &MF) {
+  for (auto &MBB : MF)
+    for (auto &MI : MBB)
+      if (isPotentialBlockedMemCpyLd(MI.getOpcode())) {
+        int DefVR = MI.getOperand(0).getReg();
+        if (MRI->hasOneUse(DefVR))
+          for (auto UI = MRI->use_nodbg_begin(DefVR), UE = MRI->use_nodbg_end();
+               UI != UE;) {
+            MachineOperand &StoreMO = *UI++;
+            MachineInstr &StoreMI = *StoreMO.getParent();
+            if (isPotentialBlockedMemCpySt(MI.getOpcode(),
+                                           StoreMI.getOpcode()) &&
+                (StoreMI.getParent() == MI.getParent()))
+              if (isRelevantAddressingMode(&MI) &&
+                  isRelevantAddressingMode(&StoreMI))
+                BlockedLoadsStores.push_back(
+                    std::pair<MachineInstr *, MachineInstr *>(&MI, &StoreMI));
+          }
+      }
+}
+unsigned FixupSFBPass::getRegSizeInBytes(MachineInstr *LoadInst) {
+  return TRI->getRegSizeInBits(*TII->getRegClass(
+             TII->get(LoadInst->getOpcode()), 0, MRI->getTargetRegisterInfo(),
+             *LoadInst->getParent()->getParent())) /
+         8;
+}
+
+void FixupSFBPass::breakBlockedCopies(
+    MachineInstr *LoadInst, MachineInstr *StoreInst,
+    std::map<int64_t, unsigned> *BlockingStoresDisp) {
+  MachineOperand *LoadBase = getBaseOperand(LoadInst);
+  MachineOperand *StoreBase = getBaseOperand(StoreInst);
+  int64_t LdDispImm = getDispOperand(LoadInst)->getImm();
+  int64_t StDispImm = getDispOperand(StoreInst)->getImm();
+
+  int64_t ldDisp1 = LdDispImm;
+  int64_t ldDisp2 = 0;
+  int64_t stDisp1 = StDispImm;
+  int64_t stDisp2 = 0;
+  unsigned size1 = 0;
+  unsigned size2 = 0;
+  int64_t ldStDelta = StDispImm - LdDispImm;
+  for (auto inst : *BlockingStoresDisp) {
+    ldDisp2 = inst.first;
+    stDisp2 = inst.first + ldStDelta;
+    size1 = std::abs(std::abs(ldDisp2) - std::abs(ldDisp1));
+    size2 = inst.second;
+    buildCopies(size1, LoadInst, LoadBase, ldDisp1, StoreInst, StoreBase,
+                stDisp1);
+    buildCopies(size2, LoadInst, LoadBase, ldDisp2, StoreInst, StoreBase,
+                stDisp2);
+    ldDisp1 = ldDisp2 + size2;
+    stDisp1 = stDisp2 + size2;
+  }
+  unsigned size3 = (LdDispImm + getRegSizeInBytes(LoadInst)) - ldDisp1;
+  buildCopies(size3, LoadInst, LoadBase, ldDisp1, StoreInst, StoreBase,
+              stDisp1);
+}
+
+bool FixupSFBPass::runOnMachineFunction(MachineFunction &MF) {
+  bool Changed = false;
+
+  if (DisableX86FixupSFB || skipFunction(MF.getFunction()))
+    return false;
+
+  MRI = &MF.getRegInfo();
+  assert(MRI->isSSA() && "Expected MIR to be in SSA form");
+  TII = MF.getSubtarget<X86Subtarget>().getInstrInfo();
+  TRI = MF.getSubtarget<X86Subtarget>().getRegisterInfo();
+
+  DEBUG(dbgs() << "Start X86FixupSFB\n";);
+  // Look for a load then a store to XMM/YMM which look like a memcpy
+  findPotentialylBlockedCopies(MF);
+
+  for (auto LoadStoreInst : BlockedLoadsStores) {
+    MachineInstr *LoadInst = LoadStoreInst.first;
+    SmallVector<MachineInstr *, 2> PotentialBlockers =
+        findPotentialBlockers(LoadInst);
+
+    MachineOperand *LoadBase = getBaseOperand(LoadInst);
+    int64_t LdDispImm = getDispOperand(LoadInst)->getImm();
+    std::map<int64_t, unsigned> BlockingStoresDisp;
+    int LdBaseReg =
+        LoadBase->isReg() ? LoadBase->getReg() : LoadBase->getIndex();
+
+    for (auto PBInst : PotentialBlockers) {
+      if (isPotentialBlockingStoreInst(PBInst->getOpcode(),
+                                       LoadInst->getOpcode())) {
+        if (!isRelevantAddressingMode(PBInst))
+          continue;
+        MachineOperand *PBstoreBase = getBaseOperand(PBInst);
+        int64_t PBstDispImm = getDispOperand(PBInst)->getImm();
+        unsigned PBstSize = getMemSizeInBytes(PBInst->getOpcode());
+        int PBstBaseReg = PBstoreBase->isReg() ? PBstoreBase->getReg()
+                                               : PBstoreBase->getIndex();
+        // This check doesn't cover all cases, but it will suffice for now.
+        // TODO: take branch probability into consideration, if the blocking
+        // store is in an unreached block, breaking the memcopy could lose
+        // performance.
+        if (((LoadBase->isReg() && PBstoreBase->isReg()) ||
+             (LoadBase->isFI() && PBstoreBase->isFI())) &&
+            LdBaseReg == PBstBaseReg &&
+            ((PBstDispImm >= LdDispImm) &&
+             (PBstDispImm <=
+              LdDispImm + (getRegSizeInBytes(LoadInst) - PBstSize)))) {
+          if (BlockingStoresDisp.count(PBstDispImm)) {
+            if (BlockingStoresDisp[PBstDispImm] > PBstSize)
+              BlockingStoresDisp[PBstDispImm] = PBstSize;
+
+          } else
+            BlockingStoresDisp[PBstDispImm] = PBstSize;
+        }
+      }
+    }
+
+    if (BlockingStoresDisp.size() == 0)
+      continue;
+
+    // We found a store forward block, break the memcpy's load and store
+    // into smaller copies such that each smaller store that was causing
+    // a store block would now be copied separately.
+    MachineInstr *StoreInst = LoadStoreInst.second;
+    DEBUG(dbgs() << "Blocked load and store instructions: \n");
+    DEBUG(LoadInst->dump());
+    DEBUG(StoreInst->dump());
+    DEBUG(dbgs() << "Replaced with:\n");
+    breakBlockedCopies(LoadInst, StoreInst, &BlockingStoresDisp);
+    updateKillStatus(LoadInst, StoreInst);
+    ForRemoval.push_back(LoadInst);
+    ForRemoval.push_back(StoreInst);
+  }
+  for (auto removedInst : ForRemoval) {
+    removedInst->eraseFromParent();
+  }
+  ForRemoval.clear();
+  BlockedLoadsStores.clear();
+  DEBUG(dbgs() << "End X86FixupSFB\n";);
+
+  return Changed;
+}

lib/Target/X86/X86TargetMachine.cpp

 }
 
 void X86PassConfig::addPreRegAlloc() {
   if (getOptLevel() != CodeGenOpt::None) {
     addPass(&LiveRangeShrinkID);
     addPass(createX86FixupSetCC());
     addPass(createX86OptimizeLEAs());
     addPass(createX86CallFrameOptimization());
+    addPass(createX86FixupSFB());
   }
 
   addPass(createX86WinAllocaExpander());
 }
 void X86PassConfig::addMachineSSAOptimization() {
   addPass(createX86DomainReassignmentPass());
   TargetPassConfig::addMachineSSAOptimization();
 }

test/CodeGen/X86/fixup-sfb.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-linux | FileCheck %s -check-prefix=CHECK
+; RUN: llc --disable-fixup-SFB < %s -mtriple=x86_64-linux | FileCheck %s --check-prefix=DISABLED
+; RUN: llc < %s -mtriple=x86_64-linux -mcpu=core-avx2 | FileCheck %s -check-prefix=CHECK-AVX2
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+%struct.S = type { i32, i32, i32, i32 }
+
+; Function Attrs: nounwind uwtable
+define void @test_conditional_block(%struct.S* nocapture %s1, %struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3, %struct.S* nocapture readonly %s4) local_unnamed_addr #0 {
+; CHECK-LABEL: test_conditional_block:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB0_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movl %edx, 4(%rdi)
+; CHECK-NEXT:  .LBB0_2: # %if.end
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movl (%rdi), %eax
+; CHECK-NEXT:    movl %eax, (%rsi)
+; CHECK-NEXT:    movl 4(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 4(%rsi)
+; CHECK-NEXT:    movq 8(%rdi), %rax
+; CHECK-NEXT:    movq %rax, 8(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_conditional_block:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB0_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movl %edx, 4(%rdi)
+; DISABLED-NEXT:  .LBB0_2: # %if.end
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_conditional_block:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB0_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movl %edx, 4(%rdi)
+; CHECK-AVX2-NEXT:  .LBB0_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r8), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%rcx)
+; CHECK-AVX2-NEXT:    movl (%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, (%rsi)
+; CHECK-AVX2-NEXT:    movl 4(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 4(%rsi)
+; CHECK-AVX2-NEXT:    movq 8(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 8(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 1
+  store i32 %x, i32* %b, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S* %s3 to i8*
+  %1 = bitcast %struct.S* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
+  %2 = bitcast %struct.S* %s2 to i8*
+  %3 = bitcast %struct.S* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 4, i1 false)
+  ret void
+}
+
+; Function Attrs: nounwind uwtable
+define void @test_imm_store(%struct.S* nocapture %s1, %struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3) local_unnamed_addr #0 {
+; CHECK-LABEL: test_imm_store:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl $0, (%rdi)
+; CHECK-NEXT:    movl $1, (%rcx)
+; CHECK-NEXT:    movl (%rdi), %eax
+; CHECK-NEXT:    movl %eax, (%rsi)
+; CHECK-NEXT:    movq 4(%rdi), %rax
+; CHECK-NEXT:    movq %rax, 4(%rsi)
+; CHECK-NEXT:    movl 12(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 12(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_imm_store:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    movl $0, (%rdi)
+; DISABLED-NEXT:    movl $1, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_imm_store:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    movl $0, (%rdi)
+; CHECK-AVX2-NEXT:    movl $1, (%rcx)
+; CHECK-AVX2-NEXT:    movl (%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, (%rsi)
+; CHECK-AVX2-NEXT:    movq 4(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 4(%rsi)
+; CHECK-AVX2-NEXT:    movl 12(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 12(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %a = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 0
+  store i32 0, i32* %a, align 4
+  %a1 = getelementptr inbounds %struct.S, %struct.S* %s3, i64 0, i32 0
+  store i32 1, i32* %a1, align 4
+  %0 = bitcast %struct.S* %s2 to i8*
+  %1 = bitcast %struct.S* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
+  ret void
+}
+
+; Function Attrs: nounwind uwtable
+define void @test_nondirect_br(%struct.S* nocapture %s1, %struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3, %struct.S* nocapture readonly %s4, i32 %x2) local_unnamed_addr #0 {
+; CHECK-LABEL: test_nondirect_br:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB2_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movl %edx, 4(%rdi)
+; CHECK-NEXT:  .LBB2_2: # %if.end
+; CHECK-NEXT:    cmpl $14, %r9d
+; CHECK-NEXT:    jl .LBB2_4
+; CHECK-NEXT:  # %bb.3: # %if.then2
+; CHECK-NEXT:    movl %r9d, 12(%rdi)
+; CHECK-NEXT:  .LBB2_4: # %if.end3
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movq (%rdi), %rax
+; CHECK-NEXT:    movq %rax, (%rsi)
+; CHECK-NEXT:    movl 8(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 8(%rsi)
+; CHECK-NEXT:    movl 12(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 12(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_nondirect_br:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB2_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movl %edx, 4(%rdi)
+; DISABLED-NEXT:  .LBB2_2: # %if.end
+; DISABLED-NEXT:    cmpl $14, %r9d
+; DISABLED-NEXT:    jl .LBB2_4
+; DISABLED-NEXT:  # %bb.3: # %if.then2
+; DISABLED-NEXT:    movl %r9d, 12(%rdi)
+; DISABLED-NEXT:  .LBB2_4: # %if.end3
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_nondirect_br:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB2_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movl %edx, 4(%rdi)
+; CHECK-AVX2-NEXT:  .LBB2_2: # %if.end
+; CHECK-AVX2-NEXT:    cmpl $14, %r9d
+; CHECK-AVX2-NEXT:    jl .LBB2_4
+; CHECK-AVX2-NEXT:  # %bb.3: # %if.then2
+; CHECK-AVX2-NEXT:    movl %r9d, 12(%rdi)
+; CHECK-AVX2-NEXT:  .LBB2_4: # %if.end3
+; CHECK-AVX2-NEXT:    vmovups (%r8), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%rcx)
+; CHECK-AVX2-NEXT:    movq (%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, (%rsi)
+; CHECK-AVX2-NEXT:    movl 8(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 8(%rsi)
+; CHECK-AVX2-NEXT:    movl 12(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 12(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 1
+  store i32 %x, i32* %b, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %cmp1 = icmp sgt i32 %x2, 13
+  br i1 %cmp1, label %if.then2, label %if.end3
+
+if.then2:                                         ; preds = %if.end
+  %d = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 3
+  store i32 %x2, i32* %d, align 4
+  br label %if.end3
+
+if.end3:                                          ; preds = %if.then2, %if.end
+  %0 = bitcast %struct.S* %s3 to i8*
+  %1 = bitcast %struct.S* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
+  %2 = bitcast %struct.S* %s2 to i8*
+  %3 = bitcast %struct.S* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 4, i1 false)
+  ret void
+}
+
+; Function Attrs: nounwind uwtable
+define void @test_2preds_block(%struct.S* nocapture %s1, %struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3, %struct.S* nocapture readonly %s4, i32 %x2) local_unnamed_addr #0 {
+; CHECK-LABEL: test_2preds_block:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %r9d, 12(%rdi)
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB3_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movl %edx, 4(%rdi)
+; CHECK-NEXT:  .LBB3_2: # %if.end
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movl (%rdi), %eax
+; CHECK-NEXT:    movl %eax, (%rsi)
+; CHECK-NEXT:    movl 4(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 4(%rsi)
+; CHECK-NEXT:    movl 8(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 8(%rsi)
+; CHECK-NEXT:    movl 12(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 12(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_2preds_block:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    movl %r9d, 12(%rdi)
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB3_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movl %edx, 4(%rdi)
+; DISABLED-NEXT:  .LBB3_2: # %if.end
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_2preds_block:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    movl %r9d, 12(%rdi)
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB3_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movl %edx, 4(%rdi)
+; CHECK-AVX2-NEXT:  .LBB3_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r8), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%rcx)
+; CHECK-AVX2-NEXT:    movl (%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, (%rsi)
+; CHECK-AVX2-NEXT:    movl 4(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 4(%rsi)
+; CHECK-AVX2-NEXT:    movl 8(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 8(%rsi)
+; CHECK-AVX2-NEXT:    movl 12(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 12(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %d = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 3
+  store i32 %x2, i32* %d, align 4
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 1
+  store i32 %x, i32* %b, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S* %s3 to i8*
+  %1 = bitcast %struct.S* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
+  %2 = bitcast %struct.S* %s2 to i8*
+  %3 = bitcast %struct.S* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 4, i1 false)
+  ret void
+}
+%struct.S2 = type { i64, i64 }
+
+; Function Attrs: nounwind uwtable
+define void @test_type64(%struct.S2* nocapture %s1, %struct.S2* nocapture %s2, i32 %x, %struct.S2* nocapture %s3, %struct.S2* nocapture readonly %s4) local_unnamed_addr #0 {
+; CHECK-LABEL: test_type64:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB4_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movslq %edx, %rax
+; CHECK-NEXT:    movq %rax, 8(%rdi)
+; CHECK-NEXT:  .LBB4_2: # %if.end
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movq (%rdi), %rax
+; CHECK-NEXT:    movq %rax, (%rsi)
+; CHECK-NEXT:    movq 8(%rdi), %rax
+; CHECK-NEXT:    movq %rax, 8(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_type64:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB4_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movslq %edx, %rax
+; DISABLED-NEXT:    movq %rax, 8(%rdi)
+; DISABLED-NEXT:  .LBB4_2: # %if.end
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_type64:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB4_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movslq %edx, %rax
+; CHECK-AVX2-NEXT:    movq %rax, 8(%rdi)
+; CHECK-AVX2-NEXT:  .LBB4_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r8), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%rcx)
+; CHECK-AVX2-NEXT:    movq (%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, (%rsi)
+; CHECK-AVX2-NEXT:    movq 8(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 8(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %conv = sext i32 %x to i64
+  %b = getelementptr inbounds %struct.S2, %struct.S2* %s1, i64 0, i32 1
+  store i64 %conv, i64* %b, align 8
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S2* %s3 to i8*
+  %1 = bitcast %struct.S2* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 8, i1 false)
+  %2 = bitcast %struct.S2* %s2 to i8*
+  %3 = bitcast %struct.S2* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 8, i1 false)
+  ret void
+}
+%struct.S3 = type { i64, i8, i8, i16, i32 }
+
+; Function Attrs: noinline nounwind uwtable
+define void @test_mixed_type(%struct.S3* nocapture %s1, %struct.S3* nocapture %s2, i32 %x, %struct.S3* nocapture readnone %s3, %struct.S3* nocapture readnone %s4) local_unnamed_addr #0 {
+; CHECK-LABEL: test_mixed_type:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB5_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movslq %edx, %rax
+; CHECK-NEXT:    movq %rax, (%rdi)
+; CHECK-NEXT:    movb %dl, 8(%rdi)
+; CHECK-NEXT:  .LBB5_2: # %if.end
+; CHECK-NEXT:    movq (%rdi), %rax
+; CHECK-NEXT:    movq %rax, (%rsi)
+; CHECK-NEXT:    movb 8(%rdi), %al
+; CHECK-NEXT:    movb %al, 8(%rsi)
+; CHECK-NEXT:    movl 9(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 9(%rsi)
+; CHECK-NEXT:    movzwl 13(%rdi), %eax
+; CHECK-NEXT:    movw %ax, 13(%rsi)
+; CHECK-NEXT:    movb 15(%rdi), %al
+; CHECK-NEXT:    movb %al, 15(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_mixed_type:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB5_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movslq %edx, %rax
+; DISABLED-NEXT:    movq %rax, (%rdi)
+; DISABLED-NEXT:    movb %dl, 8(%rdi)
+; DISABLED-NEXT:  .LBB5_2: # %if.end
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_mixed_type:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB5_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movslq %edx, %rax
+; CHECK-AVX2-NEXT:    movq %rax, (%rdi)
+; CHECK-AVX2-NEXT:    movb %dl, 8(%rdi)
+; CHECK-AVX2-NEXT:  .LBB5_2: # %if.end
+; CHECK-AVX2-NEXT:    movq (%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, (%rsi)
+; CHECK-AVX2-NEXT:    movb 8(%rdi), %al
+; CHECK-AVX2-NEXT:    movb %al, 8(%rsi)
+; CHECK-AVX2-NEXT:    movl 9(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 9(%rsi)
+; CHECK-AVX2-NEXT:    movzwl 13(%rdi), %eax
+; CHECK-AVX2-NEXT:    movw %ax, 13(%rsi)
+; CHECK-AVX2-NEXT:    movb 15(%rdi), %al
+; CHECK-AVX2-NEXT:    movb %al, 15(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %conv = sext i32 %x to i64
+  %a = getelementptr inbounds %struct.S3, %struct.S3* %s1, i64 0, i32 0
+  store i64 %conv, i64* %a, align 8
+  %conv1 = trunc i32 %x to i8
+  %b = getelementptr inbounds %struct.S3, %struct.S3* %s1, i64 0, i32 1
+  store i8 %conv1, i8* %b, align 8
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S3* %s2 to i8*
+  %1 = bitcast %struct.S3* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 8, i1 false)
+  ret void
+}
+%struct.S4 = type { i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 }
+
+; Function Attrs: nounwind uwtable
+define void @test_multiple_blocks(%struct.S4* nocapture %s1, %struct.S4* nocapture %s2) local_unnamed_addr #0 {
+; CHECK-LABEL: test_multiple_blocks:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl $0, 4(%rdi)
+; CHECK-NEXT:    movl $0, 36(%rdi)
+; CHECK-NEXT:    movups 16(%rdi), %xmm0
+; CHECK-NEXT:    movups %xmm0, 16(%rsi)
+; CHECK-NEXT:    movl 32(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 32(%rsi)
+; CHECK-NEXT:    movl 36(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 36(%rsi)
+; CHECK-NEXT:    movq 40(%rdi), %rax
+; CHECK-NEXT:    movq %rax, 40(%rsi)
+; CHECK-NEXT:    movl (%rdi), %eax
+; CHECK-NEXT:    movl %eax, (%rsi)
+; CHECK-NEXT:    movl 4(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 4(%rsi)
+; CHECK-NEXT:    movq 8(%rdi), %rax
+; CHECK-NEXT:    movq %rax, 8(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_multiple_blocks:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    movl $0, 4(%rdi)
+; DISABLED-NEXT:    movl $0, 36(%rdi)
+; DISABLED-NEXT:    movups 16(%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, 16(%rsi)
+; DISABLED-NEXT:    movups 32(%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, 32(%rsi)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_multiple_blocks:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    movl $0, 4(%rdi)
+; CHECK-AVX2-NEXT:    movl $0, 36(%rdi)
+; CHECK-AVX2-NEXT:    vmovups 16(%rdi), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, 16(%rsi)
+; CHECK-AVX2-NEXT:    movl 32(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 32(%rsi)
+; CHECK-AVX2-NEXT:    movl 36(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 36(%rsi)
+; CHECK-AVX2-NEXT:    movq 40(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 40(%rsi)
+; CHECK-AVX2-NEXT:    movl (%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, (%rsi)
+; CHECK-AVX2-NEXT:    movl 4(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 4(%rsi)
+; CHECK-AVX2-NEXT:    vmovups 8(%rdi), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, 8(%rsi)
+; CHECK-AVX2-NEXT:    movq 24(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 24(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %b = getelementptr inbounds %struct.S4, %struct.S4* %s1, i64 0, i32 1
+  store i32 0, i32* %b, align 4
+  %b3 = getelementptr inbounds %struct.S4, %struct.S4* %s1, i64 0, i32 9
+  store i32 0, i32* %b3, align 4
+  %0 = bitcast %struct.S4* %s2 to i8*
+  %1 = bitcast %struct.S4* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 48, i32 4, i1 false)
+  ret void
+}
+%struct.S5 = type { i16, i16, i16, i16, i16, i16, i16, i16 }
+
+; Function Attrs: nounwind uwtable
+define void @test_type16(%struct.S5* nocapture %s1, %struct.S5* nocapture %s2, i32 %x, %struct.S5* nocapture %s3, %struct.S5* nocapture readonly %s4) local_unnamed_addr #0 {
+; CHECK-LABEL: test_type16:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB7_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movw %dx, 2(%rdi)
+; CHECK-NEXT:  .LBB7_2: # %if.end
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movzwl (%rdi), %eax
+; CHECK-NEXT:    movw %ax, (%rsi)
+; CHECK-NEXT:    movzwl 2(%rdi), %eax
+; CHECK-NEXT:    movw %ax, 2(%rsi)
+; CHECK-NEXT:    movq 4(%rdi), %rax
+; CHECK-NEXT:    movq %rax, 4(%rsi)
+; CHECK-NEXT:    movl 12(%rdi), %eax
+; CHECK-NEXT:    movl %eax, 12(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_type16:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB7_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movw %dx, 2(%rdi)
+; DISABLED-NEXT:  .LBB7_2: # %if.end
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_type16:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB7_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movw %dx, 2(%rdi)
+; CHECK-AVX2-NEXT:  .LBB7_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r8), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%rcx)
+; CHECK-AVX2-NEXT:    movzwl (%rdi), %eax
+; CHECK-AVX2-NEXT:    movw %ax, (%rsi)
+; CHECK-AVX2-NEXT:    movzwl 2(%rdi), %eax
+; CHECK-AVX2-NEXT:    movw %ax, 2(%rsi)
+; CHECK-AVX2-NEXT:    movq 4(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 4(%rsi)
+; CHECK-AVX2-NEXT:    movl 12(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 12(%rsi)
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %conv = trunc i32 %x to i16
+  %b = getelementptr inbounds %struct.S5, %struct.S5* %s1, i64 0, i32 1
+  store i16 %conv, i16* %b, align 2
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S5* %s3 to i8*
+  %1 = bitcast %struct.S5* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 2, i1 false)
+  %2 = bitcast %struct.S5* %s2 to i8*
+  %3 = bitcast %struct.S5* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 2, i1 false)
+  ret void
+}
+
+%struct.S6 = type { [4 x i32], i32, i32, i32, i32 }
+
+; Function Attrs: nounwind uwtable
+define void @test_stack(%struct.S6* noalias nocapture sret %agg.result, %struct.S6* byval nocapture readnone align 8 %s1, %struct.S6* byval nocapture align 8 %s2, i32 %x) local_unnamed_addr #0 {
+; CHECK-LABEL: test_stack:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl %esi, {{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%rdi)
+; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; CHECK-NEXT:    movq %rax, 16(%rdi)
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %eax
+; CHECK-NEXT:    movl %eax, 24(%rdi)
+; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %eax
+; CHECK-NEXT:    movl %eax, 28(%rdi)
+; CHECK-NEXT:    movq %rdi, %rax
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_stack:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    movl %esi, {{[0-9]+}}(%rsp)
+; DISABLED-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%rdi)
+; DISABLED-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm0
+; DISABLED-NEXT:    movups %xmm0, 16(%rdi)
+; DISABLED-NEXT:    movq %rdi, %rax
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_stack:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    movl %esi, {{[0-9]+}}(%rsp)
+; CHECK-AVX2-NEXT:    vmovups {{[0-9]+}}(%rsp), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%rdi)
+; CHECK-AVX2-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 16(%rdi)
+; CHECK-AVX2-NEXT:    movl {{[0-9]+}}(%rsp), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 24(%rdi)
+; CHECK-AVX2-NEXT:    movl {{[0-9]+}}(%rsp), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 28(%rdi)
+; CHECK-AVX2-NEXT:    movq %rdi, %rax
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %s6.sroa.0.0..sroa_cast1 = bitcast %struct.S6* %s2 to i8*
+  %s6.sroa.3.0..sroa_idx4 = getelementptr inbounds %struct.S6, %struct.S6* %s2, i64 0, i32 3
+  store i32 %x, i32* %s6.sroa.3.0..sroa_idx4, align 8
+  %0 = bitcast %struct.S6* %agg.result to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* nonnull %s6.sroa.0.0..sroa_cast1, i64 32, i32 4, i1 false)
+  ret void
+}
+
+; Function Attrs: nounwind uwtable
+define void @test_limit_all(%struct.S* %s1, %struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3, %struct.S* nocapture readonly %s4, i32 %x2) local_unnamed_addr #0 {
+; CHECK-LABEL: test_limit_all:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    pushq %r15
+; CHECK-NEXT:    .cfi_def_cfa_offset 24
+; CHECK-NEXT:    pushq %r14
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    pushq %r12
+; CHECK-NEXT:    .cfi_def_cfa_offset 40
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    .cfi_def_cfa_offset 48
+; CHECK-NEXT:    .cfi_offset %rbx, -48
+; CHECK-NEXT:    .cfi_offset %r12, -40
+; CHECK-NEXT:    .cfi_offset %r14, -32
+; CHECK-NEXT:    .cfi_offset %r15, -24
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %r8, %r15
+; CHECK-NEXT:    movq %rcx, %r14
+; CHECK-NEXT:    movl %edx, %ebp
+; CHECK-NEXT:    movq %rsi, %r12
+; CHECK-NEXT:    movq %rdi, %rbx
+; CHECK-NEXT:    movl %r9d, 12(%rbx)
+; CHECK-NEXT:    callq bar
+; CHECK-NEXT:    cmpl $18, %ebp
+; CHECK-NEXT:    jl .LBB9_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movl %ebp, 4(%rbx)
+; CHECK-NEXT:    movq %rbx, %rdi
+; CHECK-NEXT:    callq bar
+; CHECK-NEXT:  .LBB9_2: # %if.end
+; CHECK-NEXT:    movups (%r15), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%r14)
+; CHECK-NEXT:    movups (%rbx), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%r12)
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %r12
+; CHECK-NEXT:    popq %r14
+; CHECK-NEXT:    popq %r15
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_limit_all:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    pushq %rbp
+; DISABLED-NEXT:    .cfi_def_cfa_offset 16
+; DISABLED-NEXT:    pushq %r15
+; DISABLED-NEXT:    .cfi_def_cfa_offset 24
+; DISABLED-NEXT:    pushq %r14
+; DISABLED-NEXT:    .cfi_def_cfa_offset 32
+; DISABLED-NEXT:    pushq %r12
+; DISABLED-NEXT:    .cfi_def_cfa_offset 40
+; DISABLED-NEXT:    pushq %rbx
+; DISABLED-NEXT:    .cfi_def_cfa_offset 48
+; DISABLED-NEXT:    .cfi_offset %rbx, -48
+; DISABLED-NEXT:    .cfi_offset %r12, -40
+; DISABLED-NEXT:    .cfi_offset %r14, -32
+; DISABLED-NEXT:    .cfi_offset %r15, -24
+; DISABLED-NEXT:    .cfi_offset %rbp, -16
+; DISABLED-NEXT:    movq %r8, %r15
+; DISABLED-NEXT:    movq %rcx, %r14
+; DISABLED-NEXT:    movl %edx, %ebp
+; DISABLED-NEXT:    movq %rsi, %r12
+; DISABLED-NEXT:    movq %rdi, %rbx
+; DISABLED-NEXT:    movl %r9d, 12(%rbx)
+; DISABLED-NEXT:    callq bar
+; DISABLED-NEXT:    cmpl $18, %ebp
+; DISABLED-NEXT:    jl .LBB9_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movl %ebp, 4(%rbx)
+; DISABLED-NEXT:    movq %rbx, %rdi
+; DISABLED-NEXT:    callq bar
+; DISABLED-NEXT:  .LBB9_2: # %if.end
+; DISABLED-NEXT:    movups (%r15), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%r14)
+; DISABLED-NEXT:    movups (%rbx), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%r12)
+; DISABLED-NEXT:    popq %rbx
+; DISABLED-NEXT:    popq %r12
+; DISABLED-NEXT:    popq %r14
+; DISABLED-NEXT:    popq %r15
+; DISABLED-NEXT:    popq %rbp
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_limit_all:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    pushq %rbp
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-AVX2-NEXT:    pushq %r15
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 24
+; CHECK-AVX2-NEXT:    pushq %r14
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-AVX2-NEXT:    pushq %r12
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 40
+; CHECK-AVX2-NEXT:    pushq %rbx
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 48
+; CHECK-AVX2-NEXT:    .cfi_offset %rbx, -48
+; CHECK-AVX2-NEXT:    .cfi_offset %r12, -40
+; CHECK-AVX2-NEXT:    .cfi_offset %r14, -32
+; CHECK-AVX2-NEXT:    .cfi_offset %r15, -24
+; CHECK-AVX2-NEXT:    .cfi_offset %rbp, -16
+; CHECK-AVX2-NEXT:    movq %r8, %r15
+; CHECK-AVX2-NEXT:    movq %rcx, %r14
+; CHECK-AVX2-NEXT:    movl %edx, %ebp
+; CHECK-AVX2-NEXT:    movq %rsi, %r12
+; CHECK-AVX2-NEXT:    movq %rdi, %rbx
+; CHECK-AVX2-NEXT:    movl %r9d, 12(%rbx)
+; CHECK-AVX2-NEXT:    callq bar
+; CHECK-AVX2-NEXT:    cmpl $18, %ebp
+; CHECK-AVX2-NEXT:    jl .LBB9_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movl %ebp, 4(%rbx)
+; CHECK-AVX2-NEXT:    movq %rbx, %rdi
+; CHECK-AVX2-NEXT:    callq bar
+; CHECK-AVX2-NEXT:  .LBB9_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r15), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%r14)
+; CHECK-AVX2-NEXT:    vmovups (%rbx), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%r12)
+; CHECK-AVX2-NEXT:    popq %rbx
+; CHECK-AVX2-NEXT:    popq %r12
+; CHECK-AVX2-NEXT:    popq %r14
+; CHECK-AVX2-NEXT:    popq %r15
+; CHECK-AVX2-NEXT:    popq %rbp
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %d = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 3
+  store i32 %x2, i32* %d, align 4
+  tail call void @bar(%struct.S* %s1) #3
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 1
+  store i32 %x, i32* %b, align 4
+  tail call void @bar(%struct.S* nonnull %s1) #3
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S* %s3 to i8*
+  %1 = bitcast %struct.S* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
+  %2 = bitcast %struct.S* %s2 to i8*
+  %3 = bitcast %struct.S* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 4, i1 false)
+  ret void
+}
+
+; Function Attrs: nounwind uwtable
+define void @test_limit_one_pred(%struct.S* %s1, %struct.S* nocapture %s2, i32 %x, %struct.S* nocapture %s3, %struct.S* nocapture readonly %s4, i32 %x2) local_unnamed_addr #0 {
+; CHECK-LABEL: test_limit_one_pred:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %r15
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    pushq %r14
+; CHECK-NEXT:    .cfi_def_cfa_offset 24
+; CHECK-NEXT:    pushq %r12
+; CHECK-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    .cfi_def_cfa_offset 40
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 48
+; CHECK-NEXT:    .cfi_offset %rbx, -40
+; CHECK-NEXT:    .cfi_offset %r12, -32
+; CHECK-NEXT:    .cfi_offset %r14, -24
+; CHECK-NEXT:    .cfi_offset %r15, -16
+; CHECK-NEXT:    movq %r8, %r12
+; CHECK-NEXT:    movq %rcx, %r15
+; CHECK-NEXT:    movq %rsi, %r14
+; CHECK-NEXT:    movq %rdi, %rbx
+; CHECK-NEXT:    movl %r9d, 12(%rbx)
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB10_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movl %edx, 4(%rbx)
+; CHECK-NEXT:    movq %rbx, %rdi
+; CHECK-NEXT:    callq bar
+; CHECK-NEXT:  .LBB10_2: # %if.end
+; CHECK-NEXT:    movups (%r12), %xmm0
+; CHECK-NEXT:    movups %xmm0, (%r15)
+; CHECK-NEXT:    movq (%rbx), %rax
+; CHECK-NEXT:    movq %rax, (%r14)
+; CHECK-NEXT:    movl 8(%rbx), %eax
+; CHECK-NEXT:    movl %eax, 8(%r14)
+; CHECK-NEXT:    movl 12(%rbx), %eax
+; CHECK-NEXT:    movl %eax, 12(%r14)
+; CHECK-NEXT:    addq $8, %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %r12
+; CHECK-NEXT:    popq %r14
+; CHECK-NEXT:    popq %r15
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_limit_one_pred:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    pushq %r15
+; DISABLED-NEXT:    .cfi_def_cfa_offset 16
+; DISABLED-NEXT:    pushq %r14
+; DISABLED-NEXT:    .cfi_def_cfa_offset 24
+; DISABLED-NEXT:    pushq %r12
+; DISABLED-NEXT:    .cfi_def_cfa_offset 32
+; DISABLED-NEXT:    pushq %rbx
+; DISABLED-NEXT:    .cfi_def_cfa_offset 40
+; DISABLED-NEXT:    pushq %rax
+; DISABLED-NEXT:    .cfi_def_cfa_offset 48
+; DISABLED-NEXT:    .cfi_offset %rbx, -40
+; DISABLED-NEXT:    .cfi_offset %r12, -32
+; DISABLED-NEXT:    .cfi_offset %r14, -24
+; DISABLED-NEXT:    .cfi_offset %r15, -16
+; DISABLED-NEXT:    movq %r8, %r15
+; DISABLED-NEXT:    movq %rcx, %r14
+; DISABLED-NEXT:    movq %rsi, %r12
+; DISABLED-NEXT:    movq %rdi, %rbx
+; DISABLED-NEXT:    movl %r9d, 12(%rbx)
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB10_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movl %edx, 4(%rbx)
+; DISABLED-NEXT:    movq %rbx, %rdi
+; DISABLED-NEXT:    callq bar
+; DISABLED-NEXT:  .LBB10_2: # %if.end
+; DISABLED-NEXT:    movups (%r15), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%r14)
+; DISABLED-NEXT:    movups (%rbx), %xmm0
+; DISABLED-NEXT:    movups %xmm0, (%r12)
+; DISABLED-NEXT:    addq $8, %rsp
+; DISABLED-NEXT:    popq %rbx
+; DISABLED-NEXT:    popq %r12
+; DISABLED-NEXT:    popq %r14
+; DISABLED-NEXT:    popq %r15
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_limit_one_pred:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    pushq %r15
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-AVX2-NEXT:    pushq %r14
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 24
+; CHECK-AVX2-NEXT:    pushq %r12
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 32
+; CHECK-AVX2-NEXT:    pushq %rbx
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 40
+; CHECK-AVX2-NEXT:    pushq %rax
+; CHECK-AVX2-NEXT:    .cfi_def_cfa_offset 48
+; CHECK-AVX2-NEXT:    .cfi_offset %rbx, -40
+; CHECK-AVX2-NEXT:    .cfi_offset %r12, -32
+; CHECK-AVX2-NEXT:    .cfi_offset %r14, -24
+; CHECK-AVX2-NEXT:    .cfi_offset %r15, -16
+; CHECK-AVX2-NEXT:    movq %r8, %r12
+; CHECK-AVX2-NEXT:    movq %rcx, %r15
+; CHECK-AVX2-NEXT:    movq %rsi, %r14
+; CHECK-AVX2-NEXT:    movq %rdi, %rbx
+; CHECK-AVX2-NEXT:    movl %r9d, 12(%rbx)
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB10_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movl %edx, 4(%rbx)
+; CHECK-AVX2-NEXT:    movq %rbx, %rdi
+; CHECK-AVX2-NEXT:    callq bar
+; CHECK-AVX2-NEXT:  .LBB10_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r12), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, (%r15)
+; CHECK-AVX2-NEXT:    movq (%rbx), %rax
+; CHECK-AVX2-NEXT:    movq %rax, (%r14)
+; CHECK-AVX2-NEXT:    movl 8(%rbx), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 8(%r14)
+; CHECK-AVX2-NEXT:    movl 12(%rbx), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 12(%r14)
+; CHECK-AVX2-NEXT:    addq $8, %rsp
+; CHECK-AVX2-NEXT:    popq %rbx
+; CHECK-AVX2-NEXT:    popq %r12
+; CHECK-AVX2-NEXT:    popq %r14
+; CHECK-AVX2-NEXT:    popq %r15
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %d = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 3
+  store i32 %x2, i32* %d, align 4
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S, %struct.S* %s1, i64 0, i32 1
+  store i32 %x, i32* %b, align 4
+  tail call void @bar(%struct.S* nonnull %s1) #3
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S* %s3 to i8*
+  %1 = bitcast %struct.S* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 16, i32 4, i1 false)
+  %2 = bitcast %struct.S* %s2 to i8*
+  %3 = bitcast %struct.S* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 16, i32 4, i1 false)
+  ret void
+}
+
+
+declare void @bar(%struct.S*) local_unnamed_addr #1
+
+
+; Function Attrs: argmemonly nounwind
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture writeonly, i8* nocapture readonly, i64, i32, i1) #1
+
+attributes #0 = { nounwind uwtable "target-cpu"="x86-64" }
+
+%struct.S7 = type { float, float, float , float, float, float, float, float }
+
+; Function Attrs: nounwind uwtable
+define void @test_conditional_block_float(%struct.S7* nocapture %s1, %struct.S7* nocapture %s2, i32 %x, %struct.S7* nocapture %s3, %struct.S7* nocapture readonly %s4, float %y) local_unnamed_addr #0 {
+; CHECK-LABEL: test_conditional_block_float:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB11_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movl $1065353216, 4(%rdi) # imm = 0x3F800000
+; CHECK-NEXT:  .LBB11_2: # %if.end
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups 16(%r8), %xmm1
+; CHECK-NEXT:    movups %xmm1, 16(%rcx)
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movl (%rdi), %eax
+; CHECK-NEXT:    movl 4(%rdi), %ecx
+; CHECK-NEXT:    movq 8(%rdi), %rdx
+; CHECK-NEXT:    movups 16(%rdi), %xmm0
+; CHECK-NEXT:    movups %xmm0, 16(%rsi)
+; CHECK-NEXT:    movl %eax, (%rsi)
+; CHECK-NEXT:    movl %ecx, 4(%rsi)
+; CHECK-NEXT:    movq %rdx, 8(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_conditional_block_float:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB11_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movl $1065353216, 4(%rdi) # imm = 0x3F800000
+; DISABLED-NEXT:  .LBB11_2: # %if.end
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups 16(%r8), %xmm1
+; DISABLED-NEXT:    movups %xmm1, 16(%rcx)
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups 16(%rdi), %xmm1
+; DISABLED-NEXT:    movups %xmm1, 16(%rsi)
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_conditional_block_float:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB11_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movl $1065353216, 4(%rdi) # imm = 0x3F800000
+; CHECK-AVX2-NEXT:  .LBB11_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r8), %ymm0
+; CHECK-AVX2-NEXT:    vmovups %ymm0, (%rcx)
+; CHECK-AVX2-NEXT:    movl (%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, (%rsi)
+; CHECK-AVX2-NEXT:    movl 4(%rdi), %eax
+; CHECK-AVX2-NEXT:    movl %eax, 4(%rsi)
+; CHECK-AVX2-NEXT:    vmovups 8(%rdi), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, 8(%rsi)
+; CHECK-AVX2-NEXT:    movq 24(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 24(%rsi)
+; CHECK-AVX2-NEXT:    vzeroupper
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S7, %struct.S7* %s1, i64 0, i32 1
+  store float 1.0, float* %b, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S7* %s3 to i8*
+  %1 = bitcast %struct.S7* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 32, i32 4, i1 false)
+  %2 = bitcast %struct.S7* %s2 to i8*
+  %3 = bitcast %struct.S7* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 32, i32 4, i1 false)
+  ret void
+}
+
+%struct.S8 = type { i64, i64, i64, i64, i64, i64 }
+
+; Function Attrs: nounwind uwtable
+define void @test_conditional_block_ymm(%struct.S8* nocapture %s1, %struct.S8* nocapture %s2, i32 %x, %struct.S8* nocapture %s3, %struct.S8* nocapture readonly %s4) local_unnamed_addr #0 {
+; CHECK-LABEL: test_conditional_block_ymm:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    cmpl $18, %edx
+; CHECK-NEXT:    jl .LBB12_2
+; CHECK-NEXT:  # %bb.1: # %if.then
+; CHECK-NEXT:    movq $1, 8(%rdi)
+; CHECK-NEXT:  .LBB12_2: # %if.end
+; CHECK-NEXT:    movups (%r8), %xmm0
+; CHECK-NEXT:    movups 16(%r8), %xmm1
+; CHECK-NEXT:    movups %xmm1, 16(%rcx)
+; CHECK-NEXT:    movups %xmm0, (%rcx)
+; CHECK-NEXT:    movq (%rdi), %rax
+; CHECK-NEXT:    movq 8(%rdi), %rcx
+; CHECK-NEXT:    movups 16(%rdi), %xmm0
+; CHECK-NEXT:    movups %xmm0, 16(%rsi)
+; CHECK-NEXT:    movq %rax, (%rsi)
+; CHECK-NEXT:    movq %rcx, 8(%rsi)
+; CHECK-NEXT:    retq
+;
+; DISABLED-LABEL: test_conditional_block_ymm:
+; DISABLED:       # %bb.0: # %entry
+; DISABLED-NEXT:    cmpl $18, %edx
+; DISABLED-NEXT:    jl .LBB12_2
+; DISABLED-NEXT:  # %bb.1: # %if.then
+; DISABLED-NEXT:    movq $1, 8(%rdi)
+; DISABLED-NEXT:  .LBB12_2: # %if.end
+; DISABLED-NEXT:    movups (%r8), %xmm0
+; DISABLED-NEXT:    movups 16(%r8), %xmm1
+; DISABLED-NEXT:    movups %xmm1, 16(%rcx)
+; DISABLED-NEXT:    movups %xmm0, (%rcx)
+; DISABLED-NEXT:    movups (%rdi), %xmm0
+; DISABLED-NEXT:    movups 16(%rdi), %xmm1
+; DISABLED-NEXT:    movups %xmm1, 16(%rsi)
+; DISABLED-NEXT:    movups %xmm0, (%rsi)
+; DISABLED-NEXT:    retq
+;
+; CHECK-AVX2-LABEL: test_conditional_block_ymm:
+; CHECK-AVX2:       # %bb.0: # %entry
+; CHECK-AVX2-NEXT:    cmpl $18, %edx
+; CHECK-AVX2-NEXT:    jl .LBB12_2
+; CHECK-AVX2-NEXT:  # %bb.1: # %if.then
+; CHECK-AVX2-NEXT:    movq $1, 8(%rdi)
+; CHECK-AVX2-NEXT:  .LBB12_2: # %if.end
+; CHECK-AVX2-NEXT:    vmovups (%r8), %ymm0
+; CHECK-AVX2-NEXT:    vmovups %ymm0, (%rcx)
+; CHECK-AVX2-NEXT:    movq (%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, (%rsi)
+; CHECK-AVX2-NEXT:    movq 8(%rdi), %rax
+; CHECK-AVX2-NEXT:    movq %rax, 8(%rsi)
+; CHECK-AVX2-NEXT:    vmovups 16(%rdi), %xmm0
+; CHECK-AVX2-NEXT:    vmovups %xmm0, 16(%rsi)
+; CHECK-AVX2-NEXT:    vzeroupper
+; CHECK-AVX2-NEXT:    retq
+entry:
+  %cmp = icmp sgt i32 %x, 17
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %b = getelementptr inbounds %struct.S8, %struct.S8* %s1, i64 0, i32 1
+  store i64 1, i64* %b, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  %0 = bitcast %struct.S8* %s3 to i8*
+  %1 = bitcast %struct.S8* %s4 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* %1, i64 32, i32 4, i1 false)
+  %2 = bitcast %struct.S8* %s2 to i8*
+  %3 = bitcast %struct.S8* %s1 to i8*
+  tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %2, i8* %3, i64 32, i32 4, i1 false)
+  ret void
+}
+