[MIR] Add simple PRE pass to MachineCSE

[anton-afanasyev]

Summary

This is the second part of the commit fixing PR38917 (hoisting
partitially redundant machine instruction). Most of PRE (partitial
redundancy elimination) and CSE work is done on LLVM IR, but some of
redundancy arises during DAG legalization. Machine CSE is not enough
to deal with it. This simple PRE implementation works a little bit
intricately: it passes before CSE, looking for partitial redundancy
and transforming it to fully redundancy, anticipating that the next
CSE step will eliminate this created redundancy. If CSE doesn't
eliminate this, than created instruction will remain dead and eliminated
later by Remove Dead Machine Instructions pass.

The third part of the commit is supposed to refactor MachineCSE,
to make it more clear and to merge MachinePRE with MachineCSE,
so one need no rely on further Remove Dead pass to clear instrs
not eliminated by CSE.

First step: https://reviews.llvm.org/D54839

Fixes llvm.org/PR38917

[anton-afanasyev]

Ping.
Should I clarify anything?

[anton-afanasyev]

Ping!

[hfinkel]

In D56772#1365972, @anton-afanasyev wrote:

Ping!

FYI: Generally accepted ping frequency is once per week.

lib/CodeGen/MachineCSE.cpp

54

Our general style is not to use line continuations like this, but instead rely on adjacent string concatenation, so this would be:

STATISTIC(NumPREs,      "Number of partial redundant expression"
                            " transformed to fully redundant");

(where the starting quotes are vertically aligned)

754

Comments should be full sentences and end with a period (or other punctuation). Also, please explain *why* the conditions are stronger in this case.

790

It looks like this could be:

if (!PREMap.count(MI)) {
  PREMap[MI] = MBB;
  continue;
}

and then you don't need the else and extra associated indentation.

812

It would be better to not create, and then delete, instructions speculatively. isProfitableToCSE does look at the parent blocks of the instructions, but we could abstract that by:

1. Making isProfitableToCSE also take the blocks of the instructions as arguments; explain that isProfitableToCSE treats the instructions as though they are in the specified parent blocks (even if they're not currently).
2. Make an overload isProfitableToCSE which retains the current behavior (where the parent blocks come from the instructions themselves) which calls the underlying implementation from (1).

[anton-afanasyev]

In D56772#1367096, @hfinkel wrote:

In D56772#1365972, @anton-afanasyev wrote:

Ping!

FYI: Generally accepted ping frequency is once per week.

Ok, thanks!

lib/CodeGen/MachineCSE.cpp

54

Ok, fixed.

754

Ok, changed.

790

Thanks, that is better! Fixed.

812

Thanks, changed. Actually isProfitableToCSE() doesn't need CSMI (new instr) as argument, only its basic block, so just changed function arglist.

[hfinkel]

Have you measured the compile-time impact?

lib/CodeGen/MachineCSE.cpp

755

I think it would be useful to elaborate on under what conditions we create these PRE instructions that are not later CSEd. Do you know how often that happens (I assume that you can figure this out by looking at the statistics and comparing runs where this happens vs. where it doesn't)?

[anton-afanasyev]

In D56772#1369571, @hfinkel wrote:

Have you measured the compile-time impact?

What do you mean by compile-time impact? I've run test suite of course, compared with baseline.
The results for exec_time and size..text metrics are like here:

> ~/llvm/test-suite/utils/compare.py  --filter-short -m exec_time ~/llvm/test-suite-results/results_rel_base.json ~/llvm/test-suite-results/results_rel_exp3.json 
...
Program                                         results_rel_base  results_rel_exp3 diff  
                                                                                        
MicroBench...RayFDRMultiThreaded/threads:32   1055.94           931.21            -11.8%
Bitcode/Be...rid/halide_bilateral_grid.test    30.26             26.70            -11.8%
MultiSourc...mbolics-flt/Symbolics-flt.test     1.10              0.98            -10.9%
MicroBench...XRayFDRMultiThreaded/threads:2   497.85            547.92            10.1% 
MicroBench...lcalsCRaw.test:BM_EOS_RAW/5001    10.33              9.49            -8.2% 
MicroBench...bda.test:BM_PIC_1D_LAMBDA/5001    75.75             81.43             7.5% 
MicroBench...XRayFDRMultiThreaded/threads:8   1029.95           1100.16            6.8% 
MicroBench...XRayFDRMultiThreaded/threads:4   626.46            667.17             6.5% 
MicroBench...ambda.test:BM_EOS_LAMBDA/44217    87.21             92.66             6.2% 
MultiSourc...flt/LoopRestructuring-flt.test     3.76              3.53            -6.2% 
MicroBench...calsCRaw.test:BM_EOS_RAW/44217    85.94             90.96             5.8% 
MicroBench...Lambda.test:BM_ADI_LAMBDA/5001   128.27            135.35             5.5% 
MicroBench...est:BM_ReturnNeverInstrumented     1.60              1.68             5.3% 
MicroBench....test:BM_MULADDSUB_LAMBDA/5001    14.14             14.84             4.9% 
SingleSour...ncils/fdtd-apml/fdtd-apml.test     0.63              0.66             4.4%
...
> ~/llvm/test-suite/utils/compare.py  --filter-short -m size..text ~/llvm/test-suite-results/results_rel_base.json ~/llvm/test-suite-results/results_rel_exp3.json 
...
Program                                         results_rel_base  results_rel_exp3 diff 
                                                                                       
SingleSour...marks/Shootout/Shootout-random   658.00            642.00            -2.4%
SingleSour...ce/UnitTests/SignlessTypes/rem   4834.00           4930.00            2.0%
SingleSour...ootout-C++/Shootout-C++-random   818.00            802.00            -2.0%
MultiSource/Benchmarks/Olden/mst/mst          2658.00           2706.00            1.8%
SingleSource/Benchmarks/Misc/ffbench          2322.00           2290.00           -1.4%
MultiSource/Benchmarks/NPB-serial/is/is       4946.00           4898.00           -1.0%
MultiSource/Benchmarks/McCat/18-imp/imp       13266.00          13138.00          -1.0%
MultiSourc...rks/McCat/03-testtrie/testtrie   1810.00           1794.00           -0.9%
MultiSourc...chmarks/McCat/01-qbsort/qbsort   2066.00           2050.00           -0.8%
MultiSourc...iabench/g721/g721encode/encode   7394.00           7346.00           -0.6%
SingleSource/Benchmarks/Misc/fbench           2546.00           2562.00            0.6%
SingleSour...ce/Benchmarks/Misc/ReedSolomon   11170.00          11234.00           0.6%
SingleSource/Benchmarks/Stanford/Towers       2802.00           2786.00           -0.6%
MultiSourc...e/Applications/SIBsim4/SIBsim4   47666.00          47938.00           0.6%
MultiSourc...OE-ProxyApps-C/miniGMG/miniGMG   76258.00          76674.00           0.5%

Although I'm not sure exec_time is good metrics. I've also tried compile_time, never used it before:

~/llvm/test-suite/utils/compare.py  --filter-short -m compile_time ~/llvm/test-suite-results/results_rel_base.json ~/llvm/test-suite-results/results_rel_exp3.json 
...
Program                                         results_rel_base  results_rel_exp3 diff 
                                                                                       
SingleSour.../Benchmarks/Misc-C++/Large/ray     0.68              0.74             7.6%
MultiSourc...Prolangs-C/simulator/simulator     1.56              1.64             5.7%
MultiSource/Benchmarks/McCat/18-imp/imp         0.82              0.78            -5.3%
MultiSourc...OE-ProxyApps-C/RSBench/rsbench     0.88              0.84            -4.6%
MultiSource/Benchmarks/Ptrdist/bc/bc            2.42              2.52             4.1%
SingleSour...enchmarks/CoyoteBench/fftbench     1.08              1.03            -4.1%
SingleSour...UnitTests/Vectorizer/gcc-loops     1.42              1.48             3.9%
MultiSourc...chmarks/Prolangs-C++/city/city     3.20              3.08            -3.9%
MultiSource/Benchmarks/sim/sim                  0.72              0.75             3.9%
MultiSourc...nchmarks/FreeBench/pifft/pifft     2.56              2.65             3.6%
MultiSource/Applications/spiff/spiff            1.58              1.64             3.5%
MultiSourc...lds-flt/CrossingThresholds-flt     3.80              3.94             3.5%
MultiSourc...s/Prolangs-C/compiler/compiler     0.84              0.81            -3.3%
MultiSourc...rks/Prolangs-C++/employ/employ     0.74              0.71            -3.3%
SingleSour...ks/Misc-C++/stepanov_container     3.01              3.11             3.2%
...

Also one can see many tests increased their machine-cse.NumCSEs statistics (meaning that PRE step created common subexpressions eliminated by CSE):

> ~/llvm/test-suite/utils/compare.py  --filter-short -m machine-cse.NumCSEs ~/llvm/test-suite-results_deb_base.json ~/llvm/test-suite-results/results_deb_exp.json
...
Program                                         results_deb_base  results_deb_exp diff   
                                                                                        
SingleSour...ce/UnitTests/SignlessTypes/rem     1.00             19.00           1800.0%
MultiSource/Benchmarks/llubenchmark/llu         1.00             10.00           900.0% 
MultiSource/Benchmarks/NPB-serial/is/is         1.00              9.00           800.0% 
SingleSource/Benchmarks/Misc/fbench             1.00              7.00           600.0% 
MultiSourc...OE-ProxyApps-C/XSBench/XSBench     2.00              8.00           300.0% 
SingleSource/Benchmarks/Misc/ffbench            2.00              8.00           300.0% 
SingleSour...chmarks/Misc-C++/stepanov_v1p2     1.00              3.00           200.0% 
MultiSourc...arks/FreeBench/distray/distray     7.00             20.00           185.7% 
MicroBench.../ImageProcessing/Dither/Dither     3.00              8.00           166.7% 
MultiSourc...ench/telecomm-FFT/telecomm-fft     5.00             13.00           160.0% 
MultiSourc...enchmarks/mafft/pairlocalalign   482.00            1214.00          151.9% 
MultiSourc...aran/netbench-url/netbench-url     3.00              7.00           133.3% 
MultiSourc...Benchmarks/SciMark2-C/scimark2     6.00             13.00           116.7% 
MultiSourc...nch/mpeg2/mpeg2dec/mpeg2decode    57.00            115.00           101.8% 
MultiSourc...chmarks/McCat/01-qbsort/qbsort     2.00              4.00           100.0%
...

lib/CodeGen/MachineCSE.cpp

755

Thanks for pointing this out, I've cleaned this place. I've removed needless checks like MI->isReturn() after comparing statistics (no change).

Mostly, these checks are implicitly done while CSE work (not inside isCSECandidate()). I've left aux checks of virtual registers though -- they are also implicitly done.

I've also updated comment.

This looks artificially, but I'm going to refactor this place in the third (and last) step of this work -- merging PRE and CSE, so we need no different checks.

[anton-afanasyev]

Fixed revision, forgot to add changes.

[anton-afanasyev]

Rebase

[anton-afanasyev]

Ping. Does it have a chance to get lgtm?

[RKSimon]

@hfinkel @MatzeB Any comments?

[anton-afanasyev]

Ping. Please, review this.

[hfinkel]

lib/CodeGen/MachineCSE.cpp

800

Are these calls expensive?

SingleSour.../Benchmarks/Misc-C++/Large/ray 0.68 0.74 7.6%

That 7% increase in compile time, and a few other others, look significant.

(Maybe there's something you can cache?)

[anton-afanasyev]

lib/CodeGen/MachineCSE.cpp

800

I believe caching this gives nothing, cause two common subexpressions unlikely belong to the same basic blocks pair. 7% are only for one test among hundreds, there are also -5% and so on. I'm to run benchmarks again to compare results.

[anton-afanasyev]

Hi @hfinkel,
Unfortunately, the results of compile_time are unstable and not reproducible. Below are two different runs.

The first run:

> ~/llvm/test-suite/utils/compare.py  --filter-short -m compile_time ~/llvm/test-suite-results/results_rel_base.json ~/llvm/test-suite-results/results_rel_exp.json 
Tests: 1163
Short Running: 1010 (filtered out)
Remaining: 153
Metric: compile_time

Program                                         results_rel_base  results_rel_exp diff 
                                                                                      
MultiSourc...OE-ProxyApps-C/XSBench/XSBench     0.74              0.79            6.5%
MicroBench...y/ReturnReference/retref-bench     1.24              1.17           -6.1%
SingleSour...rks/Misc-C++/Large/sphereflake     0.71              0.68           -4.5%
MultiSourc...ps-C++/HACCKernels/HACCKernels     0.72              0.76            4.4%
MultiSourc...chmarks/Prolangs-C/gnugo/gnugo     1.40              1.34           -4.3%
MultiSourc...rks/Prolangs-C++/employ/employ     0.76              0.73           -4.2%
MicroBench...arks/ImageProcessing/Blur/blur     1.61              1.55           -4.0%
MultiSourc...rks/Trimaran/enc-3des/enc-3des     0.62              0.65            3.8%
MultiSourc...rks/mediabench/gsm/toast/toast     2.29              2.38            3.7%
MultiSourc...Benchmarks/SciMark2-C/scimark2     0.79              0.82            3.6%
MicroBench.../ImageProcessing/Dilate/Dilate     1.32              1.36            2.7%
MicroBench...oopInterchange/LoopInterchange     0.74              0.72           -2.7%
MultiSourc...s/Prolangs-C/unix-tbl/unix-tbl     2.22              2.16           -2.7%
MultiSourc...rolangs-C/archie-client/archie     1.20              1.17           -2.7%
MultiSourc...marks/Prolangs-C/loader/loader     0.62              0.60           -2.6%
       results_rel_base  results_rel_exp        diff
count  153.000000        152.000000       152.000000
mean   7.235216          7.270763        -0.000904  
std    12.322023         12.328571        0.016021  
min    0.600000          0.604000        -0.061093  
25%    1.516000          1.535000        -0.007504  
50%    3.784000          3.800000         0.000000  
75%    5.568000          5.581000         0.006387  
max    88.724000         88.972000        0.064516

and the second run:

> ~/llvm/test-suite/utils/compare.py  --filter-short -m compile_time ~/llvm/test-suite-results/results_rel_base2.json ~/llvm/test-suite-results/results_rel_exp2.json 
Tests: 1163
Short Running: 1009 (filtered out)
Remaining: 154
Metric: compile_time

Program                                         results_rel_base2  results_rel_exp2 diff 
                                                                                        
ultiSource/Benchmarks/McCat/09-vor/vor          0.66               0.62            -6.6%
ultiSource...Benchmarks/SciMark2-C/scimark2     0.80               0.75            -5.5%
ultiSource...rolangs-C/archie-client/archie     1.16               1.22             5.2%
ultiSource...OE-ProxyApps-C/XSBench/XSBench     0.79               0.76            -4.1%
ultiSource/Benchmarks/McCat/18-imp/imp          0.80               0.83             4.0%
ultiSource...OE-ProxyApps-C/RSBench/rsbench     0.87               0.90             3.7%
ultiSource...rks/Trimaran/enc-3des/enc-3des     0.66               0.63            -3.7%
ultiSource...hmarks/MallocBench/cfrac/cfrac     1.64               1.70             3.6%
icroBenchm.../ImageProcessing/Dilate/Dilate     1.37               1.32            -3.5%
ultiSource...ALAC/decode/alacconvert-decode     2.41               2.34            -3.0%
ultiSource...chmarks/Prolangs-C++/city/city     3.22               3.13            -3.0%
ultiSource...xyApps-C/Pathfinder/PathFinder     1.77               1.72            -2.9%
ultiSource...s/Prolangs-C/compiler/compiler     0.86               0.84            -2.8%
ultiSource...rks/Prolangs-C++/employ/employ     0.73               0.71            -2.7%
ultiSource/Applications/sgefa/sgefa             0.99               1.02             2.4%
       results_rel_base2  results_rel_exp2        diff
count  152.000000         153.000000        151.000000
mean   7.284842           7.236418         -0.002551  
std    12.346005          12.320806         0.015407  
min    0.600000           0.604000         -0.066265  
25%    1.567000           1.504000         -0.009324  
50%    3.794000           3.776000         -0.001112  
75%    5.609000           5.584000          0.005669  
max    89.004000          89.036000         0.051724

[anton-afanasyev]

Also, I don't think the heavy functions here (like findNearestCommonDominator() and isPotentiallyReachable()) can significant impact compile time -- they are called rarely only for two BBs with common subexpressions, which is unusual. Taking the machine-cse.NumCSE statistics above as example, one can make an estimation of no more than 10-100 calls of these functions for the benchs from test suite. Also, these heavy calls are already cached in a sense, they are using prebuilt Machine Dominator Tree.

[anton-afanasyev]

Hi all, please review this.

@hfinkel, should I clarify anything?

[anton-afanasyev]

Ping!

[anton-afanasyev]

Rebase. Please, review this.

[RKSimon]

@hfinkel Any more comments?

[RKSimon]

LGTM - based on the improved codegen and the negligible effect on compile time. Thanks.

llvm/lib/CodeGen/MachineCSE.cpp

784 ↗

(On Diff #196406)

Very minor - but why can't the ++I be moved inside the for-loop increment statement?

[anton-afanasyev]

llvm/lib/CodeGen/MachineCSE.cpp

784 ↗

(On Diff #196406)

It can, I'm to fix it, thanks.

[anton-afanasyev]

Small fix and tests update

[RKSimon]

llvm/lib/CodeGen/MachineCSE.cpp

784 ↗

(On Diff #196406)

This one too?

[anton-afanasyev]

llvm/lib/CodeGen/MachineCSE.cpp

784 ↗

(On Diff #196406)

Hmm, looks like I've eventually fixed L506 -- the original loop inside ProcessBlockCSE() instead of ProcessBlockPRE().

[anton-afanasyev]

Update, clang-format

[spatel]

We should reopen this review since the patch was reverted at rL359875?

[anton-afanasyev]

Closed by commit https://reviews.llvm.org/rL362901

[john.brawn]

This is causing incorrect code generation for this piece of IR:

target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-arm-none-eabi"

@var = hidden local_unnamed_addr global i32 0, align 4
@_ZTIi = external dso_local constant i8*
declare dso_local void @_Z2fnv() local_unnamed_addr #1
declare dso_local i32 @__gxx_personality_v0(...)
declare i32 @llvm.eh.typeid.for(i8*) #2
declare dso_local i8* @__cxa_begin_catch(i8*) local_unnamed_addr
declare dso_local void @__cxa_end_catch() local_unnamed_addr

define hidden i32 @_Z7examplev() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
entry:
  invoke void @_Z2fnv()
          to label %try.cont unwind label %lpad

lpad:                                             ; preds = %entry
  %0 = landingpad { i8*, i32 }
          catch i8* bitcast (i8** @_ZTIi to i8*)
          catch i8* null
  %1 = extractvalue { i8*, i32 } %0, 0
  %2 = extractvalue { i8*, i32 } %0, 1
  %3 = tail call i32 @llvm.eh.typeid.for(i8* bitcast (i8** @_ZTIi to i8*))
  %matches = icmp eq i32 %2, %3
  %4 = tail call i8* @__cxa_begin_catch(i8* %1)
  %5 = load i32, i32* @var, align 4
  br i1 %matches, label %catch1, label %catch

catch1:                                           ; preds = %lpad
  %or3 = or i32 %5, 4
  store i32 %or3, i32* @var, align 4
  tail call void @__cxa_end_catch()
  br label %try.cont

try.cont:                                         ; preds = %entry, %catch1, %catch
  %6 = load i32, i32* @var, align 4
  ret i32 %6

catch:                                            ; preds = %lpad
  %or = or i32 %5, 8
  store i32 %or, i32* @var, align 4
  tail call void @__cxa_end_catch()
  br label %try.cont
}

As part of the accesses to var an ADRP instruction is being generated and MachineCSE is hoisting it to the entry block, but it gets hoisted to after the call to fn so when we catch an exception the ADRP hasn't been executed so the loads and stores use an undefined base register.

[anton-afanasyev]

In D56772#1537772, @john.brawn wrote:

This is causing incorrect code generation for this piece of IR:

...
As part of the accesses to var an ADRP instruction is being generated and MachineCSE is hoisting it to the entry block, but it gets hoisted to after the call to fn so when we catch an exception the ADRP hasn't been executed so the loads and stores use an undefined base register.

Hi @john.brawn , thanks, I'm looking into it. At first glance, this instruction should have special flag to being non-hoistable.

[anton-afanasyev]

The issue actually is not with this instruction, but it's related to exception handling. Hoisted instruction is inserted before getFirstTerminator(), but there could be EH_LABEL's which are not terminators, but could change control flow.
I'm to fix it.

[anton-afanasyev]

Hi @john.brawn , here is the fix, could you please look to it: https://reviews.llvm.org/D63148

[anton-afanasyev]

Last issue fixed by this revision: https://reviews.llvm.org/D63148

[LuoYuanke]

llvm/trunk/lib/CodeGen/MachineCSE.cpp

814 ↗

(On Diff #197945)

Do we need to enhance the algorithm to consider more about register pressure on the profit calculation? I'm afraid there is performance drop when the register pressure is heavy.

[anton-afanasyev]

llvm/trunk/lib/CodeGen/MachineCSE.cpp

814 ↗

(On Diff #197945)

Hi @LuoYuanke, yes, this could be the case. Actually this commit doesn't change profit calculation, ProcessBlockPRE() uses the same isProfitableToCommit() function as ProcessBlockCSE() uses. Do you have concrete test cases where register pressure increases?

[LuoYuanke]

llvm/trunk/lib/CodeGen/MachineCSE.cpp

814 ↗

(On Diff #197945)

CSE only eliminate MI, but current PRE insert MI to the common dominated block, so the instruction is hoisted to dominated block. It increase register pressure more than CSE.
I have some cases that got performance regression with this patch due to the register pressure, but I am not able to extract a small case from it.
I notice in LICM, MachineLICMBase::IsProfitableToHoist() is more considerate for the register pressure. I wonder if PRE can apply the same algorithm. Do you have any idea for a better register pressure solution?

[anton-afanasyev]

llvm/trunk/lib/CodeGen/MachineCSE.cpp

814 ↗

(On Diff #197945)

Hi @LuoYuanke, actually PRE doesn't hoist instruction, but _duplicates_ it, real hoisting is made by CSE when it _eliminates_ instruction. I do believe that CSE should consider register pressure while eliminating instruction.

Have you tried MachineLICMBase::isProfitableToHoist() with your test case? We can adopt and insert it to CSE.

[LuoYuanke]

llvm/trunk/lib/CodeGen/MachineCSE.cpp

814 ↗

(On Diff #197945)

Hi @anton-afanasyev, sorry for reply late. I'm trying to figure out this issue, but it seems complex for me. Can you run SPEC cpu2017/500.perlbench_r? There is some significant performance drop on X86 with the patch. If you have resource to run the benchmark, could you help to look into the issue?

[LuoYuanke]

@anton-afanasyev
Hi,
Did you look into the SPEC cpu2017/500.perlbench_r issue? There is some significant performance drop on X86 with the patch. I ask you to revert the patch first, and when the SPEC2017 regression is fixed, we can submit the patch again. How do you think?

[anton-afanasyev]

In D56772#1592951, @LuoYuanke wrote:

@anton-afanasyev
Hi,
Did you look into the SPEC cpu2017/500.perlbench_r issue? There is some significant performance drop on X86 with the patch. I ask you to revert the patch first, and when the SPEC2017 regression is fixed, we can submit the patch again. How do you think?

Hi @LuoYuanke, I'm sorry, it looks like my previous answer to you remained unsubmitted.
Here it is:

Hi @LuoYuanke , I have only access to SPEC cpu2006, I see 400.perlbench there. Could it be the case you know regressed sample from cpu2006?
Also, I have prepared new patch for review: https://reviews.llvm.org/D63934 -- I believe it can potentially decrease reg pressure.

At the moment, I believe that another patch (from Kai Luo) https://reviews.llvm.org/D64394 fixes this regression.

[anton-afanasyev]

Btw, this fix (https://reviews.llvm.org/D64394) was commited recently.

[lkail]

llvm/trunk/lib/CodeGen/MachineCSE.cpp

807 ↗

(On Diff #197945)

Hi @anton-afanasyev , I have a concern here that CFG of LLVM IR might not be equivalent to CFG of Machine IR.

[LuoYuanke]

@anton-afanasyev
Hi,
Do you have any performance data for the patch? I'd like to know what benchmark has performance gain with your patch. https://reviews.llvm.org/D64394 fixed perlbench regression, but I wonder what the performance gain do we achieve with the 2 patch?

[anton-afanasyev]

In D56772#1596814, @LuoYuanke wrote:

@anton-afanasyev
Hi,
Do you have any performance data for the patch? I'd like to know what benchmark has performance gain with your patch. https://reviews.llvm.org/D64394 fixed perlbench regression, but I wonder what the performance gain do we achieve with the 2 patch?

Hi @LuoYuanke, I've benchmarked the first patch and posted results in two posts started from here: https://reviews.llvm.org/D56772#1376284.
But you are right: with the second patch the performance gain may be eliminated! I'm to measure second patch effect.

llvm/trunk/lib/CodeGen/MachineCSE.cpp

807 ↗

(On Diff #197945)

Hi @lkail, yes this is possible case. But that is not an issue actually, for this case in place it would be just non-optimization. PRE will create instruction, but if CSE doesn't eliminate original instructions, than "hoisted" instruction will be deleted by Remove Dead Machine Instructions pass later.

[anton-afanasyev]

Hi @LuoYuanke, I've benchmarked the effect of the revertion my and @lkail patches.
The benchmark showed some increase of the exec_time:

~/llvm/test-suite/utils/compare.py  --filter-short -m exec_time ~/llvm/test-suite-results/results_rel_base.json ~/llvm/test-suite-results/results_rel_base2.json vs ~/llvm/test-suite-results/results_rel_exp.json  ~/llvm/test-suite-results/results_rel_exp2.json 

Program                                        lhs    rhs     diff 
 test-suite...Raw.test:BM_MULADDSUB_RAW/5001    13.96  17.35  24.3%
 test-suite...XRayFDRMultiThreaded/threads:4   568.16 634.38  11.7%
 test-suite...RayFDRMultiThreaded/threads:32   858.02 944.45  10.1%
 test-suite...XRayFDRMultiThreaded/threads:2   464.93 501.92   8.0%
 test-suite...algebra/kernels/symm/symm.test    14.00  15.12   7.9%
 test-suite...st:BM_BAND_LIN_EQ_LAMBDA/44217    39.41  37.34  -5.2%
 test-suite...test:BM_MULADDSUB_LAMBDA/44217   141.97 148.38   4.5%
 test-suite...Lambda.test:BM_EOS_LAMBDA/5001     9.88   9.48  -4.0%
 test-suite...ambda.test:BM_EOS_LAMBDA/44217    89.08  85.78  -3.7%
 test-suite...s/Halide/blur/halide_blur.test     1.77   1.83   3.6%
 test-suite...BenchmarkGame/Large/fasta.test     0.72   0.70  -3.3%
 test-suite...lcalsCRaw.test:BM_EOS_RAW/5001     9.86   9.54  -3.3%
 test-suite...calsCRaw.test:BM_EOS_RAW/44217    88.90  86.00  -3.3%
 test-suite...XRayFDRMultiThreaded/threads:8   983.92 1014.32  3.1%
 test-suite...RayFDRMultiThreaded/threads:16   979.55 1008.68  3.0%

So I'd like to leave these changes unreverted and to proceed with particular patches (like this one https://reviews.llvm.org/D63934). Please tell if you have any objections.

[LuoYuanke]

@anton-afanasyev, I have no objection. Thank you for the effort to improve the performance.

lib/CodeGen/MachineCSE.cpp

 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/ScopedHashTable.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AliasAnalysis.h"
+#include "llvm/Analysis/CFG.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/Passes.h"

 #include <vector>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "machine-cse"
 
 STATISTIC(NumCoalesces, "Number of copies coalesced");
 STATISTIC(NumCSEs,      "Number of common subexpression eliminated");
+STATISTIC(NumPREs,      "Number of partial redundant expression \
+transformed to fully redundant");
 STATISTIC(NumPhysCSEs,
           "Number of physreg referencing common subexpr eliminated");
 STATISTIC(NumCrossBBCSEs,
           "Number of cross-MBB physreg referencing CS eliminated");
 STATISTIC(NumCommutes,  "Number of copies coalesced after commuting");
 
 namespace {
 
void getAnalysisUsage(AnalysisUsage &AU) const override {
       AU.addRequired<AAResultsWrapperPass>();
       AU.addPreservedID(MachineLoopInfoID);
       AU.addRequired<MachineDominatorTree>();
       AU.addPreserved<MachineDominatorTree>();
     }
 
     void releaseMemory() override {
       ScopeMap.clear();
+      PREMap.clear();
       Exps.clear();
     }
 
   private:
     using AllocatorTy = RecyclingAllocator<BumpPtrAllocator,
                             ScopedHashTableVal<MachineInstr *, unsigned>>;
     using ScopedHTType =
         ScopedHashTable<MachineInstr *, unsigned, MachineInstrExpressionTrait,
                         AllocatorTy>;
     using ScopeType = ScopedHTType::ScopeTy;
 
     unsigned LookAheadLimit = 0;
     DenseMap<MachineBasicBlock *, ScopeType *> ScopeMap;
+    DenseMap<MachineInstr *, MachineBasicBlock *, MachineInstrExpressionTrait> PREMap;
     ScopedHTType VNT;
     SmallVector<MachineInstr *, 64> Exps;
     unsigned CurrVN = 0;
 
     bool PerformTrivialCopyPropagation(MachineInstr *MI,
                                        MachineBasicBlock *MBB);
     bool isPhysDefTriviallyDead(unsigned Reg,
                                 MachineBasicBlock::const_iterator I,
private:
     bool isProfitableToCSE(unsigned CSReg, unsigned Reg,
                            MachineInstr *CSMI, MachineInstr *MI);
     void EnterScope(MachineBasicBlock *MBB);
     void ExitScope(MachineBasicBlock *MBB);
     bool ProcessBlock(MachineBasicBlock *MBB);
     void ExitScopeIfDone(MachineDomTreeNode *Node,
                          DenseMap<MachineDomTreeNode*, unsigned> &OpenChildren);
     bool PerformCSE(MachineDomTreeNode *Node);
+
+    bool isPRECandidate(MachineInstr *MI);
+    bool ProcessBlockPRE(MachineDominatorTree *MDT, MachineBasicBlock *MBB);
+    bool PerformSimplePRE(MachineDominatorTree *DT);
   };
 
 } // end anonymous namespace
 
 char MachineCSE::ID = 0;
 
 char &llvm::MachineCSEID = MachineCSE::ID;
 
for (MachineDomTreeNode *Node : Scopes) {
     Changed |= ProcessBlock(MBB);
     // If it's a leaf node, it's done. Traverse upwards to pop ancestors.
     ExitScopeIfDone(Node, OpenChildren);
   }
 
   return Changed;
 }
 
+// Stronger conditions are used for PREed instrs rather than for CSE ones
+bool MachineCSE::isPRECandidate(MachineInstr *MI) {
+  if (!isCSECandidate(MI) ||
+      MI->isTransient() ||
+      MI->isBranch() ||
+      MI->isBundled() ||
+      MI->isNotDuplicable() ||
+      MI->isAsCheapAsAMove() ||
+      MI->isBarrier() ||
+      MI->isReturn() ||
+      MI->getNumDefs() != 1 ||
+      MI->getNumExplicitDefs() != 1)
+    return false;
+
+  for (auto def: MI->defs())
+    if (!TRI->isVirtualRegister(def.getReg()))
+      return false;
+
+  for (auto use: MI->uses())
+    if (use.isReg() && !TRI->isVirtualRegister(use.getReg()))
+      return false;
+
+  return true;
+}
+
+bool MachineCSE::ProcessBlockPRE(MachineDominatorTree *DT, MachineBasicBlock *MBB) {
+  bool Changed = false;
+  for (MachineBasicBlock::iterator I = MBB->begin(), E = MBB->end(); I != E; ) {
+    MachineInstr *MI = &*I;
+    ++I;
+
+    if (!isPRECandidate(MI))
+      continue;
+
+    if (!PREMap.count(MI)) {
+      PREMap[MI] = MBB;
+    } else {
+      auto MBB1 = PREMap[MI];
+      assert(!DT->properlyDominates(MBB, MBB1) &&
+        "MBB cannot properly dominate MBB1 while DFS through dominators tree!");
+      auto CMBB = DT->findNearestCommonDominator(MBB, MBB1);
+
+      // Two instrs are partial redundant if their basic blocks are reachable
+      // from one to another but one doesn't dominate another.
+      if (CMBB != MBB1) {
+        auto BB = MBB->getBasicBlock(), BB1 = MBB1->getBasicBlock();
+        if (BB != nullptr && BB1 != nullptr &&
+            (isPotentiallyReachable(BB1, BB) ||
+             isPotentiallyReachable(BB, BB1))) {
+
+          assert(MI->getOperand(0).isDef() &&
+                 "First operand of instr with one explicit def must be this def");
+          MachineInstr &NewMI = TII->duplicate(*CMBB, CMBB->getFirstTerminator(), *MI);
+          unsigned VReg = MI->getOperand(0).getReg();
+          unsigned NewReg = MRI->cloneVirtualRegister(VReg);
+          NewMI.getOperand(0).setReg(NewReg);
+
+          if (!isProfitableToCSE(VReg, NewReg, MI, &NewMI)) {
+            NewMI.eraseFromParent();
+            continue;
+          }
+
+          PREMap[MI] = CMBB;
+          ++NumPREs;
+          Changed = true;
+        }
+      }
+    }
+  }
+  return Changed;
+}
+
+// This simple PRE (partial redundancy elimination) pass doesn't actually
+// eliminate partial redundancy but transforms it to full redundancy,
+// anticipating that the next CSE step will eliminate this created redundancy.
+// If CSE doesn't eliminate this, than created instruction will remain dead
+// and eliminated later by Remove Dead Machine Instructions pass.
+bool MachineCSE::PerformSimplePRE(MachineDominatorTree *DT) {
+  SmallVector<MachineDomTreeNode*, 32> BBs;
+
+  PREMap.clear();
+  bool Changed = false;
+  BBs.push_back(DT->getRootNode());
+  do {
+    auto Node = BBs.pop_back_val();
+    const std::vector<MachineDomTreeNode*> &Children = Node->getChildren();
+    for (MachineDomTreeNode *Child : Children)
+      BBs.push_back(Child);
+
+    MachineBasicBlock *MBB = Node->getBlock();
+    Changed |= ProcessBlockPRE(DT, MBB);
+
+  } while (!BBs.empty());
+
+  return Changed;
+}
+
 bool MachineCSE::runOnMachineFunction(MachineFunction &MF) {
   if (skipFunction(MF.getFunction()))
     return false;
 
   TII = MF.getSubtarget().getInstrInfo();
   TRI = MF.getSubtarget().getRegisterInfo();
   MRI = &MF.getRegInfo();
   AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
   DT = &getAnalysis<MachineDominatorTree>();
   LookAheadLimit = TII->getMachineCSELookAheadLimit();
-  return PerformCSE(DT->getRootNode());
+  bool ChangedPRE, ChangedCSE;
+  ChangedPRE = PerformSimplePRE(DT);
+  ChangedCSE = PerformCSE(DT->getRootNode());
+  return ChangedPRE || ChangedCSE;
 }

test/CodeGen/Mips/internalfunc.ll

 if.then:                                          ; preds = %entry
   %tmp1 = load void (...)*, void (...)** @caller.sf1, align 4
   tail call void (...) %tmp1() nounwind
   br label %if.end
 
 if.end:                                           ; preds = %entry, %if.then
 ; CHECK: lw  $[[R2:[0-9]+]], %got(sf2)
 ; CHECK: addiu ${{[0-9]+}}, $[[R2]], %lo(sf2)
-; CHECK: lw  $[[R3:[0-9]+]], %got(caller.sf1)
-; CHECK: sw  ${{[0-9]+}}, %lo(caller.sf1)($[[R3]])
+; CHECK: sw  ${{[0-9]+}}, %lo(caller.sf1)($[[R1]])
   %tobool3 = icmp ne i32 %a0, 0
   %tmp4 = load void (...)*, void (...)** @gf1, align 4
   %cond = select i1 %tobool3, void (...)* %tmp4, void (...)* bitcast (void ()* @sf2 to void (...)*)
   store void (...)* %cond, void (...)** @caller.sf1, align 4
   ret void
 }
 
 define internal void @sf2() nounwind {

test/CodeGen/X86/avx2-masked-gather.ll

 ; NOGATHER-NEXT:    vpextrb $4, %xmm1, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB4_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
 ; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
 ; NOGATHER-NEXT:    vpinsrd $1, (%rax), %xmm2, %xmm2
 ; NOGATHER-NEXT:  .LBB4_4: # %else2
 ; NOGATHER-NEXT:    vpextrb $8, %xmm1, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB4_6
 ; NOGATHER-NEXT:  # %bb.5: # %cond.load4
-; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; NOGATHER-NEXT:    vmovq %xmm3, %rax
+; NOGATHER-NEXT:    vmovq %xmm0, %rax
 ; NOGATHER-NEXT:    vpinsrd $2, (%rax), %xmm2, %xmm2
 ; NOGATHER-NEXT:  .LBB4_6: # %else5
 ; NOGATHER-NEXT:    vpextrb $12, %xmm1, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB4_8
 ; NOGATHER-NEXT:  # %bb.7: # %cond.load7
-; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
 ; NOGATHER-NEXT:    vpinsrd $3, (%rax), %xmm2, %xmm2
 ; NOGATHER-NEXT:  .LBB4_8: # %else8
 ; NOGATHER-NEXT:    vmovdqa %xmm2, %xmm0
 ; NOGATHER-NEXT:    vzeroupper
 ; NOGATHER-NEXT:    retq
 entry:
   %res = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 0, <4 x i1> %masks, <4 x i32> %passthro)

 ; NOGATHER-NEXT:    vpextrb $4, %xmm1, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB5_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
 ; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; NOGATHER-NEXT:  .LBB5_4: # %else2
 ; NOGATHER-NEXT:    vpextrb $8, %xmm1, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB5_6
 ; NOGATHER-NEXT:  # %bb.5: # %cond.load4
-; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; NOGATHER-NEXT:    vmovq %xmm3, %rax
+; NOGATHER-NEXT:    vmovq %xmm0, %rax
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; NOGATHER-NEXT:  .LBB5_6: # %else5
 ; NOGATHER-NEXT:    vpextrb $12, %xmm1, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB5_8
 ; NOGATHER-NEXT:  # %bb.7: # %cond.load7
-; NOGATHER-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; NOGATHER-NEXT:  .LBB5_8: # %else8
 ; NOGATHER-NEXT:    vmovaps %xmm2, %xmm0
 ; NOGATHER-NEXT:    vzeroupper
 ; NOGATHER-NEXT:    retq
 entry:
   %res = call <4 x float> @llvm.masked.gather.v4float(<4 x float*> %ptrs, i32 0, <4 x i1> %masks, <4 x float> %passthro)

 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
 ; NOGATHER-NEXT:    vpextrq $1, %xmm3, %rax
 ; NOGATHER-NEXT:    vpinsrd $1, (%rax), %xmm1, %xmm4
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm4[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB6_4: # %else2
 ; NOGATHER-NEXT:    vpextrb $4, %xmm0, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm3
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_6
 ; NOGATHER-NEXT:  # %bb.5: # %cond.load4
-; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm4
-; NOGATHER-NEXT:    vmovq %xmm4, %rax
+; NOGATHER-NEXT:    vmovq %xmm3, %rax
 ; NOGATHER-NEXT:    vpinsrd $2, (%rax), %xmm1, %xmm4
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm4[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB6_6: # %else5
 ; NOGATHER-NEXT:    vpextrb $6, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_8
 ; NOGATHER-NEXT:  # %bb.7: # %cond.load7
-; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm3
 ; NOGATHER-NEXT:    vpextrq $1, %xmm3, %rax
 ; NOGATHER-NEXT:    vpinsrd $3, (%rax), %xmm1, %xmm3
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm3[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB6_8: # %else8
 ; NOGATHER-NEXT:    vpextrb $8, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_10
 ; NOGATHER-NEXT:  # %bb.9: # %cond.load10
 ; NOGATHER-NEXT:    vmovq %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vpinsrd $0, (%rax), %xmm3, %xmm3
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB6_10: # %else11
 ; NOGATHER-NEXT:    vpextrb $10, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_12
 ; NOGATHER-NEXT:  # %bb.11: # %cond.load13
 ; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vpinsrd $1, (%rax), %xmm3, %xmm3
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB6_12: # %else14
 ; NOGATHER-NEXT:    vpextrb $12, %xmm0, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm2
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_14
 ; NOGATHER-NEXT:  # %bb.13: # %cond.load16
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm3
-; NOGATHER-NEXT:    vmovq %xmm3, %rax
+; NOGATHER-NEXT:    vmovq %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vpinsrd $2, (%rax), %xmm3, %xmm3
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB6_14: # %else17
 ; NOGATHER-NEXT:    vpextrb $14, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB6_16
 ; NOGATHER-NEXT:  # %bb.15: # %cond.load19
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
-; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
+; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
 ; NOGATHER-NEXT:    vpinsrd $3, (%rax), %xmm0, %xmm0
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB6_16: # %else20
 ; NOGATHER-NEXT:    vmovaps %ymm1, %ymm0
 ; NOGATHER-NEXT:    retq
 entry:
   %ld  = load <8 x i32*>, <8 x i32*>* %ptr

 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
 ; NOGATHER-NEXT:    vpextrq $1, %xmm3, %rax
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm4 = xmm1[0],mem[0],xmm1[2,3]
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm4[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB7_4: # %else2
 ; NOGATHER-NEXT:    vpextrb $4, %xmm0, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm3
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_6
 ; NOGATHER-NEXT:  # %bb.5: # %cond.load4
-; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm4
-; NOGATHER-NEXT:    vmovq %xmm4, %rax
+; NOGATHER-NEXT:    vmovq %xmm3, %rax
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm4 = xmm1[0,1],mem[0],xmm1[3]
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm4[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB7_6: # %else5
 ; NOGATHER-NEXT:    vpextrb $6, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_8
 ; NOGATHER-NEXT:  # %bb.7: # %cond.load7
-; NOGATHER-NEXT:    vextractf128 $1, %ymm3, %xmm3
 ; NOGATHER-NEXT:    vpextrq $1, %xmm3, %rax
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm3 = xmm1[0,1,2],mem[0]
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm3[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB7_8: # %else8
 ; NOGATHER-NEXT:    vpextrb $8, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_10
 ; NOGATHER-NEXT:  # %bb.9: # %cond.load10
 ; NOGATHER-NEXT:    vmovq %xmm2, %rax
 ; NOGATHER-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm4
 ; NOGATHER-NEXT:    vblendps {{.*#+}} xmm3 = xmm3[0],xmm4[1,2,3]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB7_10: # %else11
 ; NOGATHER-NEXT:    vpextrb $10, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_12
 ; NOGATHER-NEXT:  # %bb.11: # %cond.load13
 ; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm3 = xmm3[0],mem[0],xmm3[2,3]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB7_12: # %else14
 ; NOGATHER-NEXT:    vpextrb $12, %xmm0, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm2
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_14
 ; NOGATHER-NEXT:  # %bb.13: # %cond.load16
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm3
-; NOGATHER-NEXT:    vmovq %xmm3, %rax
+; NOGATHER-NEXT:    vmovq %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm3 = xmm3[0,1],mem[0],xmm3[3]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB7_14: # %else17
 ; NOGATHER-NEXT:    vpextrb $14, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB7_16
 ; NOGATHER-NEXT:  # %bb.15: # %cond.load19
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
-; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
+; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
 ; NOGATHER-NEXT:    vinsertps {{.*#+}} xmm0 = xmm0[0,1,2],mem[0]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB7_16: # %else20
 ; NOGATHER-NEXT:    vmovaps %ymm1, %ymm0
 ; NOGATHER-NEXT:    retq
 entry:
   %ld  = load <8 x float*>, <8 x float*>* %ptr

 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB8_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
 ; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vpinsrq $1, (%rax), %xmm1, %xmm3
 ; NOGATHER-NEXT:    vblendps {{.*#+}} ymm1 = ymm3[0,1,2,3],ymm1[4,5,6,7]
 ; NOGATHER-NEXT:  .LBB8_4: # %else2
 ; NOGATHER-NEXT:    vpextrb $8, %xmm0, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm2
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB8_6
 ; NOGATHER-NEXT:  # %bb.5: # %cond.load4
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm3
-; NOGATHER-NEXT:    vmovq %xmm3, %rax
+; NOGATHER-NEXT:    vmovq %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vpinsrq $0, (%rax), %xmm3, %xmm3
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB8_6: # %else5
 ; NOGATHER-NEXT:    vpextrb $12, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB8_8
 ; NOGATHER-NEXT:  # %bb.7: # %cond.load7
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
-; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
+; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
 ; NOGATHER-NEXT:    vpinsrq $1, (%rax), %xmm0, %xmm0
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB8_8: # %else8
 ; NOGATHER-NEXT:    vmovaps %ymm1, %ymm0
 ; NOGATHER-NEXT:    retq
 entry:
   %ld  = load <4 x i64*>, <4 x i64*>* %ptr

 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB9_4
 ; NOGATHER-NEXT:  # %bb.3: # %cond.load1
 ; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vmovhpd {{.*#+}} xmm3 = xmm1[0],mem[0]
 ; NOGATHER-NEXT:    vblendpd {{.*#+}} ymm1 = ymm3[0,1],ymm1[2,3]
 ; NOGATHER-NEXT:  .LBB9_4: # %else2
 ; NOGATHER-NEXT:    vpextrb $8, %xmm0, %eax
+; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm2
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB9_6
 ; NOGATHER-NEXT:  # %bb.5: # %cond.load4
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm3
-; NOGATHER-NEXT:    vmovq %xmm3, %rax
+; NOGATHER-NEXT:    vmovq %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm3
 ; NOGATHER-NEXT:    vmovlpd {{.*#+}} xmm3 = mem[0],xmm3[1]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB9_6: # %else5
 ; NOGATHER-NEXT:    vpextrb $12, %xmm0, %eax
 ; NOGATHER-NEXT:    testb $1, %al
 ; NOGATHER-NEXT:    je .LBB9_8
 ; NOGATHER-NEXT:  # %bb.7: # %cond.load7
-; NOGATHER-NEXT:    vextractf128 $1, %ymm2, %xmm0
-; NOGATHER-NEXT:    vpextrq $1, %xmm0, %rax
+; NOGATHER-NEXT:    vpextrq $1, %xmm2, %rax
 ; NOGATHER-NEXT:    vextractf128 $1, %ymm1, %xmm0
 ; NOGATHER-NEXT:    vmovhpd {{.*#+}} xmm0 = xmm0[0],mem[0]
 ; NOGATHER-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm1
 ; NOGATHER-NEXT:  .LBB9_8: # %else8
 ; NOGATHER-NEXT:    vmovapd %ymm1, %ymm0
 ; NOGATHER-NEXT:    retq
 entry:
   %ld  = load <4 x double*>, <4 x double*>* %ptr

test/CodeGen/X86/masked_gather.ll

 ; AVX1-NEXT:  # %bb.3: # %cond.load1
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; AVX1-NEXT:  .LBB0_4: # %else2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpextrb $8, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB0_6
 ; AVX1-NEXT:  # %bb.5: # %cond.load4
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vmovq %xmm3, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; AVX1-NEXT:  .LBB0_6: # %else5
 ; AVX1-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB0_8
 ; AVX1-NEXT:  # %bb.7: # %cond.load7
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; AVX1-NEXT:  .LBB0_8: # %else8
 ; AVX1-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: gather_v4f32_ptr_v4i32:

 ; AVX2-NEXT:  # %bb.3: # %cond.load1
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; AVX2-NEXT:  .LBB0_4: # %else2
 ; AVX2-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX2-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX2-NEXT:    vpextrb $8, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    je .LBB0_6
 ; AVX2-NEXT:  # %bb.5: # %cond.load4
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
-; AVX2-NEXT:    vmovq %xmm3, %rax
+; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; AVX2-NEXT:  .LBB0_6: # %else5
 ; AVX2-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB0_8
 ; AVX2-NEXT:  # %bb.7: # %cond.load7
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; AVX2-NEXT:  .LBB0_8: # %else8
 ; AVX2-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: gather_v4f32_ptr_v4i32:

 ; AVX1-NEXT:  # %bb.3: # %cond.load1
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; AVX1-NEXT:  .LBB1_4: # %else2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpextrb $8, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB1_6
 ; AVX1-NEXT:  # %bb.5: # %cond.load4
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vmovq %xmm3, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; AVX1-NEXT:  .LBB1_6: # %else5
 ; AVX1-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB1_8
 ; AVX1-NEXT:  # %bb.7: # %cond.load7
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; AVX1-NEXT:  .LBB1_8: # %else8
 ; AVX1-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: gather_v4f32_v4i32_v4i32:

 ; AVX2-NEXT:  # %bb.3: # %cond.load1
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; AVX2-NEXT:  .LBB1_4: # %else2
 ; AVX2-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX2-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX2-NEXT:    vpextrb $8, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    je .LBB1_6
 ; AVX2-NEXT:  # %bb.5: # %cond.load4
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
-; AVX2-NEXT:    vmovq %xmm3, %rax
+; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; AVX2-NEXT:  .LBB1_6: # %else5
 ; AVX2-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB1_8
 ; AVX2-NEXT:  # %bb.7: # %cond.load7
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; AVX2-NEXT:  .LBB1_8: # %else8
 ; AVX2-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: gather_v4f32_v4i32_v4i32:

 ; AVX1-NEXT:  # %bb.3: # %cond.load1
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; AVX1-NEXT:  .LBB2_4: # %else2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpextrb $8, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB2_6
 ; AVX1-NEXT:  # %bb.5: # %cond.load4
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
-; AVX1-NEXT:    vmovq %xmm3, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; AVX1-NEXT:  .LBB2_6: # %else5
 ; AVX1-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB2_8
 ; AVX1-NEXT:  # %bb.7: # %cond.load7
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; AVX1-NEXT:  .LBB2_8: # %else8
 ; AVX1-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: gather_v4f32_v4i64_v4i32:

 ; AVX2-NEXT:  # %bb.3: # %cond.load1
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0],mem[0],xmm2[2,3]
 ; AVX2-NEXT:  .LBB2_4: # %else2
 ; AVX2-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX2-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX2-NEXT:    vpextrb $8, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    je .LBB2_6
 ; AVX2-NEXT:  # %bb.5: # %cond.load4
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
-; AVX2-NEXT:    vmovq %xmm3, %rax
+; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1],mem[0],xmm2[3]
 ; AVX2-NEXT:  .LBB2_6: # %else5
 ; AVX2-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB2_8
 ; AVX2-NEXT:  # %bb.7: # %cond.load7
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vinsertps {{.*#+}} xmm2 = xmm2[0,1,2],mem[0]
 ; AVX2-NEXT:  .LBB2_8: # %else8
 ; AVX2-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: gather_v4f32_v4i64_v4i32:

 ; AVX1-NEXT:    vpextrb $1, %xmm6, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_4
 ; AVX1-NEXT:  # %bb.3: # %cond.load1
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $1, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_4: # %else2
 ; AVX1-NEXT:    vpmovsxdq %xmm7, %xmm6
-; AVX1-NEXT:    vpaddq %xmm5, %xmm4, %xmm8
+; AVX1-NEXT:    vpaddq %xmm5, %xmm4, %xmm5
 ; AVX1-NEXT:    vpxor %xmm7, %xmm7, %xmm7
 ; AVX1-NEXT:    vpcmpeqb %xmm7, %xmm2, %xmm7
 ; AVX1-NEXT:    vpextrb $2, %xmm7, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB3_6
 ; AVX1-NEXT:  # %bb.5: # %cond.load4
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
-; AVX1-NEXT:    vmovq %xmm5, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $2, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_6: # %else5
 ; AVX1-NEXT:    vpaddq %xmm6, %xmm4, %xmm6
 ; AVX1-NEXT:    vpextrb $3, %xmm7, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_8
 ; AVX1-NEXT:  # %bb.7: # %cond.load7
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $3, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_8: # %else8
-; AVX1-NEXT:    vinsertf128 $1, %xmm6, %ymm8, %ymm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm6, %ymm5, %ymm0
 ; AVX1-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX1-NEXT:    vpcmpeqb %xmm5, %xmm2, %xmm5
 ; AVX1-NEXT:    vpextrb $4, %xmm5, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_10
 ; AVX1-NEXT:  # %bb.9: # %cond.load10
 ; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $4, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_10: # %else11
 ; AVX1-NEXT:    vpshufd {{.*#+}} xmm6 = xmm1[2,3,0,1]
 ; AVX1-NEXT:    vpmovsxdq %xmm1, %xmm7
 ; AVX1-NEXT:    vpextrb $5, %xmm5, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_12
 ; AVX1-NEXT:  # %bb.11: # %cond.load13
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $5, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_12: # %else14
 ; AVX1-NEXT:    vpmovsxdq %xmm6, %xmm6
-; AVX1-NEXT:    vpaddq %xmm7, %xmm4, %xmm8
+; AVX1-NEXT:    vpaddq %xmm7, %xmm4, %xmm5
 ; AVX1-NEXT:    vpxor %xmm7, %xmm7, %xmm7
 ; AVX1-NEXT:    vpcmpeqb %xmm7, %xmm2, %xmm7
 ; AVX1-NEXT:    vpextrb $6, %xmm7, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB3_14
 ; AVX1-NEXT:  # %bb.13: # %cond.load16
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
-; AVX1-NEXT:    vmovq %xmm5, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $6, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_14: # %else17
 ; AVX1-NEXT:    vpaddq %xmm6, %xmm4, %xmm6
 ; AVX1-NEXT:    vpextrb $7, %xmm7, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_16
 ; AVX1-NEXT:  # %bb.15: # %cond.load19
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $7, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_16: # %else20
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm1
-; AVX1-NEXT:    vinsertf128 $1, %xmm6, %ymm8, %ymm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm6, %ymm5, %ymm0
 ; AVX1-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX1-NEXT:    vpcmpeqb %xmm5, %xmm2, %xmm5
 ; AVX1-NEXT:    vpextrb $8, %xmm5, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_18
 ; AVX1-NEXT:  # %bb.17: # %cond.load22
 ; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $8, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_18: # %else23
 ; AVX1-NEXT:    vpshufd {{.*#+}} xmm6 = xmm1[2,3,0,1]
 ; AVX1-NEXT:    vpmovsxdq %xmm1, %xmm1
 ; AVX1-NEXT:    vpextrb $9, %xmm5, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_20
 ; AVX1-NEXT:  # %bb.19: # %cond.load25
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $9, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_20: # %else26
 ; AVX1-NEXT:    vpmovsxdq %xmm6, %xmm5
 ; AVX1-NEXT:    vpaddq %xmm1, %xmm4, %xmm1
 ; AVX1-NEXT:    vpxor %xmm6, %xmm6, %xmm6
 ; AVX1-NEXT:    vpcmpeqb %xmm6, %xmm2, %xmm6
 ; AVX1-NEXT:    vpextrb $10, %xmm6, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB3_22
 ; AVX1-NEXT:  # %bb.21: # %cond.load28
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm7
-; AVX1-NEXT:    vmovq %xmm7, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $10, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_22: # %else29
 ; AVX1-NEXT:    vpaddq %xmm5, %xmm4, %xmm4
 ; AVX1-NEXT:    vpextrb $11, %xmm6, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_24
 ; AVX1-NEXT:  # %bb.23: # %cond.load31
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $11, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_24: # %else32
 ; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm1, %ymm0
 ; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqb %xmm1, %xmm2, %xmm1
 ; AVX1-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_26
 ; AVX1-NEXT:  # %bb.25: # %cond.load34
 ; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $12, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_26: # %else35
 ; AVX1-NEXT:    vpextrb $13, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_28
 ; AVX1-NEXT:  # %bb.27: # %cond.load37
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $13, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_28: # %else38
 ; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqb %xmm1, %xmm2, %xmm1
 ; AVX1-NEXT:    vpextrb $14, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    je .LBB3_30
 ; AVX1-NEXT:  # %bb.29: # %cond.load40
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm2
-; AVX1-NEXT:    vmovq %xmm2, %rax
+; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $14, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_30: # %else41
 ; AVX1-NEXT:    vpextrb $15, %xmm1, %eax
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    je .LBB3_32
 ; AVX1-NEXT:  # %bb.31: # %cond.load43
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm0
 ; AVX1-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX1-NEXT:    vpinsrb $15, (%rax), %xmm3, %xmm3
 ; AVX1-NEXT:  .LBB3_32: # %else44
 ; AVX1-NEXT:    vmovdqa %xmm3, %xmm0
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: gather_v16i8_v16i32_v16i8:

 ; AVX2-NEXT:    vpextrq $1, %xmm5, %rax
 ; AVX2-NEXT:    vpinsrb $1, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_4: # %else2
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpxor %xmm6, %xmm6, %xmm6
 ; AVX2-NEXT:    vpcmpeqb %xmm6, %xmm2, %xmm6
 ; AVX2-NEXT:    vpextrb $2, %xmm6, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm5, %xmm5
 ; AVX2-NEXT:    je .LBB3_6
 ; AVX2-NEXT:  # %bb.5: # %cond.load4
-; AVX2-NEXT:    vextracti128 $1, %ymm5, %xmm7
-; AVX2-NEXT:    vmovq %xmm7, %rax
+; AVX2-NEXT:    vmovq %xmm5, %rax
 ; AVX2-NEXT:    vpinsrb $2, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_6: # %else5
 ; AVX2-NEXT:    vpmovsxdq %xmm0, %ymm0
 ; AVX2-NEXT:    vpextrb $3, %xmm6, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_8
 ; AVX2-NEXT:  # %bb.7: # %cond.load7
-; AVX2-NEXT:    vextracti128 $1, %ymm5, %xmm5
 ; AVX2-NEXT:    vpextrq $1, %xmm5, %rax
 ; AVX2-NEXT:    vpinsrb $3, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_8: # %else8
 ; AVX2-NEXT:    vpaddq %ymm0, %ymm4, %ymm0
 ; AVX2-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX2-NEXT:    vpcmpeqb %xmm5, %xmm2, %xmm5
 ; AVX2-NEXT:    vpextrb $4, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_10
 ; AVX2-NEXT:  # %bb.9: # %cond.load10
 ; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $4, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_10: # %else11
 ; AVX2-NEXT:    vpextrb $5, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_12
 ; AVX2-NEXT:  # %bb.11: # %cond.load13
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $5, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_12: # %else14
 ; AVX2-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX2-NEXT:    vpcmpeqb %xmm5, %xmm2, %xmm5
 ; AVX2-NEXT:    vpextrb $6, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    je .LBB3_14
 ; AVX2-NEXT:  # %bb.13: # %cond.load16
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm6
-; AVX2-NEXT:    vmovq %xmm6, %rax
+; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $6, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_14: # %else17
 ; AVX2-NEXT:    vpmovsxdq %xmm1, %ymm6
 ; AVX2-NEXT:    vpextrb $7, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_16
 ; AVX2-NEXT:  # %bb.15: # %cond.load19
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $7, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_16: # %else20
 ; AVX2-NEXT:    vpaddq %ymm6, %ymm4, %ymm0
 ; AVX2-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX2-NEXT:    vpcmpeqb %xmm5, %xmm2, %xmm5
 ; AVX2-NEXT:    vpextrb $8, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al

 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $9, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_20: # %else26
 ; AVX2-NEXT:    vextracti128 $1, %ymm1, %xmm1
 ; AVX2-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX2-NEXT:    vpcmpeqb %xmm5, %xmm2, %xmm5
 ; AVX2-NEXT:    vpextrb $10, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    je .LBB3_22
 ; AVX2-NEXT:  # %bb.21: # %cond.load28
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm6
-; AVX2-NEXT:    vmovq %xmm6, %rax
+; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $10, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_22: # %else29
 ; AVX2-NEXT:    vpmovsxdq %xmm1, %ymm1
 ; AVX2-NEXT:    vpextrb $11, %xmm5, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_24
 ; AVX2-NEXT:  # %bb.23: # %cond.load31
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $11, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_24: # %else32
 ; AVX2-NEXT:    vpaddq %ymm1, %ymm4, %ymm0
 ; AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; AVX2-NEXT:    vpcmpeqb %xmm1, %xmm2, %xmm1
 ; AVX2-NEXT:    vpextrb $12, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_26
 ; AVX2-NEXT:  # %bb.25: # %cond.load34
 ; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $12, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_26: # %else35
 ; AVX2-NEXT:    vpextrb $13, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_28
 ; AVX2-NEXT:  # %bb.27: # %cond.load37
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $13, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_28: # %else38
 ; AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; AVX2-NEXT:    vpcmpeqb %xmm1, %xmm2, %xmm1
 ; AVX2-NEXT:    vpextrb $14, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    je .LBB3_30
 ; AVX2-NEXT:  # %bb.29: # %cond.load40
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm2
-; AVX2-NEXT:    vmovq %xmm2, %rax
+; AVX2-NEXT:    vmovq %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $14, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_30: # %else41
 ; AVX2-NEXT:    vpextrb $15, %xmm1, %eax
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    je .LBB3_32
 ; AVX2-NEXT:  # %bb.31: # %cond.load43
-; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm0
 ; AVX2-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX2-NEXT:    vpinsrb $15, (%rax), %xmm3, %xmm3
 ; AVX2-NEXT:  .LBB3_32: # %else44
 ; AVX2-NEXT:    vmovdqa %xmm3, %xmm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: gather_v16i8_v16i32_v16i8:

 ; AVX512-NEXT:  .LBB3_8: # %else8
 ; AVX512-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX512-NEXT:    vpcmpeqb %xmm5, %xmm1, %xmm5
 ; AVX512-NEXT:    vpmovsxbd %xmm5, %zmm5
 ; AVX512-NEXT:    vptestmd %zmm5, %zmm5, %k0
 ; AVX512-NEXT:    kshiftrw $4, %k0, %k1
 ; AVX512-NEXT:    kmovw %k1, %eax
 ; AVX512-NEXT:    testb $1, %al
+; AVX512-NEXT:    vextracti32x4 $2, %zmm4, %xmm5
 ; AVX512-NEXT:    je .LBB3_10
 ; AVX512-NEXT:  # %bb.9: # %cond.load10
-; AVX512-NEXT:    vextracti32x4 $2, %zmm4, %xmm5
 ; AVX512-NEXT:    vmovq %xmm5, %rax
 ; AVX512-NEXT:    vpinsrb $4, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_10: # %else11
 ; AVX512-NEXT:    kshiftrw $5, %k0, %k0
 ; AVX512-NEXT:    kmovw %k0, %eax
 ; AVX512-NEXT:    testb $1, %al
 ; AVX512-NEXT:    je .LBB3_12
 ; AVX512-NEXT:  # %bb.11: # %cond.load13
-; AVX512-NEXT:    vextracti32x4 $2, %zmm4, %xmm5
 ; AVX512-NEXT:    vpextrq $1, %xmm5, %rax
 ; AVX512-NEXT:    vpinsrb $5, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_12: # %else14
 ; AVX512-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
 ; AVX512-NEXT:    vpxor %xmm5, %xmm5, %xmm5
 ; AVX512-NEXT:    vpcmpeqb %xmm5, %xmm1, %xmm5
 ; AVX512-NEXT:    vpmovsxbd %xmm5, %zmm5
 ; AVX512-NEXT:    vptestmd %zmm5, %zmm5, %k0
 ; AVX512-NEXT:    kshiftrw $6, %k0, %k1
 ; AVX512-NEXT:    kmovw %k1, %eax
 ; AVX512-NEXT:    testb $1, %al
+; AVX512-NEXT:    vextracti32x4 $3, %zmm4, %xmm4
 ; AVX512-NEXT:    je .LBB3_14
 ; AVX512-NEXT:  # %bb.13: # %cond.load16
-; AVX512-NEXT:    vextracti32x4 $3, %zmm4, %xmm5
-; AVX512-NEXT:    vmovq %xmm5, %rax
+; AVX512-NEXT:    vmovq %xmm4, %rax
 ; AVX512-NEXT:    vpinsrb $6, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_14: # %else17
 ; AVX512-NEXT:    vpmovsxdq %ymm0, %zmm0
 ; AVX512-NEXT:    kshiftrw $7, %k0, %k0
 ; AVX512-NEXT:    kmovw %k0, %eax
 ; AVX512-NEXT:    testb $1, %al
 ; AVX512-NEXT:    je .LBB3_16
 ; AVX512-NEXT:  # %bb.15: # %cond.load19
-; AVX512-NEXT:    vextracti32x4 $3, %zmm4, %xmm4
 ; AVX512-NEXT:    vpextrq $1, %xmm4, %rax
 ; AVX512-NEXT:    vpinsrb $7, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_16: # %else20
 ; AVX512-NEXT:    vpaddq %zmm0, %zmm3, %zmm0
 ; AVX512-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX512-NEXT:    vpcmpeqb %xmm3, %xmm1, %xmm3
 ; AVX512-NEXT:    vpmovsxbd %xmm3, %zmm3
 ; AVX512-NEXT:    vptestmd %zmm3, %zmm3, %k0

 ; AVX512-NEXT:  .LBB3_24: # %else32
 ; AVX512-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX512-NEXT:    vpcmpeqb %xmm3, %xmm1, %xmm3
 ; AVX512-NEXT:    vpmovsxbd %xmm3, %zmm3
 ; AVX512-NEXT:    vptestmd %zmm3, %zmm3, %k0
 ; AVX512-NEXT:    kshiftrw $12, %k0, %k1
 ; AVX512-NEXT:    kmovw %k1, %eax
 ; AVX512-NEXT:    testb $1, %al
+; AVX512-NEXT:    vextracti32x4 $2, %zmm0, %xmm3
 ; AVX512-NEXT:    je .LBB3_26
 ; AVX512-NEXT:  # %bb.25: # %cond.load34
-; AVX512-NEXT:    vextracti32x4 $2, %zmm0, %xmm3
 ; AVX512-NEXT:    vmovq %xmm3, %rax
 ; AVX512-NEXT:    vpinsrb $12, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_26: # %else35
 ; AVX512-NEXT:    kshiftrw $13, %k0, %k0
 ; AVX512-NEXT:    kmovw %k0, %eax
 ; AVX512-NEXT:    testb $1, %al
 ; AVX512-NEXT:    je .LBB3_28
 ; AVX512-NEXT:  # %bb.27: # %cond.load37
-; AVX512-NEXT:    vextracti32x4 $2, %zmm0, %xmm3
 ; AVX512-NEXT:    vpextrq $1, %xmm3, %rax
 ; AVX512-NEXT:    vpinsrb $13, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_28: # %else38
 ; AVX512-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX512-NEXT:    vpcmpeqb %xmm3, %xmm1, %xmm1
 ; AVX512-NEXT:    vpmovsxbd %xmm1, %zmm1
 ; AVX512-NEXT:    vptestmd %zmm1, %zmm1, %k0
 ; AVX512-NEXT:    kshiftrw $14, %k0, %k1
 ; AVX512-NEXT:    kmovw %k1, %eax
 ; AVX512-NEXT:    testb $1, %al
+; AVX512-NEXT:    vextracti32x4 $3, %zmm0, %xmm0
 ; AVX512-NEXT:    je .LBB3_30
 ; AVX512-NEXT:  # %bb.29: # %cond.load40
-; AVX512-NEXT:    vextracti32x4 $3, %zmm0, %xmm1
-; AVX512-NEXT:    vmovq %xmm1, %rax
+; AVX512-NEXT:    vmovq %xmm0, %rax
 ; AVX512-NEXT:    vpinsrb $14, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_30: # %else41
 ; AVX512-NEXT:    kshiftrw $15, %k0, %k0
 ; AVX512-NEXT:    kmovw %k0, %eax
 ; AVX512-NEXT:    testb $1, %al
 ; AVX512-NEXT:    je .LBB3_32
 ; AVX512-NEXT:  # %bb.31: # %cond.load43
-; AVX512-NEXT:    vextracti32x4 $3, %zmm0, %xmm0
 ; AVX512-NEXT:    vpextrq $1, %xmm0, %rax
 ; AVX512-NEXT:    vpinsrb $15, (%rax), %xmm2, %xmm2
 ; AVX512-NEXT:  .LBB3_32: # %else44
 ; AVX512-NEXT:    vmovdqa %xmm2, %xmm0
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
   %vptr0 = insertelement <16 x i8*> undef, i8* %base, i32 0
   %vptr1 = shufflevector <16 x i8*> %vptr0, <16 x i8*> undef, <16 x i32> zeroinitializer