[SLP] Fix for PR30787: Failure to beneficially vectorize 'copyable' elements in integer binary ops.

[dtemirbulatov]

Summary

Patch tries to improve vectorization of the following code:

void add1(int * __restrict dst, const int * __restrict src) {
  *dst++ = *src++;
  *dst++ = *src++ + 1;
  *dst++ = *src++ + 2;
  *dst++ = *src++ + 3;
}

Currently this code cannot be vectorized because the very first operation is not a binary add, but just a load.

[dtemirbulatov]

test/Transforms/SLPVectorizer/X86/internal-dep.ll

10 ↗

(On Diff #131752)

ok, Thanks

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

4645

probably level 2 or 3 dependencies might be ok since it is not encapsulated in a single operation.

[RKSimon]

What is happening with this patch? It's been in development for over a year now and still seems to be having problems. Getting PR30787 fixed would be VERY useful and I'm thinking we should be looking at alternatives if this patch is going to carry on stalling/reverting.

[dtemirbulatov]

I will update the solution shortly, I am currently in changing scheduler in order to avoid non-alternative opcodes in a bundle.

[dtemirbulatov]

Removed internal vector dependency check as incorrect and I added the dependency check in case of partial bundle vectorization with non-alternative operations only. At BoUpSLP::vectorizeTree, BoUpSLP::buildTree.

[ABataev]

test/Transforms/SLPVectorizer/SystemZ/pr34619.ll

2 ↗

(On Diff #140208)

Commit this test as a separate NFC patch with the checks against trunk

test/Transforms/SLPVectorizer/X86/partail.ll

2 ↗

(On Diff #140208)

Commit this test as a separate NFC patch with the checks against trunk

test/Transforms/SLPVectorizer/X86/pr35497.ll

2

Commit this test as a separate NFC patch with the checks against trunk

test/Transforms/SLPVectorizer/X86/resched.ll

2 ↗

(On Diff #140208)

Commit this test as a separate NFC patch with the checks against trunk

[dtemirbulatov]

Update after tests commit, formatting, delete Instruction::ExtractElement restricktion from tryToRepresentAsInstArg.

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1603

1. Remove extra parens around Entry->State.IsNonAlt
2. Why are you skipping bundles with non alternarive opcodes only?

3165–3169

Rewrite it this way:

SmallPtrSet<Instruction *, 4> BundleInst;
Bundle = Bundle->FirstInBundle;
LastInst = Bundle->Inst;
while (Bundle) {
  BundleInst.insert(Bundle->Inst);
  Bundle = Bundle->NextInBundle;
}

3170–3172

Why do you need to scan all the instructions in the basic block starting from First? Why you can't use only scheduled instructions?

3173

Move ++Iter to the header of for loop

4026

Name of the variable must start from capital letter.

4029

1. The code is not formatted
2. Seems to me you missed break;

[dtemirbulatov]

Update after Alexey's remarks.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1603

yes, we don't need to check Entry->State.IsNonAlt here, that is similar to getTreeEntry(U, Scalar) == Entry.

3165–3169

Done. Thanks.

3170–3172

What do you mean? please elaborate. If you mean ScheduleData here then it is also not always gives up correct sequence of scheduled instructions in a block. Like, bundle member with NextInBundle equals to null is not guaranty to be the last instruction of this bundle among other instructions. If we start with First then it is highly likely that we could iterate across all bundle members and exit instead of iterating to the end of the basic block. Also, I am thinking, to avoid this overhead we could note the last scheduled instruction in BB during scheduling in scheduleBlock() and keep this information in ScheduleData structure.

3173

Done.

4026

Done.

4029

Correct, Thanks.

[dtemirbulatov]

Rebased, improved complexity of setInsertPointAfterBundle() for already scheduled instructions, minor changes in scheduleBlock()

[ABataev]

Also, your code seems not quite formatted. Please, use clang-format on your changes to format it properly.

lib/Transforms/Vectorize/SLPVectorizer.cpp

285

Enclose it into braces

287

This too

427

bool IsNonAlt = llvm::one_of(VL, [Opcode, AltOpcode](Value *V) {return isa<Instruction>(V) && !sameOpcodeOrAlt(Opcode, AltOpcode, cast<Instruction>(V)->getOpcode());});

1661

Add the debug message here

3166

Why you can't put bundles in the list in the right order: from the very first instruction to the very last?

4028–4035

Better to do it this way:

if (llvm::any_of(Scalar->users(), [this, Entry, Scalar](User *U){return !getTreeEntry(U) && getTreeEntry(U, Scalar) == Entry;}))
  continue;

4641

Remove this, it is not needed

4649

pickedInst->PickedInst

4655–4657

Remove it, does not do anything

[dtemirbulatov]

Update after remarks from Alexey Bataev.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

I could do this in scheduleBlock() function with a queue, but that could add additional complexity.

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

Could you explain why?

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

One instruction could belong to one or more separate bundles... and while we try to change order in bundles at scheduleBlock() we have to update ScheduleDataMap, ExtraScheduleDataMap.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

I mean pseudo operation could occur in more than one bundle.

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

But these schedule bundles must have different scheduling region id and they must be in a different bundles, why their order changes?

[dtemirbulatov]

Implement post scheduling bundle reorder of the last element of the bundle according how it was scheduled.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

The bundle is differerent, but scheduling region id is the same.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

I mean, for example, for this function:
define void @add0(i32* noalias %dst, i32* noalias %src) {
entry:

%incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
%0 = load i32, i32* %src, align 4
%add = add nsw i32 %0, 1
%incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
store i32 %add, i32* %dst, align 4
%incdec.ptr2 = getelementptr inbounds i32, i32* %src, i64 2
%1 = load i32, i32* %incdec.ptr, align 4
%add3 = add nsw i32 %1, 1
%incdec.ptr4 = getelementptr inbounds i32, i32* %dst, i64 2
store i32 %add3, i32* %incdec.ptr1, align 4
%incdec.ptr5 = getelementptr inbounds i32, i32* %src, i64 3
%2 = load i32, i32* %incdec.ptr2, align 4
%add6 = add nsw i32 %2, 2
%incdec.ptr7 = getelementptr inbounds i32, i32* %dst, i64 3
store i32 %add6, i32* %incdec.ptr4, align 4
%3 = load i32, i32* %incdec.ptr5, align 4
%add9 = add nsw i32 %3, 3
store i32 %add9, i32* %incdec.ptr7, align 4
ret void

}

We have two bundles:
[ %3 = load i32, i32* %src, align 4; %add3 = add nsw i32 %2, 1; %add6 = add nsw i32 %1, 2; %add9 = add nsw i32 %0, 3]
and
[ %3 = load i32, i32* %src, align 4; %2 = load i32, i32* %incdec.ptr, align 4; %1 = load i32, i32* %incdec.ptr2, align 4; %0 = load i32, i32* %incdec.ptr5, align 4]
with the same instruction %3 = load i32, i32* %src, align 4 and one is a pseudo instruction in this bundle [ %3 = load i32, i32* %src, align 4; %add3; %add6; %add9]
and all in the same scheduling region id that equal to 1.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

why their order changes?

sometimes we have to reschedule a pseudo instruction first in both bundles in order to form correct dependencies.

[RKSimon]

@dtemirbulatov @ABataev Is there anything that can be done to please keep this patch moving? The next release branch won't be far away now and I originally requested this over a year and a half ago (PR30787), and ideally I'd like to get this in and then I can more easily implement PR33744 as well in time.

I notice there's still some minor refactoring in the patch to use InstructionState in newTreeEntry/tryScheduleBundle calls - is it worth getting that done quickly first?

What about testing - how much testing has been done with external code?

[dtemirbulatov]

In D28907#1120098, @RKSimon wrote:

@dtemirbulatov @ABataev Is there anything that can be done to please keep this patch moving? The next release branch won't be far away now and I originally requested this over a year and a half ago (PR30787), and ideally I'd like to get this in and then I can more easily implement PR33744 as well in time.

I think that algorithm at 4512 is incorrect, we need to copy common instructions for bundles, instead of reordering bundles. Alexey?

[ABataev]

In D28907#1120098, @RKSimon wrote:

@dtemirbulatov @ABataev Is there anything that can be done to please keep this patch moving? The next release branch won't be far away now and I originally requested this over a year and a half ago (PR30787), and ideally I'd like to get this in and then I can more easily implement PR33744 as well in time.

I notice there's still some minor refactoring in the patch to use InstructionState in newTreeEntry/tryScheduleBundle calls - is it worth getting that done quickly first?

What about testing - how much testing has been done with external code?

Hi guys, sorry, but I cannot review this patch currently as I'm on the vacation for 3 more weeks. It would be good if somebody else could look at this patch. Try to ask Ayal Zaks (@Ayal), maybe he could help you. As to the implementation itself, I don't like the way it is implemented now. Current scheduling implementation looks like a crutch or a hack. It is very complex, has some unclear logic that may be broken in many ways.

[ABataev]

Hi, when do you plan to update the path for the latest version? I have an idea of improving the scheduling model, maybe it will help us to resolve the problems with the patch.

[dtemirbulatov]

In D28907#1149639, @ABataev wrote:

Hi, when do you plan to update the path for the latest version? I have an idea of improving the scheduling model, maybe it will help us to resolve the problems with the patch.

Hi, I think I have one issue remaining with LNT's MultiSource/Benchmarks/Olden/bh/bh, I will rebased the current version today, tomorrow. Thanks.

[RKSimon]

@dtemirbulatov @ABataev I'm not sure how much this will cross over but I'm investigating how to extend the alternate opcode mechanism to work with non-binary instructions. Initially I'm just looking at cast + call operators (e.g. sitofp/uitofp, floor/ceil etc.) but even that requires moderate refactoring of the getEntryCost and vectorizeTree ShuffleVector handling. What I'm not sure of is whether we could extend this idea to accept any partial vectorization and the alternate becomes a pass through - what do you think?

If not there is scope to further simplify this patch, for instance @spatel's work in InstCombine for PR37806 should mean that you can rely on InstCombine to perform much of the work in getDefaultConstantForOpcode etc. and all SLP needs to do is create a "passthrough" SK_Select shuffle stage.

[spatel]

In D28907#1149688, @RKSimon wrote:

If not there is scope to further simplify this patch, for instance @spatel's work in InstCombine for PR37806 should mean that you can rely on InstCombine to perform much of the work in getDefaultConstantForOpcode etc. and all SLP needs to do is create a "passthrough" SK_Select shuffle stage.

For reference in D48830, I'm proposing that we use (and extend) ConstantExpr::getBinOpIdentity(). IIUC, that would be the same thing that is shown as getDefaultConstantForOpcode() here.

[ABataev]

In D28907#1149688, @RKSimon wrote:

@dtemirbulatov @ABataev I'm not sure how much this will cross over but I'm investigating how to extend the alternate opcode mechanism to work with non-binary instructions. Initially I'm just looking at cast + call operators (e.g. sitofp/uitofp, floor/ceil etc.) but even that requires moderate refactoring of the getEntryCost and vectorizeTree ShuffleVector handling. What I'm not sure of is whether we could extend this idea to accept any partial vectorization and the alternate becomes a pass through - what do you think?

If not there is scope to further simplify this patch, for instance @spatel's work in InstCombine for PR37806 should mean that you can rely on InstCombine to perform much of the work in getDefaultConstantForOpcode etc. and all SLP needs to do is create a "passthrough" SK_Select shuffle stage.

I think this is possible if I correctly understood your idea. But we need to make this patch land at first. To do so we need to resolve the problems with the scheduling.

[ABataev]

In D28907#1149688, @RKSimon wrote:

If not there is scope to further simplify this patch, for instance @spatel's work in InstCombine for PR37806 should mean that you can rely on InstCombine to perform much of the work in getDefaultConstantForOpcode etc. and all SLP needs to do is create a "passthrough" SK_Select shuffle stage.

Yes, this is also a possible solution. What we need to do in this case is to tweak the cost model + improve gathering algorithm. Yes, that might work.

[ABataev]

In D28907#1149700, @spatel wrote:

In D28907#1149688, @RKSimon wrote:

If not there is scope to further simplify this patch, for instance @spatel's work in InstCombine for PR37806 should mean that you can rely on InstCombine to perform much of the work in getDefaultConstantForOpcode etc. and all SLP needs to do is create a "passthrough" SK_Select shuffle stage.

For reference in D48830, I'm proposing that we use (and extend) ConstantExpr::getBinOpIdentity(). IIUC, that would be the same thing that is shown as getDefaultConstantForOpcode() here.

Yes, I see, we can try to use it. AT least we need to think about this solution, need to estimate all pros/cons here

[dtemirbulatov]

here is change before 74cd05c4a4f94a27daf2d3fc173e7213060cc47c commit, I am currently rebaseing the change.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

678

we don't need to do this.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

3166

I will move this solution to a dedicated function, so we don't have to measure distance here.

[RKSimon]

OK - please shout if there is anything I can do to help - I realise my alternate opcode refactoring has made rebasing this patch less straightforward!

[dtemirbulatov]

Rebase. Still, I need to implement bundle reordering instead of the change in setInsertPointAfterBundle(), fully tested this change. I will split this change in several independent reviews.

[dtemirbulatov]

Fixed one more issue with duplicating memory dependencies in calculateDependencies() by checking whether we already counted this particular dependency.

[RKSimon]

lib/Transforms/Vectorize/SLPVectorizer.cpp

530

InstructionState was keeping the Base/Alt instructions the same if AltOpcodeNum ==0 (BaseIndex ==AltIndex) - why are you inserting nulls?

1613

Can we use ConstantExpr::getBinOpIdentity instead?

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

530

yes, correct, Thanks.

1613

no, ConstantExpr::getBinOpIdentity does support only commutative operations.

[lebedev.ri]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1613

Are you sure?

Constant *ConstantExpr::getBinOpIdentity(unsigned Opcode, Type *Ty,
                                         bool AllowRHSConstant) {
...

  // Non-commutative opcodes: AllowRHSConstant must be set.
  if (!AllowRHSConstant)
    return nullptr;

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1613

oh, yes, sorry, it just a different behaviour here. we want for example 1 division and ConstantExpr::getBinOpIdentity would return 0.

[dtemirbulatov]

Rebase, Fixed RKSimon's remark, implemented bundle reordering function.

[dtemirbulatov]

Add hash code based indexing instead of instruction based, split ScheduleData into InstScheduleData, PseudoScheduleData, improve diagnostics of bundle scheduling.

[ABataev]

Why you don't want to use pair<Instruction*, Opcode> as the key in all maps/sets? I expect that it will lead to much more simpler slution.

lib/Transforms/Vectorize/SLPVectorizer.cpp

274

Instead, I would check for the supported instructions rather than for the unsupported ones.

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

452

Not sure if this always produces unique hashes, might lead to the incorrect compiler work.

[dtemirbulatov]

Implemented pair<Value*, Opcode> as the key in all maps/sets.
Fixed issue with incorrect memory dependency that is attached in testcase memory-dep.ll.
Allow Non-alterative operations to be stored in InstScheduleDataMap.
Removed IsNonAlt field out of InstructionsState.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

274

The other list is a bit long in my implementation, this one looks better.

[dtemirbulatov]

Update after I found another couple of errors after additional testing the change. Here are changes:
Removed OpValue field out of PseudoScheduleData.
Forbid any bundles with non-alternative operations and remainder operation, see rem-bundle.ll.
Fixed error in setInsertPointAfterBundle() function by using getScheduleData() instead of getInstrScheduleData and if a bundle member is present multiple bundles at the same time then walk through the bundle to find the last scheduled member of the bundle. see insert-after-multiple-bundle.ll
Restore MemoryDependencies to SmallVector, we don't have to count a member presents in calculateDependencies().

[dtemirbulatov]

Implemented Map<Instruction*, std::pair<Value *Parent, unsigned Opcode> indexing for ScalarToTreeEntry, PseudoInstScheduleDataMap.
Added reorderBundles() function to reorder bundles that have common instructions and common instructions were scheduled at least twice. We don't want to note which bundle was scheduled first. We could determine instruction layout after SLP scheduling.

[RKSimon]

lib/Transforms/Vectorize/SLPVectorizer.cpp

270

Repeated Instruction::SRem

[RKSimon]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1638

@spatel @dtemirbulatov Can we use getBinOpIdentity yet ?

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

274

I still think it is better to limit this with the list of supported operations rather than the list of the unsupported ones.

1177

Very strange that you still need particular scheduling class for the pseudo instructions, I think you can use the original data structure if you correctly implement the pseudo instruction itself. I still don't see all the changes we talked about.

[spatel]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1638

Yes - if anyone has suggestions for making that 'AllowRHSConstant' param clearer, let me know.

The only problem that I see is that this code is returning +0.0 as the default constant for an fadd (because the caller guarantees 'nsz'?).
I worked around something like that here:
rL346143

I can't tell if SLP would want to do something like that.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

270

oh, thanks. I missed that.

274

ok, I will change that.

1177

ok, I am implementing now.

1638

ok, yes, looks like it is going to work.

[dtemirbulatov]

Looks like I fixed all previous remarks also during testing I found two more issues with the change and I fixed both.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

4184

We have to clear all dependancy calculations since the pseudo instruction might use already calculated SD node with calculated dependency, look at scheduling_pseudo.ll testcase.

4637

Please check schedule-bundle1.ll testcase, without this change scheduling is not correct.

[dtemirbulatov]

Oops, I found a few typos, Formated tryToRepresentAsInstArg() and removed the second SRem from isRemainder().

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

270

Still 2 SRems

383

Restore the original function here.

438

You definitely need comments here

849–850

Instead of ArrayRef<Value *> I expected to see something like ArrayRef<InstructionOrPseudoInstruction>, where InstructionOrPseudoInstruction is the class that represents the instruction/Value itself, or the pseudo instruction.

872–876

Why do you need this?

884–894

Again, you need all this code because you did not implemented what we discussed. Try to use the InstructionOrPseudoInstruction like class to represent values/instructions and pseudo-instructions. It should be much easier to implement and a lot of changes will just go away.

[RKSimon]

@dtemirbulatov Any movement on this? It'd be great to get this in for the 8.0 release!

[dtemirbulatov]

In D28907#1318422, @RKSimon wrote:

@dtemirbulatov Any movement on this? It'd be great to get this in for the 8.0 release!

yes, I just refactored getTreeEntry and ... according to ABataev request, I will update after additional testing in a day or two.

[dtemirbulatov]

Introduced InstructionOrPseudo structure and removed "Instruction::Invoke" out of tryToRepresentAsInstArg() function with error example in invoke.ll testcase.

[lebedev.ri]

test/Transforms/SLPVectorizer/X86/invoke.ll

3

Unless this currently crashes, precommit?

[dtemirbulatov]

test/Transforms/SLPVectorizer/X86/invoke.ll

3

yes, thanks, I will remove this line.

[dtemirbulatov]

test/Transforms/SLPVectorizer/X86/invoke.ll

3

Oh, No crash currently on pre-commit, I will remove the whole test.

[lebedev.ri]

test/Transforms/SLPVectorizer/X86/invoke.ll

3

What i have meant to say is, unless this test file currently causes a crash, it would be best to commit it now, thus the diff will show the diff, and not a whole new test file.

[dtemirbulatov]

test/Transforms/SLPVectorizer/X86/invoke.ll

3

Ok

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

334

Turn this to a class. Also, not enough comments, the design of the structure also should be improved. Currently, it is too complex. Maybe base class + templates will help.

394

Please, follow the coding standard in the new code. Also, some of the functions can be replaced by some standard functions, just like in this case it can be replaced by llvm::all_of

410

I think, the pseudo instruction is always can be considered as the one with the same block, because you can put it easily into the current block

427

I think this function also should accept ArrayRef<InstructionOrPseudo *> as an input param

801

Left and Right must be SmallVectorImpl<InstructionOrPseudo *> &?

845

Restore the original

884

Also, seems to me this must be an InstructionOrPseudoOp

1012

Why do you need this new function? No comments and explanation.

1076

Pseudo instruction must be just ignored?

1188

Again, I think it must be InstructionOrPseudoOp

1264

Again, InstructionOrPseudoOp

1324

If you implement InstructionOrPseudoOp correctly, this reorder stuff should not be required, I think

1345

Why you cannot store everything in a single map?

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

well, There should be several(>=2) independent scheduling events(one for real instruction and other for pseudos) and there is just one real instruction, in the end, I don't see how it could be done without reordering or tracking the last scheduled instance for the same instruction. We could introduce something like IsLastScheduled field in ScheduleData struct, but it would be quite similar to reordering.

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

If you add the real instruction instead of this pseudoinstruction, will you need all these scheduling events? No. Will you need to do some extra reordering etc.? No. Why you cannot simulate it with the new class/structure?

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

do you mean that the last one scheduling becomes the real one and we just ignore any pseudos?

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

do you mean that the last one scheduled becomes the real one and we just ignore any pseudos?

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

No. I mean, that if you insert the real instructions instead of those pseudo-instructions, you won't need all that reordering/new scheduling etc. Why can't you mimic this behavior with the pseudo-instruction?

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

hmm, There are at least NextLoadStore dependancies that we break if we, for example, insert real Load instruction somewhere. Or with could recalculate NextLoadStore. Or maybe mimic pseudo in some another way.

[ABataev]

lib/Transforms/Vectorize/SLPVectorizer.cpp

1324

I don't think that they can be broken as we're not going to insert new Loads/Stores, just some binops. SO, the loads/stores and the the corresponding dependencies should not be affected.

[dtemirbulatov]

removed bundle reordering by replacing pseudo instructions with real ones.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

334

I see this class as just one, BTW it is a container.

410

I have this functionality, but let us for now minimize functionality change here, I will follow this change right after that check-in.

801

That change further increases the size of this review, we can change that later.

884

no, looks like "Instruction" here is more convenient.

[RKSimon]

reopening - phab seems to be a bit broken

[xbolva00]

Reping

[dtemirbulatov]

In D28907#2278838, @xbolva00 wrote:

Reping

In process of rebasing and simplifying proposed change.

[SjoerdMeijer]

Hello, this work was brought to my attention in D102748. Would be good to get this in, and seems almost finished too. Any plans to pick this up? Otherwise I think I would be interested in doing that.

[ABataev]

In D28907#2770982, @SjoerdMeijer wrote:

Hello, this work was brought to my attention in D102748. Would be good to get this in, and seems almost finished too. Any plans to pick this up? Otherwise I think I would be interested in doing that.

This should be much easier to implement after non-power-2 in SLP support.

[SjoerdMeijer]

In D28907#2770984, @ABataev wrote:

In D28907#2770982, @SjoerdMeijer wrote:

Hello, this work was brought to my attention in D102748. Would be good to get this in, and seems almost finished too. Any plans to pick this up? Otherwise I think I would be interested in doing that.

This should be much easier to implement after non-power-2 in SLP support.

Could you elaborate a bit more on this just for my understanding? E.g., what is this exactly, and is this an ongoing effort in the SLP vectoriser?

[ABataev]

In D28907#2770989, @SjoerdMeijer wrote:

In D28907#2770984, @ABataev wrote:

In D28907#2770982, @SjoerdMeijer wrote:

Hello, this work was brought to my attention in D102748. Would be good to get this in, and seems almost finished too. Any plans to pick this up? Otherwise I think I would be interested in doing that.

This should be much easier to implement after non-power-2 in SLP support.

Could you elaborate a bit more on this just for my understanding? E.g., what is this exactly, and is this an ongoing effort in the SLP vectoriser?

Here is D57059. I'm splitting it into smaller patches and committing them step-by-step.

[SjoerdMeijer]

Thanks, interesting, having a look there!

[xbolva00]

In D28907#2770984, @ABataev wrote:

In D28907#2770982, @SjoerdMeijer wrote:

Hello, this work was brought to my attention in D102748. Would be good to get this in, and seems almost finished too. Any plans to pick this up? Otherwise I think I would be interested in doing that.

This should be much easier to implement after non-power-2 in SLP support.

Can you explain why?

x1 + 1
x2
x3 + 6
x4 + 8

How that patch will help you? You need to add “fake” noop addition “ + 0” for x2.

[ABataev]

In D28907#2775629, @xbolva00 wrote:

In D28907#2770984, @ABataev wrote:

In D28907#2770982, @SjoerdMeijer wrote:

Hello, this work was brought to my attention in D102748. Would be good to get this in, and seems almost finished too. Any plans to pick this up? Otherwise I think I would be interested in doing that.

This should be much easier to implement after non-power-2 in SLP support.

Can you explain why?

x1 + 1
x2
x3 + 6
x4 + 8

How that patch will help you? You need to add “fake” noop addition “ + 0” for x2.

It is pretty similar to non-pow-2 functionality, just instead of undefs need to use 0 for adds, 1 for mul etc. I can't provide more details currently but I'm going to revive it after non-pow-2 patch. I assume we won't need to change scheduling model.

[xbolva00]

Ah right:) I understand now.

lib/Transforms/Vectorize/SLPVectorizer.cpp

 /// meaningful vectorization path such as x86_fp80 and ppc_f128. This just
 /// avoids spending time checking the cost model and realizing that they will
 /// be inevitably scalarized.
 static bool isValidElementType(Type *Ty) {
   return VectorType::isValidElementType(Ty) && !Ty->isX86_FP80Ty() &&
          !Ty->isPPC_FP128Ty();
 }
 
-/// \returns true if all of the instructions in \p VL are in the same block or
-/// false otherwise.
-static bool allSameBlock(ArrayRef<Value *> VL) {
-  Instruction *I0 = dyn_cast<Instruction>(VL[0]);
-  if (!I0)
-    return false;
-  BasicBlock *BB = I0->getParent();
-  for (int i = 1, e = VL.size(); i < e; i++) {
-    Instruction *I = dyn_cast<Instruction>(VL[i]);
-    if (!I)
-      return false;
-
-    if (BB != I->getParent())
-      return false;
-  }
-  return true;
-}
-
-/// \returns True if all of the values in \p VL are constants.
-static bool allConstant(ArrayRef<Value *> VL) {
-  for (Value *i : VL)
-    if (!isa<Constant>(i))
-      return false;
-  return true;
-}
-
-/// \returns True if all of the values in \p VL are identical.
-static bool isSplat(ArrayRef<Value *> VL) {
-  for (unsigned i = 1, e = VL.size(); i < e; ++i)
-    if (VL[i] != VL[0])
-      return false;
-  return true;
-}
-
 /// Checks if the vector of instructions can be represented as a shuffle, like:
 /// %x0 = extractelement <4 x i8> %x, i32 0
 /// %x3 = extractelement <4 x i8> %x, i32 3
 /// %y1 = extractelement <4 x i8> %y, i32 1
 /// %y2 = extractelement <4 x i8> %y, i32 2
 /// %x0x0 = mul i8 %x0, %x0
 /// %x3x3 = mul i8 %x3, %x3
 /// %y1y1 = mul i8 %y1, %y1
isShuffle(ArrayRef<Value *> VL) {
   // If we're not crossing lanes in different vectors, consider it as blending.
   if (CommonShuffleMode == Select && Vec2)
     return TargetTransformInfo::SK_Select;
   // If Vec2 was never used, we have a permutation of a single vector, otherwise
   // we have permutation of 2 vectors.
   return Vec2 ? TargetTransformInfo::SK_PermuteTwoSrc
               : TargetTransformInfo::SK_PermuteSingleSrc;
 }
 
+static bool isRemainder(unsigned Opcode) {
+  return (Opcode == Instruction::URem ||
+          Opcode == Instruction::SRem || Opcode == Instruction::FRem);
+}
+
+/// Checks if the \p Opcode can be considered as an operand of a (possibly)
+/// binary operation \p I.
+/// \returns The code of the binary operation of instruction \p I if the
+/// instruction with \p Opcode can be considered as an operand of \p I with the
+/// default value.
+static unsigned tryToRepresentAsInstArg(unsigned Opcode, Instruction *I) {
+  if (Opcode != Instruction::PHI &&
+      Opcode != Instruction::Invoke &&
+      ((I->getType()->isIntegerTy() &&
+       (I->getOpcode() == Instruction::Add ||
+        I->getOpcode() == Instruction::And ||
+        I->getOpcode() == Instruction::AShr ||
+        I->getOpcode() == Instruction::BitCast ||
+        I->getOpcode() == Instruction::Call ||
+        I->getOpcode() == Instruction::ExtractElement ||
+        I->getOpcode() == Instruction::ExtractValue ||
+        I->getOpcode() == Instruction::ICmp ||
+        I->getOpcode() == Instruction::Load ||
+        I->getOpcode() == Instruction::LShr ||
+        I->getOpcode() == Instruction::Mul ||
+        I->getOpcode() == Instruction::Or ||
+        I->getOpcode() == Instruction::PtrToInt ||
+        I->getOpcode() == Instruction::SDiv ||
+        I->getOpcode() == Instruction::Select ||
+        I->getOpcode() == Instruction::SExt ||
+        I->getOpcode() == Instruction::Shl ||
+        I->getOpcode() == Instruction::Sub ||
+        I->getOpcode() == Instruction::Trunc ||
+        I->getOpcode() == Instruction::UDiv ||
+        I->getOpcode() == Instruction::Xor ||
+        I->getOpcode() == Instruction::ZExt)) ||
+      (isa<FPMathOperator>(I) && cast<FPMathOperator>(I)->isFast())))
+    return I->getOpcode();
+  return 0;
+}
+
 namespace {
 
 /// Main data required for vectorization of instructions.
 struct InstructionsState {
   /// The very first instruction in the list with the main opcode.
   Value *OpValue = nullptr;
 
   /// The main/alternate instruction.
   Instruction *MainOp = nullptr;
   Instruction *AltOp = nullptr;
 
   /// The main/alternate opcodes for the list of instructions.
   unsigned getOpcode() const {
     return MainOp ? MainOp->getOpcode() : 0;
   }
 
   unsigned getAltOpcode() const {
     return AltOp ? AltOp->getOpcode() : 0;
   }
 
-  /// Some of the instructions in the list have alternate opcodes.
-  bool isAltShuffle() const { return getOpcode() != getAltOpcode(); }
+  InstructionsState() = default;
+  InstructionsState(Value *OpValue, Instruction *MainOp, Instruction *AltOp)
+      : OpValue(OpValue), MainOp(MainOp), AltOp(AltOp) {}
+};
+
+struct InstructionOrPseudo {
 
-  bool isOpcodeOrAlt(Instruction *I) const {
-    unsigned CheckedOpcode = I->getOpcode();
-    return getOpcode() == CheckedOpcode || getAltOpcode() == CheckedOpcode;
+  Value *Op;
+  Value *Parent;
+  Instruction *OpInst;
+  Instruction *MainOp;
+  Instruction *AltOp;
+  unsigned MainOpcode = 0;
+  unsigned AltOpcode = 0;
+
+  InstructionOrPseudo(Value *Op, Value *Parent,
+                      const InstructionsState &S): Op(Op), Parent(Parent),
+                      OpInst(dyn_cast<Instruction>(Op)), MainOp(S.MainOp),
+                      AltOp(S.AltOp)
+  {
+    if (MainOp)
+      MainOpcode = MainOp->getOpcode();
+    if (AltOp)
+      AltOpcode = AltOp->getOpcode();
+  };
+
+  bool isPseudo() {
+    if (!OpInst)
+      return false;
+    unsigned Opcode = OpInst->getOpcode();
+    return ((Opcode != MainOpcode) && (Opcode != AltOpcode));
+  }
+
+  bool isAltShuffle() const { return (MainOpcode != 0 && AltOpcode != 0 &&
+                                      MainOpcode != AltOpcode); }
+
+  bool isOpcodeOrAlt() const {
+    unsigned CheckedOpcode = OpInst->getOpcode();
+    return MainOpcode == CheckedOpcode || AltOpcode == CheckedOpcode;
+  }
+
+  std::pair<Value *, unsigned> getKey() const {
+    assert(Parent && "Incorrect parent!");
+    return std::make_pair(Parent, MainOpcode);
   }
 
-  InstructionsState() = delete;
-  InstructionsState(Value *OpValue, Instruction *MainOp, Instruction *AltOp)
-      : OpValue(OpValue), MainOp(MainOp), AltOp(AltOp) {}
 };
 
 } // end anonymous namespace
 
 /// Chooses the correct key for scheduling data. If \p Op has the same (or
 /// alternate) opcode as \p OpValue, the key is \p Op. Otherwise the key is \p
 /// OpValue.
 static Value *isOneOf(const InstructionsState &S, Value *Op) {
   auto *I = dyn_cast<Instruction>(Op);
-  if (I && S.isOpcodeOrAlt(I))
+  if (I) {
+    unsigned CheckedOpcode = I->getOpcode();
+    if (S.getOpcode() == CheckedOpcode || S.getAltOpcode() == CheckedOpcode)
       return Op;
+  }
   return S.OpValue;
 }
 
+/// \returns True if all of the values in \p IL are constants.
+static bool allConstant(ArrayRef<InstructionOrPseudo *> IL) {
+  for (auto *i : IL)
+    if (!isa<Constant>(i->Op))
+      return false;
+  return true;
+}
+
+/// \returns True if all of the values in \p IL are identical.
+static bool isSplat(ArrayRef<InstructionOrPseudo *> IL) {
+  for (unsigned i = 1, e = IL.size(); i < e; ++i)
+    if (IL[i]->Op != IL[0]->Op)
+      return false;
+  return true;
+}
+
+/// \returns true if all of the instructions in \p VL are in the same block or
+/// false otherwise.
+static bool allSameBlock(ArrayRef<InstructionOrPseudo *> IL) {
+  if (!IL[0]->OpInst)
+    return false;
+  BasicBlock *BB = IL[0]->OpInst->getParent();
+  for (int i = 1, e = IL.size(); i < e; i++) {
+    if (!IL[i]->OpInst)
+      return false;
+
+    if (BB != IL[i]->OpInst->getParent())
+      return false;
+  }
+  return true;
+}
+
 /// \returns analysis of the Instructions in \p VL described in
 /// InstructionsState, the Opcode that we suppose the whole list
 /// could be vectorized even if its structure is diverse.
 static InstructionsState getSameOpcode(ArrayRef<Value *> VL,
                                        unsigned BaseIndex = 0) {
   // Make sure these are all Instructions.
   if (llvm::any_of(VL, [](Value *V) { return !isa<Instruction>(V); }))
     return InstructionsState(VL[BaseIndex], nullptr, nullptr);
 
+  unsigned Opcode = cast<Instruction>(VL[BaseIndex])->getOpcode();
   bool IsCastOp = isa<CastInst>(VL[BaseIndex]);
   bool IsBinOp = isa<BinaryOperator>(VL[BaseIndex]);
-  unsigned Opcode = cast<Instruction>(VL[BaseIndex])->getOpcode();
+  bool IsNonAlt = false;
   unsigned AltOpcode = Opcode;
+  // Number of Instructions in VL with Main opcode.
+  unsigned OpcodeNum = 0;
+  // Number of Instructions in VL with Alternative opcode.
+  unsigned AltOpcodeNum = 0;
+  // Number of Instructions in VL with Non-Alternative opcode.
+  unsigned NonAltNum = 0;
+  // Non-Alternative operation offset in VL.
+  unsigned NonAltIndex = 0;
   unsigned AltIndex = BaseIndex;
 
-  // Check for one alternate opcode from another BinaryOperator.
-  // TODO - generalize to support all operators (types, calls etc.).
+  // Check for an alternate opcode pattern.
   for (int Cnt = 0, E = VL.size(); Cnt < E; Cnt++) {
-    unsigned InstOpcode = cast<Instruction>(VL[Cnt])->getOpcode();
-    if (IsBinOp && isa<BinaryOperator>(VL[Cnt])) {
-      if (InstOpcode == Opcode || InstOpcode == AltOpcode)
+    auto *I = cast<Instruction>(VL[Cnt]);
+    unsigned InstOpcode = I->getOpcode();
+    if (IsCastOp && isa<CastInst>(VL[Cnt])) {
+      Type *Ty0 = cast<Instruction>(VL[BaseIndex])->getOperand(0)->getType();
+      Type *Ty1 = cast<Instruction>(VL[Cnt])->getOperand(0)->getType();
+      if (Ty0 == Ty1) {
+        if (InstOpcode == Opcode) {
+          OpcodeNum++;
           continue;
+        }
+        if (AltOpcode != Opcode && InstOpcode == AltOpcode) {
+          AltOpcodeNum++;
+          continue;
+        }
         if (Opcode == AltOpcode) {
           AltOpcode = InstOpcode;
           AltIndex = Cnt;
+          AltOpcodeNum++;
           continue;
         }
-    } else if (IsCastOp && isa<CastInst>(VL[Cnt])) {
-      Type *Ty0 = cast<Instruction>(VL[BaseIndex])->getOperand(0)->getType();
-      Type *Ty1 = cast<Instruction>(VL[Cnt])->getOperand(0)->getType();
-      if (Ty0 == Ty1) {
-        if (InstOpcode == Opcode || InstOpcode == AltOpcode)
-          continue;
-        if (Opcode == AltOpcode) {
-          AltOpcode = InstOpcode;
-          AltIndex = Cnt;
-          continue;
-        }
+      }
+      return InstructionsState(VL[BaseIndex], nullptr, nullptr);
+    }
+    if (InstOpcode == Opcode) {
+      OpcodeNum++;
+    }
+    if (AltOpcode != Opcode && InstOpcode == AltOpcode) {
+      AltOpcodeNum++;
+      continue;
+    }
+    if (InstOpcode != Opcode && InstOpcode != AltOpcode) {
+      if (IsBinOp && AltOpcode == Opcode && isa<BinaryOperator>(I)) {
+        AltOpcodeNum++;
+      if (tryToRepresentAsInstArg(Opcode, I) ||
+           (IsBinOp &&
+            tryToRepresentAsInstArg(InstOpcode,
+                                    cast<Instruction>(VL[BaseIndex])))) {
+        if (!IsNonAlt) {
+          NonAltIndex = Cnt;
+          IsNonAlt = true;
         }
-    } else if (InstOpcode == Opcode || InstOpcode == AltOpcode)
+        NonAltNum++;
         continue;
+      }
       return InstructionsState(VL[BaseIndex], nullptr, nullptr);
     }
+  }
+
+  if (IsNonAlt && VL.size() > 2 && (OpcodeNum + AltOpcodeNum) <= NonAltNum) {
+    BaseIndex = NonAltIndex;
+    AltIndex = BaseIndex;
+    Opcode = cast<Instruction>(VL[BaseIndex])->getOpcode();
+    AltOpcode = Opcode;
+    IsBinOp = isa<BinaryOperator>(VL[BaseIndex]);
+    for (int Cnt = 0, E = VL.size(); Cnt < E; Cnt++) {
+      auto *I = cast<Instruction>(VL[Cnt]);
+      unsigned InstOpcode = I->getOpcode();
+      if (Opcode == AltOpcode && IsBinOp && isa<BinaryOperator>(I)) {
+        AltOpcode = InstOpcode;
+        AltIndex = Cnt;
+      }
+    }
+  }
+
+  if (IsNonAlt && (!IsBinOp ||
+      isRemainder(Opcode) ||
+      isRemainder(AltOpcode)))
+    return InstructionsState(VL[BaseIndex], nullptr, nullptr);
 
   return InstructionsState(VL[BaseIndex], cast<Instruction>(VL[BaseIndex]),
                            cast<Instruction>(VL[AltIndex]));
 }
 
 /// \returns true if all of the values in \p VL have the same type or false
 /// otherwise.
 static bool allSameType(ArrayRef<Value *> VL) {
   Type *Ty = VL[0]->getType();
   for (int i = 1, e = VL.size(); i < e; i++)
     if (VL[i]->getType() != Ty)
       return false;
 
   return true;
 }
 
 /// \returns True if Extract{Value,Element} instruction extracts element Idx.
 static Optional<unsigned> getExtractIndex(Instruction *E) {
   unsigned Opcode = E->getOpcode();
   assert((Opcode == Instruction::ExtractElement ||
           Opcode == Instruction::ExtractValue) &&
          "Expected extractelement or extractvalue instruction.");
   if (Opcode == Instruction::ExtractElement) {
     auto *CI = dyn_cast<ConstantInt>(E->getOperand(1));
     if (!CI)
       return None;
     return CI->getZExtValue();
   }
   ExtractValueInst *EI = cast<ExtractValueInst>(E);
   if (EI->getNumIndices() != 1)
     return None;
   return *EI->idx_begin();
 }
 

 
 /// Bottom Up SLP Vectorizer.
 class BoUpSLP {
 public:
   using ValueList = SmallVector<Value *, 8>;
   using InstrList = SmallVector<Instruction *, 16>;
   using ValueSet = SmallPtrSet<Value *, 16>;
   using StoreList = SmallVector<StoreInst *, 8>;
+  using IOPList = SmallVector<InstructionOrPseudo *, 8>;
   using ExtraValueToDebugLocsMap =
       MapVector<Value *, SmallVector<Instruction *, 2>>;
 
   BoUpSLP(Function *Func, ScalarEvolution *Se, TargetTransformInfo *Tti,
           TargetLibraryInfo *TLi, AliasAnalysis *Aa, LoopInfo *Li,
           DominatorTree *Dt, AssumptionCache *AC, DemandedBits *DB,
           const DataLayout *DL, OptimizationRemarkEmitter *ORE)
       : F(Func), SE(Se), TTI(Tti), TLI(TLi), AA(Aa), LI(Li), DT(Dt), AC(AC),
public:
   /// into account (anf updating it, if required) list of externally used
   /// values stored in \p ExternallyUsedValues.
   void buildTree(ArrayRef<Value *> Roots,
                  ExtraValueToDebugLocsMap &ExternallyUsedValues,
                  ArrayRef<Value *> UserIgnoreLst = None);
 
   /// Clear the internal data structures that are created by 'buildTree'.
   void deleteTree() {
+    InstructionOrPseudoArray.clear();
     VectorizableTree.clear();
     ScalarToTreeEntry.clear();
     MustGather.clear();
     ExternalUses.clear();
     NumOpsWantToKeepOrder.clear();
     NumOpsWantToKeepOriginalOrder = 0;
     for (auto &Iter : BlocksSchedules) {
       BlockScheduling *BS = Iter.second.get();
       BS->clear();
     }
     MinBWs.clear();
private:
 
   /// Checks if all users of \p I are the part of the vectorization tree.
   bool areAllUsersVectorized(Instruction *I) const;
 
   /// \returns the cost of the vectorizable entry.
   int getEntryCost(TreeEntry *E);
 
   /// This is the recursive part of buildTree.
-  void buildTree_rec(ArrayRef<Value *> Roots, unsigned Depth, int);
+  void buildTree_rec(Value *Parent, ArrayRef<Value *> Roots, unsigned Depth,
+                     int);
 
   /// \returns true if the ExtractElement/ExtractValue instructions in \p VL can
   /// be vectorized to use the original vector (or aggregate "bitcast" to a
   /// vector) and sets \p CurrentOrder to the identity permutation; otherwise
   /// returns false, setting \p CurrentOrder to either an empty vector or a
   /// non-identity permutation that allows to reuse extract instructions.
-  bool canReuseExtract(ArrayRef<Value *> VL, Value *OpValue,
+  bool canReuseExtract(ArrayRef<InstructionOrPseudo *> IL, Value *OpValue,
                        SmallVectorImpl<unsigned> &CurrentOrder) const;
 
   /// Vectorize a single entry in the tree.
   Value *vectorizeTree(TreeEntry *E);
 
   /// Vectorize a single entry in the tree, starting in \p VL.
-  Value *vectorizeTree(ArrayRef<Value *> VL);
+  Value *vectorizeTree(ArrayRef<Value *> VL, Value *Parent);
 
   /// \returns the scalarization cost for this type. Scalarization in this
   /// context means the creation of vectors from a group of scalars.
   int getGatherCost(Type *Ty, const DenseSet<unsigned> &ShuffledIndices);
 
   /// \returns the scalarization cost for this list of values. Assuming that
   /// this subtree gets vectorized, we may need to extract the values from the
   /// roots. This method calculates the cost of extracting the values.
-  int getGatherCost(ArrayRef<Value *> VL);
+  int getGatherCost(ArrayRef<InstructionOrPseudo *> IL);
 
   /// Set the Builder insert point to one after the last instruction in
   /// the bundle
-  void setInsertPointAfterBundle(ArrayRef<Value *> VL,
-                                 const InstructionsState &S);
+  void setInsertPointAfterBundle(ArrayRef<InstructionOrPseudo *> IL);
 
   /// \returns a vector from a collection of scalars in \p VL.
-  Value *Gather(ArrayRef<Value *> VL, VectorType *Ty);
+  Value *Gather(ArrayRef<InstructionOrPseudo *> IL, VectorType *Ty);
 
   /// \returns whether the VectorizableTree is fully vectorizable and will
   /// be beneficial even the tree height is tiny.
   bool isFullyVectorizableTinyTree();
 
+  Instruction *propagateMetadata(Instruction *I,
+                                 ArrayRef<InstructionOrPseudo *> IL);
+
   /// \reorder commutative operands in alt shuffle if they result in
   ///  vectorized code.
-  void reorderAltShuffleOperands(const InstructionsState &S,
-                                 ArrayRef<Value *> VL,
+  void reorderAltShuffleOperands(ArrayRef<InstructionOrPseudo *> IL,
                                  SmallVectorImpl<Value *> &Left,
                                  SmallVectorImpl<Value *> &Right);
 
   /// \reorder commutative operands to get better probability of
   /// generating vectorized code.
-  void reorderInputsAccordingToOpcode(unsigned Opcode, ArrayRef<Value *> VL,
+  void reorderInputsAccordingToOpcode(unsigned Opcode,
+                                      ArrayRef<InstructionOrPseudo *> IL,
                                       SmallVectorImpl<Value *> &Left,
                                       SmallVectorImpl<Value *> &Right);
   struct TreeEntry {
     TreeEntry(std::vector<TreeEntry> &Container) : Container(Container) {}
 
     /// \returns true if the scalars in VL are equal to this entry.
-    bool isSame(ArrayRef<Value *> VL) const {
-      if (VL.size() == Scalars.size())
-        return std::equal(VL.begin(), VL.end(), Scalars.begin());
-      return VL.size() == ReuseShuffleIndices.size() &&
+    bool isSame(ArrayRef<InstructionOrPseudo *> IL) const {
+      if (IL.size() == Scalars.size())
+        return std::equal(IL.begin(), IL.end(), Scalars.begin(),
+               [this](InstructionOrPseudo *I1, InstructionOrPseudo *I2)
+                     { return I1->Op == I2->Op; });
+      return IL.size() == ReuseShuffleIndices.size() &&
              std::equal(
-                 VL.begin(), VL.end(), ReuseShuffleIndices.begin(),
-                 [this](Value *V, unsigned Idx) { return V == Scalars[Idx]; });
+                 IL.begin(), IL.end(), ReuseShuffleIndices.begin(),
+                 [this](InstructionOrPseudo *I1, unsigned Idx)
+                 { return I1->Op == Scalars[Idx]->Op; });
     }
 
     /// A vector of scalars.
-    ValueList Scalars;
+    IOPList Scalars;
 
     /// The Scalars are vectorized into this value. It is initialized to Null.
     Value *VectorizedValue = nullptr;
 
     /// Do we need to gather this sequence ?
     bool NeedToGather = false;
 
     /// Does this sequence require some shuffling?
     SmallVector<unsigned, 4> ReuseShuffleIndices;
 
     /// Does this entry require reordering?
     ArrayRef<unsigned> ReorderIndices;
 
     /// Points back to the VectorizableTree.
     ///
     /// Only used for Graphviz right now.  Unfortunately GraphTrait::NodeRef has
     /// to be a pointer and needs to be able to initialize the child iterator.
     /// Thus we need a reference back to the container to translate the indices
     /// to entries.
     std::vector<TreeEntry> &Container;
 
     /// The TreeEntry index containing the user of this entry.  We can actually
     /// have multiple users so the data structure is not truly a tree.
     SmallVector<int, 1> UserTreeIndices;
+
   };
 
   /// Create a new VectorizableTree entry.
-  void newTreeEntry(ArrayRef<Value *> VL, bool Vectorized, int &UserTreeIdx,
+  void newTreeEntry(ArrayRef<InstructionOrPseudo *> IL, bool Vectorized,
+                    int &UserTreeIdx,
                     ArrayRef<unsigned> ReuseShuffleIndices = None,
                     ArrayRef<unsigned> ReorderIndices = None) {
     VectorizableTree.emplace_back(VectorizableTree);
     int idx = VectorizableTree.size() - 1;
     TreeEntry *Last = &VectorizableTree[idx];
-    Last->Scalars.insert(Last->Scalars.begin(), VL.begin(), VL.end());
+    Last->Scalars.insert(Last->Scalars.begin(), IL.begin(), IL.end());
     Last->NeedToGather = !Vectorized;
     Last->ReuseShuffleIndices.append(ReuseShuffleIndices.begin(),
                                      ReuseShuffleIndices.end());
     Last->ReorderIndices = ReorderIndices;
     if (Vectorized) {
-      for (int i = 0, e = VL.size(); i != e; ++i) {
-        assert(!getTreeEntry(VL[i]) && "Scalar already in tree!");
-        ScalarToTreeEntry[VL[i]] = idx;
+      for (InstructionOrPseudo *IOP : IL) {
+        assert(
+            (ScalarToTreeEntry.find(IOP->OpInst) == ScalarToTreeEntry.end() ||
+             ScalarToTreeEntry[IOP->OpInst].find(IOP->getKey()) ==
+                 ScalarToTreeEntry[IOP->OpInst].end()) &&
+            "Scalar already in tree!");
+        ScalarToTreeEntry[IOP->OpInst][IOP->getKey()] = idx;
       }
     } else {
-      MustGather.insert(VL.begin(), VL.end());
+      for (InstructionOrPseudo *IOP : IL)
+        MustGather.insert(IOP->Op);
     }
 
     if (UserTreeIdx >= 0)
       Last->UserTreeIndices.push_back(UserTreeIdx);
     UserTreeIdx = idx;
   }
 
   /// -- Vectorization State --
   /// Holds all of the tree entries.
   std::vector<TreeEntry> VectorizableTree;
 
-  TreeEntry *getTreeEntry(Value *V) {
-    auto I = ScalarToTreeEntry.find(V);
-    if (I != ScalarToTreeEntry.end())
-      return &VectorizableTree[I->second];
+  TreeEntry *getTreeEntry(Instruction *I) {
+    if (!I)
+      return nullptr;
+    auto It = ScalarToTreeEntry.find(I);
+    if (It != ScalarToTreeEntry.end()) {
+      auto &STT = It->second;
+      for (auto STTI : STT)
+        for (auto *IOP : VectorizableTree[STTI.second].Scalars)
+          if (IOP->OpInst == I && !IOP->isPseudo())
+            return &VectorizableTree[STTI.second];
+    }
     return nullptr;
   }
 
+  std::vector<std::unique_ptr<InstructionOrPseudo>> InstructionOrPseudoArray;
+
+  InstructionOrPseudo *allocInstructionOrPseudo(Value *V, Value* Parent,
+                                                const InstructionsState &S) {
+    InstructionOrPseudoArray.emplace_back(
+                       llvm::make_unique<InstructionOrPseudo>(V, Parent, S));
+    int idx = InstructionOrPseudoArray.size() - 1;
+    return (InstructionOrPseudoArray[idx]).get();
+  }
+
   /// Maps a specific scalar to its tree entry.
-  SmallDenseMap<Value*, int> ScalarToTreeEntry;
+  SmallDenseMap<Instruction *, SmallDenseMap<std::pair<Value *, unsigned>, int>>
+      ScalarToTreeEntry;
 
   /// A list of scalars that we found that we need to keep as scalars.
   ValueSet MustGather;
 
   /// This POD struct describes one external user in the vectorized tree.
   struct ExternalUser {
     ExternalUser(Value *S, llvm::User *U, int L)
         : Scalar(S), User(U), Lane(L) {}
 
     // Which scalar in our function.
     Value *Scalar;
private:
   /// vector instruction).
   struct ScheduleData {
     // The initial value for the dependency counters. It means that the
     // dependencies are not calculated yet.
     enum { InvalidDeps = -1 };
 
     ScheduleData() = default;
 
-    void init(int BlockSchedulingRegionID, Value *OpVal) {
+    void init(int BlockSchedulingRegionID) {
       FirstInBundle = this;
       NextInBundle = nullptr;
       NextLoadStore = nullptr;
       IsScheduled = false;
       SchedulingRegionID = BlockSchedulingRegionID;
       UnscheduledDepsInBundle = UnscheduledDeps;
       clearDependencies();
-      OpValue = OpVal;
+    }
+
+    void init(int BlockSchedulingRegionID, ScheduleData *OpOrigin,
+              Value *OpParent, unsigned OpCode) {
+     FirstInBundle = this;
+     NextInBundle = nullptr;
+     NextLoadStore = OpOrigin->NextLoadStore;
+     IsScheduled = false;
+     SchedulingRegionID = BlockSchedulingRegionID;
+     UnscheduledDepsInBundle = UnscheduledDeps;
+     clearDependencies();
+     Opcode = OpCode;
+     Parent = OpParent;
+     Origin = OpOrigin;
+     Inst = OpOrigin->Inst;
     }
 
     /// Returns true if the dependency information has been calculated.
     bool hasValidDependencies() const { return Dependencies != InvalidDeps; }
 
     /// Returns true for single instructions and for bundle representatives
     /// (= the head of a bundle).
     bool isSchedulingEntity() const { return FirstInBundle == this; }
struct ScheduleData {
 
     /// Clears all dependency information.
     void clearDependencies() {
       Dependencies = InvalidDeps;
       resetUnscheduledDeps();
       MemoryDependencies.clear();
     }
 
+    /// Returns true if the instance is a pseudo instruction one.
+    bool isPseudo() const {
+      return (Origin != nullptr);
+    }
+
     void dump(raw_ostream &os) const {
       if (!isSchedulingEntity()) {
-        os << "/ " << *Inst;
+        os << "/ ";
+        if (isPseudo())
+          os << "*";
+        os << *Inst;
       } else if (NextInBundle) {
-        os << '[' << *Inst;
+        os << '[';
+        if (isPseudo())
+          os << "*";
+        os << *Inst;
         ScheduleData *SD = NextInBundle;
         while (SD) {
-          os << ';' << *SD->Inst;
+           os << ';' ;
+           if (SD->isPseudo())
+             os << "*";
+           os << *SD->Inst;
            SD = SD->NextInBundle;
         }
         os << ']';
       } else {
-        os << *Inst;
+       if (isPseudo())
+         os << "*";
       }
     }
 
     Instruction *Inst = nullptr;
 
     /// Points to the head in an instruction bundle (and always to this for
     /// single instructions).
     ScheduleData *FirstInBundle = nullptr;
struct ScheduleData {
     /// The sum of UnscheduledDeps in a bundle. Equals to UnscheduledDeps for
     /// single instructions.
     int UnscheduledDepsInBundle = InvalidDeps;
 
     /// True if this instruction is scheduled (or considered as scheduled in the
     /// dry-run).
     bool IsScheduled = false;
 
-    /// Opcode of the current instruction in the schedule data.
-    Value *OpValue = nullptr;
+    /// Opcode that represents instructions to be vectorized.
+    unsigned Opcode = 0;
+
+    Value *Parent = nullptr;
+
+    ScheduleData *Origin = nullptr;
   };
 
 #ifndef NDEBUG
   friend inline raw_ostream &operator<<(raw_ostream &os,
                                         const BoUpSLP::ScheduleData &SD) {
     SD.dump(os);
     return os;
   }
struct BlockScheduling {
     void clear() {
       ReadyInsts.clear();
       ScheduleStart = nullptr;
       ScheduleEnd = nullptr;
       FirstLoadStoreInRegion = nullptr;
       LastLoadStoreInRegion = nullptr;
 
       // Reduce the maximum schedule region size by the size of the
       // previous scheduling run.
       ScheduleRegionSizeLimit -= ScheduleRegionSize;
       if (ScheduleRegionSizeLimit < MinScheduleRegionSize)
         ScheduleRegionSizeLimit = MinScheduleRegionSize;
       ScheduleRegionSize = 0;
 
       // Make a new scheduling region, i.e. all existing ScheduleData is not
       // in the new region yet.
       ++SchedulingRegionID;
     }
 
-    ScheduleData *getScheduleData(Value *V) {
-      ScheduleData *SD = ScheduleDataMap[V];
+    ScheduleData *getScheduleData(Instruction *I) {
+      ScheduleData *SD = InstScheduleDataMap[I];
       if (SD && SD->SchedulingRegionID == SchedulingRegionID)
         return SD;
       return nullptr;
     }
 
-    ScheduleData *getScheduleData(Value *V, Value *Key) {
-      if (V == Key)
-        return getScheduleData(V);
-      auto I = ExtraScheduleDataMap.find(V);
-      if (I != ExtraScheduleDataMap.end()) {
-        ScheduleData *SD = I->second[Key];
-        if (SD && SD->SchedulingRegionID == SchedulingRegionID)
-          return SD;
+    ScheduleData *getScheduleData(Instruction *I,
+                                  std::pair<Value *, unsigned> Key) {
+      ScheduleData *SD = getScheduleData(I);
+      if (SD && SD->Parent == Key.first && SD->Opcode == Key.second)
+      auto It = PseudoInstScheduleDataMap.find(I);
+      if (It != PseudoInstScheduleDataMap.end()) {
+        ScheduleData *PSD = It->second[Key];
+        if (PSD && PSD->SchedulingRegionID == SchedulingRegionID &&
+            PSD->Parent == Key.first && PSD->Opcode == Key.second)
+          return PSD;
       }
       return nullptr;
     }
 
     bool isInSchedulingRegion(ScheduleData *SD) {
       return SD->SchedulingRegionID == SchedulingRegionID;
     }
 
     /// Marks an instruction as scheduled and puts all dependent ready
     /// instructions into the ready-list.
     template <typename ReadyListType>
     void schedule(ScheduleData *SD, ReadyListType &ReadyList) {
       SD->IsScheduled = true;
       LLVM_DEBUG(dbgs() << "SLP:   schedule " << *SD << "\n");
 
       ScheduleData *BundleMember = SD;
+      unsigned Opcode = BundleMember->Opcode;
+      Value *Parent = BundleMember->Parent;
       while (BundleMember) {
-        if (BundleMember->Inst != BundleMember->OpValue) {
-          BundleMember = BundleMember->NextInBundle;
-          continue;
-        }
+        assert(BundleMember->Opcode == Opcode &&
+               BundleMember->Parent == Parent && "Corrupt bundle member");
         // Handle the def-use chain dependencies.
         for (Use &U : BundleMember->Inst->operands()) {
           auto *I = dyn_cast<Instruction>(U.get());
           if (!I)
             continue;
           doForAllOpcodes(I, [&ReadyList](ScheduleData *OpDef) {
             if (OpDef && OpDef->hasValidDependencies() &&
                 OpDef->incrementUnscheduledDeps(-1) == 0) {
void schedule(ScheduleData *SD, ReadyListType &ReadyList) {
             LLVM_DEBUG(dbgs()
                        << "SLP:    gets ready (mem): " << *DepBundle << "\n");
           }
         }
         BundleMember = BundleMember->NextInBundle;
       }
     }
 
-    void doForAllOpcodes(Value *V,
+    void doForAllOpcodes(Instruction *I,
                          function_ref<void(ScheduleData *SD)> Action) {
-      if (ScheduleData *SD = getScheduleData(V))
+      auto It = PseudoInstScheduleDataMap.find(I);
+      if (It != PseudoInstScheduleDataMap.end()) {
+        for (auto &P : It->second) {
+          ScheduleData *SD = P.second;
+          if (SD && SD->isPartOfBundle() &&
+              SD->SchedulingRegionID == SchedulingRegionID) {
             Action(SD);
-      auto I = ExtraScheduleDataMap.find(V);
-      if (I != ExtraScheduleDataMap.end())
-        for (auto &P : I->second)
-          if (P.second->SchedulingRegionID == SchedulingRegionID)
-            Action(P.second);
+          }
+        }
+      }
+      if (ScheduleData *SD = getScheduleData(I)) {
+        Action(SD);
+      }
     }
 
     /// Put all instructions into the ReadyList which are ready for scheduling.
     template <typename ReadyListType>
     void initialFillReadyList(ReadyListType &ReadyList) {
       for (auto *I = ScheduleStart; I != ScheduleEnd; I = I->getNextNode()) {
         doForAllOpcodes(I, [&](ScheduleData *SD) {
           if (SD->isSchedulingEntity() && SD->isReady()) {
             ReadyList.insert(SD);
             LLVM_DEBUG(dbgs()
                        << "SLP:    initially in ready list: " << *I << "\n");
           }
         });
       }
     }
 
     /// Checks if a bundle of instructions can be scheduled, i.e. has no
     /// cyclic dependencies. This is only a dry-run, no instructions are
     /// actually moved at this stage.
-    bool tryScheduleBundle(ArrayRef<Value *> VL, BoUpSLP *SLP,
-                           const InstructionsState &S);
+    bool tryScheduleBundle(ArrayRef<InstructionOrPseudo *> IL, BoUpSLP *SLP);
 
     /// Un-bundles a group of instructions.
-    void cancelScheduling(ArrayRef<Value *> VL, Value *OpValue);
+    void cancelScheduling(InstructionOrPseudo *IOP);
 
     /// Allocates schedule data chunk.
     ScheduleData *allocateScheduleDataChunks();
 
-    /// Extends the scheduling region so that V is inside the region.
+    /// Extends the scheduling region so that I is inside the region.
     /// \returns true if the region size is within the limit.
-    bool extendSchedulingRegion(Value *V, const InstructionsState &S);
+    bool extendSchedulingRegion(InstructionOrPseudo *IOP);
 
     /// Initialize the ScheduleData structures for new instructions in the
     /// scheduling region.
     void initScheduleData(Instruction *FromI, Instruction *ToI,
                           ScheduleData *PrevLoadStore,
                           ScheduleData *NextLoadStore);
 
     /// Updates the dependency information of a bundle and of all instructions/
     /// bundles which depend on the original bundle.
     void calculateDependencies(ScheduleData *SD, bool InsertInReadyList,
                                BoUpSLP *SLP);
 
     /// Sets all instruction in the scheduling region to un-scheduled.
     void resetSchedule();
 
+    /// Reorder bundles from PseudoScheduleData data after scheduling,
+    /// if an Instruction is present in PseudoScheduleData that means this
+    /// Instruction is prenet in multiply bundles and FirstInBundle is not last
+    /// one scheduled for all copies of instuction in InstScheduleData and
+    /// PseudoScheduleData.
+    void reorderBundles();
+
     BasicBlock *BB;
 
     /// Simple memory allocation for ScheduleData.
     std::vector<std::unique_ptr<ScheduleData[]>> ScheduleDataChunks;
 
     /// The size of a ScheduleData array in ScheduleDataChunks.
     int ChunkSize;
 
     /// The allocator position in the current chunk, which is the last entry
     /// of ScheduleDataChunks.
     int ChunkPos;
 
-    /// Attaches ScheduleData to Instruction.
+    /// Attaches InstScheduleData to Instruction.
     /// Note that the mapping survives during all vectorization iterations, i.e.
-    /// ScheduleData structures are recycled.
-    DenseMap<Value *, ScheduleData *> ScheduleDataMap;
+    /// InstScheduleData structures are recycled.
+    DenseMap<Instruction *, ScheduleData *> InstScheduleDataMap;
 
-    /// Attaches ScheduleData to Instruction with the leading key.
-    DenseMap<Value *, SmallDenseMap<Value *, ScheduleData *>>
-        ExtraScheduleDataMap;
+    DenseMap<Instruction *,
+             SmallDenseMap<std::pair<Value *, unsigned>, ScheduleData *>>
+        PseudoInstScheduleDataMap;
 
     struct ReadyList : SmallVector<ScheduleData *, 8> {
       void insert(ScheduleData *SD) { push_back(SD); }
     };
 
     /// The ready-list for scheduling (only used for the dry-run).
     ReadyList ReadyInsts;
 
struct BlockScheduling {
 
     /// The current size of the scheduling region.
     int ScheduleRegionSize = 0;
 
     /// The maximum size allowed for the scheduling region.
     int ScheduleRegionSizeLimit = ScheduleRegionSizeBudget;
 
     /// The ID of the scheduling region. For a new vectorization iteration this
-    /// is incremented which "removes" all ScheduleData from the region.
+    /// is incremented which "removes" all InstScheduleData from the region.
     // Make sure that the initial SchedulingRegionID is greater than the
-    // initial SchedulingRegionID in ScheduleData (which is 0).
+    // initial SchedulingRegionID in InstScheduleData (which is 0).
     int SchedulingRegionID = 1;
   };
 
   /// Attaches the BlockScheduling structures to basic blocks.
   MapVector<BasicBlock *, std::unique_ptr<BlockScheduling>> BlocksSchedules;
 
   /// Performs the "real" scheduling. Done before vectorization is actually
   /// performed in a basic block.
template <> struct DOTGraphTraits<BoUpSLP *> : public DefaultDOTGraphTraits {
   using TreeEntry = BoUpSLP::TreeEntry;
 
   DOTGraphTraits(bool isSimple = false) : DefaultDOTGraphTraits(isSimple) {}
 
   std::string getNodeLabel(const TreeEntry *Entry, const BoUpSLP *R) {
     std::string Str;
     raw_string_ostream OS(Str);
     if (isSplat(Entry->Scalars)) {
-      OS << "<splat> " << *Entry->Scalars[0];
+      OS << "<splat> " << *Entry->Scalars[0]->Op;
       return Str;
     }
-    for (auto V : Entry->Scalars) {
-      OS << *V;
-      if (std::any_of(
-              R->ExternalUses.begin(), R->ExternalUses.end(),
-              [&](const BoUpSLP::ExternalUser &EU) { return EU.Scalar == V; }))
+    for (auto *IOP : Entry->Scalars) {
+      OS << *IOP->Op;
+      if (std::any_of(R->ExternalUses.begin(), R->ExternalUses.end(),
+                      [&](const BoUpSLP::ExternalUser &EU) {
+                        return EU.Scalar == IOP->Op;
+                      }))
         OS << " <extract>";
       OS << "\n";
     }
     return Str;
   }
 
   static std::string getNodeAttributes(const TreeEntry *Entry,
                                        const BoUpSLP *) {

 
 void BoUpSLP::buildTree(ArrayRef<Value *> Roots,
                         ExtraValueToDebugLocsMap &ExternallyUsedValues,
                         ArrayRef<Value *> UserIgnoreLst) {
   deleteTree();
   UserIgnoreList = UserIgnoreLst;
   if (!allSameType(Roots))
     return;
-  buildTree_rec(Roots, 0, -1);
+  buildTree_rec(Roots[0], Roots, 0, -1);
 
   // Collect the values that we need to extract from the tree.
   for (TreeEntry &EIdx : VectorizableTree) {
     TreeEntry *Entry = &EIdx;
 
     // No need to handle users of gathered values.
     if (Entry->NeedToGather)
       continue;
 
     // For each lane:
     for (int Lane = 0, LE = Entry->Scalars.size(); Lane != LE; ++Lane) {
-      Value *Scalar = Entry->Scalars[Lane];
+      auto *Scalar = Entry->Scalars[Lane];
       int FoundLane = Lane;
+      if (Scalar->isPseudo())
+        continue;
       if (!Entry->ReuseShuffleIndices.empty()) {
         FoundLane =
             std::distance(Entry->ReuseShuffleIndices.begin(),
                           llvm::find(Entry->ReuseShuffleIndices, FoundLane));
       }
 
       // Check if the scalar is externally used as an extra arg.
-      auto ExtI = ExternallyUsedValues.find(Scalar);
+      auto ExtI = ExternallyUsedValues.find(Scalar->Op);
       if (ExtI != ExternallyUsedValues.end()) {
         LLVM_DEBUG(dbgs() << "SLP: Need to extract: Extra arg from lane "
-                          << Lane << " from " << *Scalar << ".\n");
-        ExternalUses.emplace_back(Scalar, nullptr, FoundLane);
+                          << Lane << " from " << *Scalar->Op << ".\n");
+        ExternalUses.emplace_back(Scalar->Op, nullptr, FoundLane);
       }
-      for (User *U : Scalar->users()) {
+      for (User *U : Scalar->Op->users()) {
         LLVM_DEBUG(dbgs() << "SLP: Checking user:" << *U << ".\n");
 
         Instruction *UserInst = dyn_cast<Instruction>(U);
         if (!UserInst)
           continue;
 
         // Skip in-tree scalars that become vectors
-        if (TreeEntry *UseEntry = getTreeEntry(U)) {
-          Value *UseScalar = UseEntry->Scalars[0];
+        if (TreeEntry *UseEntry = getTreeEntry(UserInst)) {
+          Value *UseScalar = UseEntry->Scalars[0]->Op;
           // Some in-tree scalars will remain as scalar in vectorized
           // instructions. If that is the case, the one in Lane 0 will
           // be used.
           if (UseScalar != U ||
-              !InTreeUserNeedToExtract(Scalar, UserInst, TLI)) {
+              !InTreeUserNeedToExtract(Scalar->Op, UserInst, TLI)) {
             LLVM_DEBUG(dbgs() << "SLP: \tInternal user will be removed:" << *U
                               << ".\n");
             assert(!UseEntry->NeedToGather && "Bad state");
             continue;
           }
         }
 
         // Ignore users in the user ignore list.
         if (is_contained(UserIgnoreList, UserInst))
           continue;
 
         LLVM_DEBUG(dbgs() << "SLP: Need to extract:" << *U << " from lane "
-                          << Lane << " from " << *Scalar << ".\n");
-        ExternalUses.push_back(ExternalUser(Scalar, U, FoundLane));
+                          << Lane << " from " << *Scalar->Op << ".\n");
+        ExternalUses.push_back(ExternalUser(Scalar->Op, U, FoundLane));
       }
     }
   }
 }
 
-void BoUpSLP::buildTree_rec(ArrayRef<Value *> VL, unsigned Depth,
+void BoUpSLP::buildTree_rec(Value *Parent, ArrayRef<Value *> VL, unsigned Depth,
                             int UserTreeIdx) {
-  assert((allConstant(VL) || allSameType(VL)) && "Invalid types!");
 
   InstructionsState S = getSameOpcode(VL);
+
+  SmallVector<InstructionOrPseudo *, 4> IL;
+
+  for (Value *V: VL)
+    IL.push_back(allocInstructionOrPseudo(V, Parent, S));
+
+  assert((allConstant(IL) || allSameType(VL)) && "Invalid types!");
+
   if (Depth == RecursionMaxDepth) {
     LLVM_DEBUG(dbgs() << "SLP: Gathering due to max recursion depth.\n");
-    newTreeEntry(VL, false, UserTreeIdx);
+    newTreeEntry(IL, false, UserTreeIdx);
     return;
   }
 
   // Don't handle vectors.
   if (S.OpValue->getType()->isVectorTy()) {
     LLVM_DEBUG(dbgs() << "SLP: Gathering due to vector type.\n");
-    newTreeEntry(VL, false, UserTreeIdx);
+    newTreeEntry(IL, false, UserTreeIdx);
     return;
   }
 
   if (StoreInst *SI = dyn_cast<StoreInst>(S.OpValue))
     if (SI->getValueOperand()->getType()->isVectorTy()) {
       LLVM_DEBUG(dbgs() << "SLP: Gathering due to store vector type.\n");
-      newTreeEntry(VL, false, UserTreeIdx);
+      newTreeEntry(IL, false, UserTreeIdx);
       return;
     }
 
   // If all of the operands are identical or constant we have a simple solution.
-  if (allConstant(VL) || isSplat(VL) || !allSameBlock(VL) || !S.getOpcode()) {
+  if (allConstant(IL) || isSplat(IL) || !allSameBlock(IL) || !S.getOpcode()) {
     LLVM_DEBUG(dbgs() << "SLP: Gathering due to C,S,B,O. \n");
-    newTreeEntry(VL, false, UserTreeIdx);
+    newTreeEntry(IL, false, UserTreeIdx);
     return;
   }
 
   // We now know that this is a vector of instructions of the same type from
   // the same block.
 
   // Don't vectorize ephemeral values.
-  for (unsigned i = 0, e = VL.size(); i != e; ++i) {
-    if (EphValues.count(VL[i])) {
+  for (unsigned i = 0, e = IL.size(); i != e; ++i) {
+    if (EphValues.count(IL[i]->Op)) {
       LLVM_DEBUG(dbgs() << "SLP: The instruction (" << *VL[i]
                         << ") is ephemeral.\n");
-      newTreeEntry(VL, false, UserTreeIdx);
+      newTreeEntry(IL, false, UserTreeIdx);
       return;
     }
   }
 
   // Check if this is a duplicate of another entry.
-  if (TreeEntry *E = getTreeEntry(S.OpValue)) {
+  TreeEntry *E = getTreeEntry(cast<Instruction>(S.OpValue));
+  if (E && E->isSame(IL)) {
     LLVM_DEBUG(dbgs() << "SLP: \tChecking bundle: " << *S.OpValue << ".\n");
-    if (!E->isSame(VL)) {
-      LLVM_DEBUG(dbgs() << "SLP: Gathering due to partial overlap.\n");
-      newTreeEntry(VL, false, UserTreeIdx);
-      return;
-    }
     // Record the reuse of the tree node.  FIXME, currently this is only used to
     // properly draw the graph rather than for the actual vectorization.
     E->UserTreeIndices.push_back(UserTreeIdx);
     LLVM_DEBUG(dbgs() << "SLP: Perfect diamond merge at " << *S.OpValue
                       << ".\n");
     return;
   }
 
   // Check that none of the instructions in the bundle are already in the tree.
-  for (unsigned i = 0, e = VL.size(); i != e; ++i) {
-    auto *I = dyn_cast<Instruction>(VL[i]);
-    if (!I)
-      continue;
-    if (getTreeEntry(I)) {
-      LLVM_DEBUG(dbgs() << "SLP: The instruction (" << *VL[i]
+  for (unsigned i = 0, e = IL.size(); i != e; ++i) {
+    auto *I = IL[i]->OpInst;
+    if (getTreeEntry(I) ||
+        (ScalarToTreeEntry.find(I) != ScalarToTreeEntry.end() &&
+         ScalarToTreeEntry[I].find(IL[i]->getKey()) !=
+         ScalarToTreeEntry[I].end())) {
+      LLVM_DEBUG(dbgs() << "SLP: The instruction (" << *I
                         << ") is already in tree.\n");
-      newTreeEntry(VL, false, UserTreeIdx);
+      newTreeEntry(IL, false, UserTreeIdx);
       return;
     }
   }
 
   // If any of the scalars is marked as a value that needs to stay scalar, then
   // we need to gather the scalars.
-  for (unsigned i = 0, e = VL.size(); i != e; ++i) {
-    if (MustGather.count(VL[i])) {
+  for (unsigned i = 0, e = IL.size(); i != e; ++i) {
+    if (MustGather.count(IL[i]->Op)) {
       LLVM_DEBUG(dbgs() << "SLP: Gathering due to gathered scalar.\n");
-      newTreeEntry(VL, false, UserTreeIdx);
+      newTreeEntry(IL, false, UserTreeIdx);
       return;
     }
   }
 
   // Check that all of the users of the scalars that we want to vectorize are
   // schedulable.
-  auto *VL0 = cast<Instruction>(S.OpValue);
+  auto *VL0 = IL[0]->MainOp;
   BasicBlock *BB = VL0->getParent();
 
   if (!DT->isReachableFromEntry(BB)) {
     // Don't go into unreachable blocks. They may contain instructions with
     // dependency cycles which confuse the final scheduling.
     LLVM_DEBUG(dbgs() << "SLP: bundle in unreachable block.\n");
-    newTreeEntry(VL, false, UserTreeIdx);
+    newTreeEntry(IL, false, UserTreeIdx);
     return;
   }
 
   // Check that every instruction appears once in this bundle.
   SmallVector<unsigned, 4> ReuseShuffleIndicies;
-  SmallVector<Value *, 4> UniqueValues;
+  SmallVector<InstructionOrPseudo *, 4> UniqueValues;
   DenseMap<Value *, unsigned> UniquePositions;
-  for (Value *V : VL) {
-    auto Res = UniquePositions.try_emplace(V, UniqueValues.size());
+  for (auto *I : IL) {
+    auto Res = UniquePositions.try_emplace(I->Op, UniqueValues.size());
     ReuseShuffleIndicies.emplace_back(Res.first->second);
     if (Res.second)
-      UniqueValues.emplace_back(V);
+      UniqueValues.emplace_back(allocInstructionOrPseudo(I->Op, Parent, S));
   }
-  if (UniqueValues.size() == VL.size()) {
+  if (UniqueValues.size() == IL.size()) {
     ReuseShuffleIndicies.clear();
   } else {
     LLVM_DEBUG(dbgs() << "SLP: Shuffle for reused scalars.\n");
     if (UniqueValues.size() <= 1 || !llvm::isPowerOf2_32(UniqueValues.size())) {
       LLVM_DEBUG(dbgs() << "SLP: Scalar used twice in bundle.\n");
-      newTreeEntry(VL, false, UserTreeIdx);
+      newTreeEntry(IL, false, UserTreeIdx);
       return;
     }
-    VL = UniqueValues;
+    IL = UniqueValues;
   }
 
   auto &BSRef = BlocksSchedules[BB];
   if (!BSRef)
     BSRef = llvm::make_unique<BlockScheduling>(BB);
 
   BlockScheduling &BS = *BSRef.get();
 
-  if (!BS.tryScheduleBundle(VL, this, S)) {
+  if (!BS.tryScheduleBundle(IL, this)) {
     LLVM_DEBUG(dbgs() << "SLP: We are not able to schedule this bundle!\n");
-    assert((!BS.getScheduleData(VL0) ||
-            !BS.getScheduleData(VL0)->isPartOfBundle()) &&
+    assert((!BS.getScheduleData(VL0, IL[0]->getKey()) ||
+            !BS.getScheduleData(VL0, IL[0]->getKey())->isPartOfBundle()) &&
            "tryScheduleBundle should cancelScheduling on failure");
-    newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+    newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
     return;
   }
   LLVM_DEBUG(dbgs() << "SLP: We are able to schedule this bundle.\n");
 
-  unsigned ShuffleOrOp = S.isAltShuffle() ?
+  unsigned ShuffleOrOp = IL[0]->isAltShuffle() ?
                 (unsigned) Instruction::ShuffleVector : S.getOpcode();
   switch (ShuffleOrOp) {
     case Instruction::PHI: {
       PHINode *PH = dyn_cast<PHINode>(VL0);
 
       // Check for terminator values (e.g. invoke).
-      for (unsigned j = 0; j < VL.size(); ++j)
+      for (unsigned j = 0; j < IL.size(); ++j)
         for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
           Instruction *Term = dyn_cast<Instruction>(
-              cast<PHINode>(VL[j])->getIncomingValueForBlock(
+              cast<PHINode>(IL[j]->OpInst)->getIncomingValueForBlock(
                   PH->getIncomingBlock(i)));
           if (Term && Term->isTerminator()) {
             LLVM_DEBUG(dbgs()
                        << "SLP: Need to swizzle PHINodes (terminator use).\n");
-            BS.cancelScheduling(VL, VL0);
-            newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+            BS.cancelScheduling(IL[0]);
+            newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
             return;
           }
         }
 
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a vector of PHINodes.\n");
 
       for (unsigned i = 0, e = PH->getNumIncomingValues(); i < e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL)
-          Operands.push_back(cast<PHINode>(j)->getIncomingValueForBlock(
+        for (auto *j : IL)
+          Operands.push_back(cast<PHINode>(j->Op)->getIncomingValueForBlock(
               PH->getIncomingBlock(i)));
 
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
     }
     case Instruction::ExtractValue:
     case Instruction::ExtractElement: {
       OrdersType CurrentOrder;
-      bool Reuse = canReuseExtract(VL, VL0, CurrentOrder);
+      bool Reuse = canReuseExtract(IL, VL0, CurrentOrder);
       if (Reuse) {
         LLVM_DEBUG(dbgs() << "SLP: Reusing or shuffling extract sequence.\n");
         ++NumOpsWantToKeepOriginalOrder;
-        newTreeEntry(VL, /*Vectorized=*/true, UserTreeIdx,
+        newTreeEntry(IL, /*Vectorized=*/true, UserTreeIdx,
                      ReuseShuffleIndicies);
         return;
       }
       if (!CurrentOrder.empty()) {
         LLVM_DEBUG({
           dbgs() << "SLP: Reusing or shuffling of reordered extract sequence "
                     "with order";
           for (unsigned Idx : CurrentOrder)
             dbgs() << " " << Idx;
           dbgs() << "\n";
         });
         // Insert new order with initial value 0, if it does not exist,
         // otherwise return the iterator to the existing one.
         auto StoredCurrentOrderAndNum =
             NumOpsWantToKeepOrder.try_emplace(CurrentOrder).first;
         ++StoredCurrentOrderAndNum->getSecond();
-        newTreeEntry(VL, /*Vectorized=*/true, UserTreeIdx, ReuseShuffleIndicies,
+        newTreeEntry(IL, /*Vectorized=*/true, UserTreeIdx,
+                     ReuseShuffleIndicies,
                      StoredCurrentOrderAndNum->getFirst());
         return;
       }
       LLVM_DEBUG(dbgs() << "SLP: Gather extract sequence.\n");
-      newTreeEntry(VL, /*Vectorized=*/false, UserTreeIdx, ReuseShuffleIndicies);
-      BS.cancelScheduling(VL, VL0);
+      newTreeEntry(IL, /*Vectorized=*/false, UserTreeIdx,
+                   ReuseShuffleIndicies);
+      BS.cancelScheduling(IL[0]);
       return;
     }
     case Instruction::Load: {
       // Check that a vectorized load would load the same memory as a scalar
       // load. For example, we don't want to vectorize loads that are smaller
       // than 8-bit. Even though we have a packed struct {<i2, i2, i2, i2>} LLVM
       // treats loading/storing it as an i8 struct. If we vectorize loads/stores
       // from such a struct, we read/write packed bits disagreeing with the
       // unvectorized version.
       Type *ScalarTy = VL0->getType();
 
       if (DL->getTypeSizeInBits(ScalarTy) !=
           DL->getTypeAllocSizeInBits(ScalarTy)) {
-        BS.cancelScheduling(VL, VL0);
-        newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+        BS.cancelScheduling(IL[0]);
+        newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
         LLVM_DEBUG(dbgs() << "SLP: Gathering loads of non-packed type.\n");
         return;
       }
 
       // Make sure all loads in the bundle are simple - we can't vectorize
       // atomic or volatile loads.
-      SmallVector<Value *, 4> PointerOps(VL.size());
+      SmallVector<Value *, 4> PointerOps(IL.size());
       auto POIter = PointerOps.begin();
-      for (Value *V : VL) {
-        auto *L = cast<LoadInst>(V);
+      for (auto *I : IL) {
+        auto *L = cast<LoadInst>(I->OpInst);
         if (!L->isSimple()) {
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           LLVM_DEBUG(dbgs() << "SLP: Gathering non-simple loads.\n");
           return;
         }
         *POIter = L->getPointerOperand();
         ++POIter;
       }
 
       OrdersType CurrentOrder;
case Instruction::Load: {
           PtrN = PointerOps[CurrentOrder.back()];
         }
         const SCEV *Scev0 = SE->getSCEV(Ptr0);
         const SCEV *ScevN = SE->getSCEV(PtrN);
         const auto *Diff =
             dyn_cast<SCEVConstant>(SE->getMinusSCEV(ScevN, Scev0));
         uint64_t Size = DL->getTypeAllocSize(ScalarTy);
         // Check that the sorted loads are consecutive.
-        if (Diff && Diff->getAPInt().getZExtValue() == (VL.size() - 1) * Size) {
+        if (Diff && Diff->getAPInt().getZExtValue() == (IL.size() - 1) * Size) {
           if (CurrentOrder.empty()) {
             // Original loads are consecutive and does not require reordering.
             ++NumOpsWantToKeepOriginalOrder;
-            newTreeEntry(VL, /*Vectorized=*/true, UserTreeIdx,
+            newTreeEntry(IL, /*Vectorized=*/true, UserTreeIdx,
                          ReuseShuffleIndicies);
             LLVM_DEBUG(dbgs() << "SLP: added a vector of loads.\n");
           } else {
             // Need to reorder.
             auto I = NumOpsWantToKeepOrder.try_emplace(CurrentOrder).first;
             ++I->getSecond();
-            newTreeEntry(VL, /*Vectorized=*/true, UserTreeIdx,
+            newTreeEntry(IL, /*Vectorized=*/true, UserTreeIdx,
                          ReuseShuffleIndicies, I->getFirst());
             LLVM_DEBUG(dbgs() << "SLP: added a vector of jumbled loads.\n");
           }
           return;
         }
       }
 
       LLVM_DEBUG(dbgs() << "SLP: Gathering non-consecutive loads.\n");
-      BS.cancelScheduling(VL, VL0);
-      newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+      BS.cancelScheduling(IL[0]);
+      newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
       return;
     }
     case Instruction::ZExt:
     case Instruction::SExt:
     case Instruction::FPToUI:
     case Instruction::FPToSI:
     case Instruction::FPExt:
     case Instruction::PtrToInt:
     case Instruction::IntToPtr:
     case Instruction::SIToFP:
     case Instruction::UIToFP:
     case Instruction::Trunc:
     case Instruction::FPTrunc:
     case Instruction::BitCast: {
       Type *SrcTy = VL0->getOperand(0)->getType();
-      for (unsigned i = 0; i < VL.size(); ++i) {
-        Type *Ty = cast<Instruction>(VL[i])->getOperand(0)->getType();
+      for (unsigned i = 0; i < IL.size(); ++i) {
+        Type *Ty = IL[i]->OpInst->getOperand(0)->getType();
         if (Ty != SrcTy || !isValidElementType(Ty)) {
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           LLVM_DEBUG(dbgs()
                      << "SLP: Gathering casts with different src types.\n");
           return;
         }
       }
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a vector of casts.\n");
 
       for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL)
-          Operands.push_back(cast<Instruction>(j)->getOperand(i));
+        for (auto *I : IL)
+          Operands.push_back(I->OpInst->getOperand(i));
 
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
     }
     case Instruction::ICmp:
     case Instruction::FCmp: {
       // Check that all of the compares have the same predicate.
       CmpInst::Predicate P0 = cast<CmpInst>(VL0)->getPredicate();
       Type *ComparedTy = VL0->getOperand(0)->getType();
-      for (unsigned i = 1, e = VL.size(); i < e; ++i) {
-        CmpInst *Cmp = cast<CmpInst>(VL[i]);
+      for (unsigned i = 1, e = IL.size(); i < e; ++i) {
+        CmpInst *Cmp = cast<CmpInst>(IL[i]->OpInst);
         if (Cmp->getPredicate() != P0 ||
             Cmp->getOperand(0)->getType() != ComparedTy) {
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           LLVM_DEBUG(dbgs()
                      << "SLP: Gathering cmp with different predicate.\n");
           return;
         }
       }
 
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a vector of compares.\n");
 
       for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL)
-          Operands.push_back(cast<Instruction>(j)->getOperand(i));
+        for (auto *I : IL)
+          Operands.push_back(I->OpInst->getOperand(i));
 
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
     }
     case Instruction::Select:
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
     case Instruction::UDiv:
     case Instruction::SDiv:
     case Instruction::FDiv:
     case Instruction::URem:
     case Instruction::SRem:
     case Instruction::FRem:
     case Instruction::Shl:
     case Instruction::LShr:
     case Instruction::AShr:
     case Instruction::And:
     case Instruction::Or:
     case Instruction::Xor:
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a vector of bin op.\n");
 
       // Sort operands of the instructions so that each side is more likely to
       // have the same opcode.
       if (isa<BinaryOperator>(VL0) && VL0->isCommutative()) {
         ValueList Left, Right;
-        reorderInputsAccordingToOpcode(S.getOpcode(), VL, Left, Right);
-        buildTree_rec(Left, Depth + 1, UserTreeIdx);
-        buildTree_rec(Right, Depth + 1, UserTreeIdx);
+        reorderInputsAccordingToOpcode(S.getOpcode(), IL, Left, Right);
+        buildTree_rec(VL0, Left, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Right, Depth + 1, UserTreeIdx);
         return;
       }
 
       for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL)
-          Operands.push_back(cast<Instruction>(j)->getOperand(i));
-
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        for (auto *Inst: IL) {
+          auto *I = Inst->OpInst;
+          if (I->getOpcode() == S.getOpcode()) {
+             Operands.push_back(I->getOperand(i));
+             continue;
+          }
+          assert(Instruction::isBinaryOp(S.getOpcode()) &&
+                  "Expected a binary operation.");
+          Operands.push_back(Inst->Op);
+        }
+        if (allSameType(Operands))
+          buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
 
     case Instruction::GetElementPtr: {
       // We don't combine GEPs with complicated (nested) indexing.
-      for (unsigned j = 0; j < VL.size(); ++j) {
-        if (cast<Instruction>(VL[j])->getNumOperands() != 2) {
+      for (unsigned j = 0; j < IL.size(); ++j) {
+        if (IL[j]->OpInst->getNumOperands() != 2) {
           LLVM_DEBUG(dbgs() << "SLP: not-vectorizable GEP (nested indexes).\n");
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           return;
         }
       }
 
       // We can't combine several GEPs into one vector if they operate on
       // different types.
       Type *Ty0 = VL0->getOperand(0)->getType();
-      for (unsigned j = 0; j < VL.size(); ++j) {
-        Type *CurTy = cast<Instruction>(VL[j])->getOperand(0)->getType();
+      for (unsigned j = 0; j < IL.size(); ++j) {
+        Type *CurTy = IL[j]->OpInst->getOperand(0)->getType();
         if (Ty0 != CurTy) {
           LLVM_DEBUG(dbgs()
                      << "SLP: not-vectorizable GEP (different types).\n");
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           return;
         }
       }
 
       // We don't combine GEPs with non-constant indexes.
-      for (unsigned j = 0; j < VL.size(); ++j) {
-        auto Op = cast<Instruction>(VL[j])->getOperand(1);
+      for (unsigned j = 0; j < IL.size(); ++j) {
+        auto Op = IL[j]->OpInst->getOperand(1);
         if (!isa<ConstantInt>(Op)) {
           LLVM_DEBUG(dbgs()
                      << "SLP: not-vectorizable GEP (non-constant indexes).\n");
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           return;
         }
       }
 
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a vector of GEPs.\n");
       for (unsigned i = 0, e = 2; i < e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL)
-          Operands.push_back(cast<Instruction>(j)->getOperand(i));
+        for (auto *j : IL)
+          Operands.push_back(j->OpInst->getOperand(i));
 
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
     }
     case Instruction::Store: {
       // Check if the stores are consecutive or of we need to swizzle them.
-      for (unsigned i = 0, e = VL.size() - 1; i < e; ++i)
-        if (!isConsecutiveAccess(VL[i], VL[i + 1], *DL, *SE)) {
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+      for (unsigned i = 0, e = IL.size() - 1; i < e; ++i)
+        if (!isConsecutiveAccess(IL[i]->Op, IL[i + 1]->Op, *DL, *SE)) {
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           LLVM_DEBUG(dbgs() << "SLP: Non-consecutive store.\n");
           return;
         }
 
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a vector of stores.\n");
 
       ValueList Operands;
-      for (Value *j : VL)
-        Operands.push_back(cast<Instruction>(j)->getOperand(0));
+      for (auto *j : IL)
+        Operands.push_back(j->OpInst->getOperand(0));
 
-      buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+      buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       return;
     }
     case Instruction::Call: {
       // Check if the calls are all to the same vectorizable intrinsic.
       CallInst *CI = cast<CallInst>(VL0);
       // Check if this is an Intrinsic call or something that can be
       // represented by an intrinsic call
       Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
       if (!isTriviallyVectorizable(ID)) {
-        BS.cancelScheduling(VL, VL0);
-        newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+        BS.cancelScheduling(IL[0]);
+        newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
         LLVM_DEBUG(dbgs() << "SLP: Non-vectorizable call.\n");
         return;
       }
       Function *Int = CI->getCalledFunction();
       Value *A1I = nullptr;
       if (hasVectorInstrinsicScalarOpd(ID, 1))
         A1I = CI->getArgOperand(1);
-      for (unsigned i = 1, e = VL.size(); i != e; ++i) {
-        CallInst *CI2 = dyn_cast<CallInst>(VL[i]);
+      for (unsigned i = 1, e = IL.size(); i != e; ++i) {
+        CallInst *CI2 = dyn_cast<CallInst>(IL[i]->OpInst);
         if (!CI2 || CI2->getCalledFunction() != Int ||
             getVectorIntrinsicIDForCall(CI2, TLI) != ID ||
             !CI->hasIdenticalOperandBundleSchema(*CI2)) {
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
-          LLVM_DEBUG(dbgs() << "SLP: mismatched calls:" << *CI << "!=" << *VL[i]
-                            << "\n");
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
+          LLVM_DEBUG(dbgs() << "SLP: mismatched calls:" << *CI << "!=" <<
+                            *IL[i]->OpInst << "\n");
           return;
         }
         // ctlz,cttz and powi are special intrinsics whose second argument
         // should be same in order for them to be vectorized.
         if (hasVectorInstrinsicScalarOpd(ID, 1)) {
           Value *A1J = CI2->getArgOperand(1);
           if (A1I != A1J) {
-            BS.cancelScheduling(VL, VL0);
-            newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+            BS.cancelScheduling(IL[0]);
+            newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
             LLVM_DEBUG(dbgs() << "SLP: mismatched arguments in call:" << *CI
                               << " argument " << A1I << "!=" << A1J << "\n");
             return;
           }
         }
         // Verify that the bundle operands are identical between the two calls.
         if (CI->hasOperandBundles() &&
             !std::equal(CI->op_begin() + CI->getBundleOperandsStartIndex(),
                         CI->op_begin() + CI->getBundleOperandsEndIndex(),
                         CI2->op_begin() + CI2->getBundleOperandsStartIndex())) {
-          BS.cancelScheduling(VL, VL0);
-          newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+          BS.cancelScheduling(IL[0]);
+          newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
           LLVM_DEBUG(dbgs() << "SLP: mismatched bundle operands in calls:"
-                            << *CI << "!=" << *VL[i] << '\n');
+                            << *CI << "!=" << *IL[i]->OpInst << '\n');
           return;
         }
       }
 
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       for (unsigned i = 0, e = CI->getNumArgOperands(); i != e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL) {
-          CallInst *CI2 = dyn_cast<CallInst>(j);
-          Operands.push_back(CI2->getArgOperand(i));
+        for (auto *VecOp : IL) {
+          auto *I = VecOp->OpInst;
+          if (VecOp->isOpcodeOrAlt()) {
+             Operands.push_back(I->getOperand(i));
+             continue;
+          }
+          assert(Instruction::isBinaryOp(S.getOpcode()) &&
+                  "Expected a binary operation.");
+          Value *Operand = ConstantExpr::getBinOpIdentity(S.getOpcode(),
+                                                       I->getType(), true);
+          Operands.push_back(Operand);
         }
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        if (allSameType(Operands))
+          buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
     }
     case Instruction::ShuffleVector:
       // If this is not an alternate sequence of opcode like add-sub
       // then do not vectorize this instruction.
-      if (!S.isAltShuffle()) {
-        BS.cancelScheduling(VL, VL0);
-        newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+      if (!IL[0]->isAltShuffle()) {
+        BS.cancelScheduling(IL[0]);
+        newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
         LLVM_DEBUG(dbgs() << "SLP: ShuffleVector are not vectorized.\n");
         return;
       }
-      newTreeEntry(VL, true, UserTreeIdx, ReuseShuffleIndicies);
+      newTreeEntry(IL, true, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: added a ShuffleVector op.\n");
 
       // Reorder operands if reordering would enable vectorization.
       if (isa<BinaryOperator>(VL0)) {
         ValueList Left, Right;
-        reorderAltShuffleOperands(S, VL, Left, Right);
-        buildTree_rec(Left, Depth + 1, UserTreeIdx);
-        buildTree_rec(Right, Depth + 1, UserTreeIdx);
+        reorderAltShuffleOperands(IL, Left, Right);
+        buildTree_rec(VL0, Left, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Right, Depth + 1, UserTreeIdx);
         return;
       }
 
       for (unsigned i = 0, e = VL0->getNumOperands(); i < e; ++i) {
         ValueList Operands;
         // Prepare the operand vector.
-        for (Value *j : VL)
-          Operands.push_back(cast<Instruction>(j)->getOperand(i));
+        for (auto *VecOp : IL) {
+          if (VecOp->isOpcodeOrAlt()) {
+            Operands.push_back(VecOp->OpInst->getOperand(i));
+            continue;
+          }
+          assert(Instruction::isBinaryOp(S.getOpcode()) &&
+                  "Expected a binary operation.");
+          Value *Operand = ConstantExpr::getBinOpIdentity(
+              S.getOpcode(), VecOp->OpInst->getType(), true);
+          Operands.push_back(Operand);
+        }
 
-        buildTree_rec(Operands, Depth + 1, UserTreeIdx);
+        buildTree_rec(VL0, Operands, Depth + 1, UserTreeIdx);
       }
       return;
 
     default:
-      BS.cancelScheduling(VL, VL0);
-      newTreeEntry(VL, false, UserTreeIdx, ReuseShuffleIndicies);
+      BS.cancelScheduling(IL[0]);
+      newTreeEntry(IL, false, UserTreeIdx, ReuseShuffleIndicies);
       LLVM_DEBUG(dbgs() << "SLP: Gathering unknown instruction.\n");
       return;
   }
 }
 
 unsigned BoUpSLP::canMapToVector(Type *T, const DataLayout &DL) const {
   unsigned N;
   Type *EltTy;
if (ST) {
     // Check that struct is homogeneous.
     for (const auto *Ty : ST->elements())
       if (Ty != EltTy)
         return 0;
   }
   return N;
 }
 
-bool BoUpSLP::canReuseExtract(ArrayRef<Value *> VL, Value *OpValue,
+bool BoUpSLP::canReuseExtract(ArrayRef<InstructionOrPseudo *> IL,
+                              Value *OpValue,
                               SmallVectorImpl<unsigned> &CurrentOrder) const {
   Instruction *E0 = cast<Instruction>(OpValue);
   assert(E0->getOpcode() == Instruction::ExtractElement ||
          E0->getOpcode() == Instruction::ExtractValue);
+  ValueList VL;
+  for (auto *I: IL)
+    VL.push_back(I->Op);
   assert(E0->getOpcode() == getSameOpcode(VL).getOpcode() && "Invalid opcode");
   // Check if all of the extracts come from the same vector and from the
   // correct offset.
   Value *Vec = E0->getOperand(0);
 
   CurrentOrder.clear();
 
   // We have to extract from a vector/aggregate with the same number of elements.
   unsigned NElts;
   if (E0->getOpcode() == Instruction::ExtractValue) {
     const DataLayout &DL = E0->getModule()->getDataLayout();
     NElts = canMapToVector(Vec->getType(), DL);
     if (!NElts)
       return false;
     // Check if load can be rewritten as load of vector.
     LoadInst *LI = dyn_cast<LoadInst>(Vec);
-    if (!LI || !LI->isSimple() || !LI->hasNUses(VL.size()))
+    if (!LI || !LI->isSimple() || !LI->hasNUses(IL.size()))
       return false;
   } else {
     NElts = Vec->getType()->getVectorNumElements();
   }
 
-  if (NElts != VL.size())
+  if (NElts != IL.size())
     return false;
 
   // Check that all of the indices extract from the correct offset.
   bool ShouldKeepOrder = true;
-  unsigned E = VL.size();
+  unsigned E = IL.size();
   // Assign to all items the initial value E + 1 so we can check if the extract
   // instruction index was used already.
   // Also, later we can check that all the indices are used and we have a
   // consecutive access in the extract instructions, by checking that no
   // element of CurrentOrder still has value E + 1.
   CurrentOrder.assign(E, E + 1);
   unsigned I = 0;
   for (; I < E; ++I) {
-    auto *Inst = cast<Instruction>(VL[I]);
+    auto *Inst = IL[I]->OpInst;
     if (Inst->getOperand(0) != Vec)
       break;
     Optional<unsigned> Idx = getExtractIndex(Inst);
     if (!Idx)
       break;
     const unsigned ExtIdx = *Idx;
     if (ExtIdx != I) {
       if (ExtIdx >= E || CurrentOrder[ExtIdx] != E + 1)
bool BoUpSLP::canReuseExtract(ArrayRef<InstructionOrPseudo *> IL,
   }
 
   return ShouldKeepOrder;
 }
 
 bool BoUpSLP::areAllUsersVectorized(Instruction *I) const {
   return I->hasOneUse() ||
          std::all_of(I->user_begin(), I->user_end(), [this](User *U) {
-           return ScalarToTreeEntry.count(U) > 0;
+           return ScalarToTreeEntry.count(dyn_cast<Instruction>(U)) > 0;
          });
 }
 
 int BoUpSLP::getEntryCost(TreeEntry *E) {
-  ArrayRef<Value*> VL = E->Scalars;
-
-  Type *ScalarTy = VL[0]->getType();
-  if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
+  ArrayRef<InstructionOrPseudo *> IL = E->Scalars;
+  Type *ScalarTy = E->Scalars[0]->Op->getType();
+  if (StoreInst *SI = dyn_cast<StoreInst>(E->Scalars[0]->Op))
     ScalarTy = SI->getValueOperand()->getType();
-  else if (CmpInst *CI = dyn_cast<CmpInst>(VL[0]))
+  else if (CmpInst *CI = dyn_cast<CmpInst>(E->Scalars[0]->Op))
     ScalarTy = CI->getOperand(0)->getType();
-  VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
+  VectorType *VecTy = VectorType::get(ScalarTy, IL.size());
 
   // If we have computed a smaller type for the expression, update VecTy so
   // that the costs will be accurate.
-  if (MinBWs.count(VL[0]))
+  if (MinBWs.count(E->Scalars[0]->Op))
     VecTy = VectorType::get(
-        IntegerType::get(F->getContext(), MinBWs[VL[0]].first), VL.size());
+        IntegerType::get(F->getContext(), MinBWs[E->Scalars[0]->Op].first),
+                         IL.size());
 
   unsigned ReuseShuffleNumbers = E->ReuseShuffleIndices.size();
   bool NeedToShuffleReuses = !E->ReuseShuffleIndices.empty();
   int ReuseShuffleCost = 0;
   if (NeedToShuffleReuses) {
     ReuseShuffleCost =
         TTI->getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, VecTy);
   }
   if (E->NeedToGather) {
-    if (allConstant(VL))
+    if (allConstant(IL))
       return 0;
-    if (isSplat(VL)) {
+    if (isSplat(IL)) {
       return ReuseShuffleCost +
              TTI->getShuffleCost(TargetTransformInfo::SK_Broadcast, VecTy, 0);
     }
-    if (getSameOpcode(VL).getOpcode() == Instruction::ExtractElement &&
-        allSameType(VL) && allSameBlock(VL)) {
+    if (IL[0]->MainOpcode == Instruction::ExtractElement && allSameBlock(IL)) {
+      ValueList VL;
+      for (auto *IOP : IL)
+        VL.push_back(IOP->Op);
       Optional<TargetTransformInfo::ShuffleKind> ShuffleKind = isShuffle(VL);
       if (ShuffleKind.hasValue()) {
         int Cost = TTI->getShuffleCost(ShuffleKind.getValue(), VecTy);
-        for (auto *V : VL) {
+        for (auto *IOP : IL) {
           // If all users of instruction are going to be vectorized and this
           // instruction itself is not going to be vectorized, consider this
           // instruction as dead and remove its cost from the final cost of the
           // vectorized tree.
-          if (areAllUsersVectorized(cast<Instruction>(V)) &&
-              !ScalarToTreeEntry.count(V)) {
+          if (areAllUsersVectorized(IOP->OpInst) &&
+              !ScalarToTreeEntry.count(IOP->OpInst)) {
             auto *IO = cast<ConstantInt>(
-                cast<ExtractElementInst>(V)->getIndexOperand());
+                cast<ExtractElementInst>(IOP->OpInst)->getIndexOperand());
             Cost -= TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy,
                                             IO->getZExtValue());
           }
         }
         return ReuseShuffleCost + Cost;
       }
     }
-    return ReuseShuffleCost + getGatherCost(VL);
+    return ReuseShuffleCost + getGatherCost(IL);
   }
-  InstructionsState S = getSameOpcode(VL);
-  assert(S.getOpcode() && allSameType(VL) && allSameBlock(VL) && "Invalid VL");
-  Instruction *VL0 = cast<Instruction>(S.OpValue);
-  unsigned ShuffleOrOp = S.isAltShuffle() ?
-               (unsigned) Instruction::ShuffleVector : S.getOpcode();
+  auto *VL0 = E->Scalars[0]->MainOp;
+  assert(IL[0]->MainOpcode != 0 && allSameBlock(IL) && "Invalid VL");
+  unsigned ShuffleOrOp = IL[0]->isAltShuffle() ?
+             (unsigned) Instruction::ShuffleVector : IL[0]->MainOpcode;
   switch (ShuffleOrOp) {
     case Instruction::PHI:
       return 0;
 
     case Instruction::ExtractValue:
     case Instruction::ExtractElement:
       if (NeedToShuffleReuses) {
         unsigned Idx = 0;
         for (unsigned I : E->ReuseShuffleIndices) {
           if (ShuffleOrOp == Instruction::ExtractElement) {
             auto *IO = cast<ConstantInt>(
-                cast<ExtractElementInst>(VL[I])->getIndexOperand());
+                cast<ExtractElementInst>(IL[I]->OpInst)->getIndexOperand());
             Idx = IO->getZExtValue();
             ReuseShuffleCost -= TTI->getVectorInstrCost(
                 Instruction::ExtractElement, VecTy, Idx);
           } else {
             ReuseShuffleCost -= TTI->getVectorInstrCost(
                 Instruction::ExtractElement, VecTy, Idx);
             ++Idx;
           }
         }
         Idx = ReuseShuffleNumbers;
-        for (Value *V : VL) {
+        for (auto *IOP : IL) {
           if (ShuffleOrOp == Instruction::ExtractElement) {
             auto *IO = cast<ConstantInt>(
-                cast<ExtractElementInst>(V)->getIndexOperand());
+                cast<ExtractElementInst>(IOP->OpInst)->getIndexOperand());
             Idx = IO->getZExtValue();
           } else {
             --Idx;
           }
           ReuseShuffleCost +=
               TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, Idx);
         }
       }
       if (!E->NeedToGather) {
         int DeadCost = ReuseShuffleCost;
         if (!E->ReorderIndices.empty()) {
           // TODO: Merge this shuffle with the ReuseShuffleCost.
           DeadCost += TTI->getShuffleCost(
               TargetTransformInfo::SK_PermuteSingleSrc, VecTy);
         }
-        for (unsigned i = 0, e = VL.size(); i < e; ++i) {
-          Instruction *E = cast<Instruction>(VL[i]);
+        for (unsigned i = 0, e = IL.size(); i < e; ++i) {
           // If all users are going to be vectorized, instruction can be
           // considered as dead.
           // The same, if have only one user, it will be vectorized for sure.
-          if (areAllUsersVectorized(E)) {
+          auto *I = IL[i]->OpInst;
+          if (areAllUsersVectorized(I)) {
             // Take credit for instruction that will become dead.
-            if (E->hasOneUse()) {
-              Instruction *Ext = E->user_back();
+            if (I->hasOneUse()) {
+              Instruction *Ext = I->user_back();
               if ((isa<SExtInst>(Ext) || isa<ZExtInst>(Ext)) &&
                   all_of(Ext->users(),
                          [](User *U) { return isa<GetElementPtrInst>(U); })) {
                 // Use getExtractWithExtendCost() to calculate the cost of
                 // extractelement/ext pair.
                 DeadCost -= TTI->getExtractWithExtendCost(
                     Ext->getOpcode(), Ext->getType(), VecTy, i);
                 // Add back the cost of s|zext which is subtracted seperately.
                 DeadCost += TTI->getCastInstrCost(
-                    Ext->getOpcode(), Ext->getType(), E->getType(), Ext);
+                    Ext->getOpcode(), Ext->getType(),
+                    I->getType(), Ext);
                 continue;
               }
             }
             DeadCost -=
                 TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, i);
           }
         }
         return DeadCost;
       }
-      return ReuseShuffleCost + getGatherCost(VL);
+      return ReuseShuffleCost + getGatherCost(IL);
 
     case Instruction::ZExt:
     case Instruction::SExt:
     case Instruction::FPToUI:
     case Instruction::FPToSI:
     case Instruction::FPExt:
     case Instruction::PtrToInt:
     case Instruction::IntToPtr:
     case Instruction::SIToFP:
     case Instruction::UIToFP:
     case Instruction::Trunc:
     case Instruction::FPTrunc:
     case Instruction::BitCast: {
       Type *SrcTy = VL0->getOperand(0)->getType();
       int ScalarEltCost =
-          TTI->getCastInstrCost(S.getOpcode(), ScalarTy, SrcTy, VL0);
+          TTI->getCastInstrCost(IL[0]->MainOpcode, ScalarTy, SrcTy, VL0);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
 
       // Calculate the cost of this instruction.
-      int ScalarCost = VL.size() * ScalarEltCost;
+      int ScalarCost = IL.size() * ScalarEltCost;
 
-      VectorType *SrcVecTy = VectorType::get(SrcTy, VL.size());
+      VectorType *SrcVecTy = VectorType::get(SrcTy, IL.size());
       int VecCost = 0;
       // Check if the values are candidates to demote.
       if (!MinBWs.count(VL0) || VecTy != SrcVecTy) {
         VecCost = ReuseShuffleCost +
-                  TTI->getCastInstrCost(S.getOpcode(), VecTy, SrcVecTy, VL0);
+                  TTI->getCastInstrCost(IL[0]->MainOpcode, VecTy,
+                                        SrcVecTy, VL0);
       }
       return VecCost - ScalarCost;
     }
     case Instruction::FCmp:
     case Instruction::ICmp:
     case Instruction::Select: {
       // Calculate the cost of this instruction.
-      int ScalarEltCost = TTI->getCmpSelInstrCost(S.getOpcode(), ScalarTy,
-                                                  Builder.getInt1Ty(), VL0);
+      int ScalarEltCost = TTI->getCmpSelInstrCost(IL[0]->MainOpcode,
+                                                  ScalarTy,
+                                                  Builder.getInt1Ty(),
+                                                  VL0);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
-      VectorType *MaskTy = VectorType::get(Builder.getInt1Ty(), VL.size());
+      VectorType *MaskTy = VectorType::get(Builder.getInt1Ty(), IL.size());
       int ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-      int VecCost = TTI->getCmpSelInstrCost(S.getOpcode(), VecTy, MaskTy, VL0);
+      int VecCost = TTI->getCmpSelInstrCost(IL[0]->MainOpcode, VecTy,
+                                            MaskTy, VL0);
       return ReuseShuffleCost + VecCost - ScalarCost;
     }
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
switch (ShuffleOrOp) {
     case Instruction::And:
     case Instruction::Or:
     case Instruction::Xor: {
       // Certain instructions can be cheaper to vectorize if they have a
       // constant second vector operand.
       TargetTransformInfo::OperandValueKind Op1VK =
           TargetTransformInfo::OK_AnyValue;
       TargetTransformInfo::OperandValueKind Op2VK =
-          TargetTransformInfo::OK_UniformConstantValue;
+          TargetTransformInfo::OK_AnyValue;
       TargetTransformInfo::OperandValueProperties Op1VP =
           TargetTransformInfo::OP_None;
       TargetTransformInfo::OperandValueProperties Op2VP =
           TargetTransformInfo::OP_PowerOf2;
 
       // If all operands are exactly the same ConstantInt then set the
       // operand kind to OK_UniformConstantValue.
       // If instead not all operands are constants, then set the operand kind
       // to OK_AnyValue. If all operands are constants but not the same,
       // then set the operand kind to OK_NonUniformConstantValue.
-      ConstantInt *CInt0 = nullptr;
-      for (unsigned i = 0, e = VL.size(); i < e; ++i) {
-        const Instruction *I = cast<Instruction>(VL[i]);
-        ConstantInt *CInt = dyn_cast<ConstantInt>(I->getOperand(1));
-        if (!CInt) {
+      if (auto *CInt = dyn_cast<ConstantInt>(IL[0]->MainOp->getOperand(1))) {
+        Op2VK = TargetTransformInfo::OK_UniformConstantValue;
+        const unsigned Opcode = IL[0]->MainOpcode;
+        for (auto *IOP : IL) {
+          auto *I = IOP->OpInst;
+          if (I == VL0 || Opcode != I->getOpcode())
+            continue;
+          if (!isa<ConstantInt>(I->getOperand(1))) {
             Op2VK = TargetTransformInfo::OK_AnyValue;
             Op2VP = TargetTransformInfo::OP_None;
             break;
           }
+          ConstantInt *CInt_cur = cast<ConstantInt>(I->getOperand(1));
+          if (Op2VK == TargetTransformInfo::OK_UniformConstantValue &&
+              CInt != cast<ConstantInt>(I->getOperand(1)))
+            Op2VK = TargetTransformInfo::OK_NonUniformConstantValue;
           if (Op2VP == TargetTransformInfo::OP_PowerOf2 &&
               !CInt->getValue().isPowerOf2())
             Op2VP = TargetTransformInfo::OP_None;
-        if (i == 0) {
-          CInt0 = CInt;
-          continue;
-        }
-        if (CInt0 != CInt)
+          if (CInt != CInt_cur)
             Op2VK = TargetTransformInfo::OK_NonUniformConstantValue;
         }
+      }
 
       SmallVector<const Value *, 4> Operands(VL0->operand_values());
       int ScalarEltCost = TTI->getArithmeticInstrCost(
-          S.getOpcode(), ScalarTy, Op1VK, Op2VK, Op1VP, Op2VP, Operands);
+          IL[0]->MainOpcode, ScalarTy, Op1VK, Op2VK, Op1VP, Op2VP, Operands);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
       int ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-      int VecCost = TTI->getArithmeticInstrCost(S.getOpcode(), VecTy, Op1VK,
-                                                Op2VK, Op1VP, Op2VP, Operands);
+      int VecCost = TTI->getArithmeticInstrCost(IL[0]->MainOpcode, VecTy,
+                                                Op1VK, Op2VK, Op1VP, Op2VP,
+                                                Operands);
       return ReuseShuffleCost + VecCost - ScalarCost;
     }
     case Instruction::GetElementPtr: {
       TargetTransformInfo::OperandValueKind Op1VK =
           TargetTransformInfo::OK_AnyValue;
       TargetTransformInfo::OperandValueKind Op2VK =
           TargetTransformInfo::OK_UniformConstantValue;
 
       int ScalarEltCost =
           TTI->getArithmeticInstrCost(Instruction::Add, ScalarTy, Op1VK, Op2VK);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
       int ScalarCost = VecTy->getNumElements() * ScalarEltCost;
       int VecCost =
           TTI->getArithmeticInstrCost(Instruction::Add, VecTy, Op1VK, Op2VK);
       return ReuseShuffleCost + VecCost - ScalarCost;
     }
     case Instruction::Load: {
       // Cost of wide load - cost of scalar loads.
       unsigned alignment = cast<LoadInst>(VL0)->getAlignment();
       int ScalarEltCost =
-          TTI->getMemoryOpCost(Instruction::Load, ScalarTy, alignment, 0, VL0);
+          TTI->getMemoryOpCost(Instruction::Load, ScalarTy, alignment, 0,
+                               VL0);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
       int ScalarLdCost = VecTy->getNumElements() * ScalarEltCost;
       int VecLdCost =
-          TTI->getMemoryOpCost(Instruction::Load, VecTy, alignment, 0, VL0);
+          TTI->getMemoryOpCost(Instruction::Load, VecTy, alignment, 0,
+                               VL0);
       if (!E->ReorderIndices.empty()) {
         // TODO: Merge this shuffle with the ReuseShuffleCost.
         VecLdCost += TTI->getShuffleCost(
             TargetTransformInfo::SK_PermuteSingleSrc, VecTy);
       }
       return ReuseShuffleCost + VecLdCost - ScalarLdCost;
     }
     case Instruction::Store: {
       // We know that we can merge the stores. Calculate the cost.
       unsigned alignment = cast<StoreInst>(VL0)->getAlignment();
       int ScalarEltCost =
           TTI->getMemoryOpCost(Instruction::Store, ScalarTy, alignment, 0, VL0);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
       int ScalarStCost = VecTy->getNumElements() * ScalarEltCost;
       int VecStCost =
           TTI->getMemoryOpCost(Instruction::Store, VecTy, alignment, 0, VL0);
       return ReuseShuffleCost + VecStCost - ScalarStCost;
     }
     case Instruction::Call: {
       CallInst *CI = cast<CallInst>(VL0);
       Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
 
       // Calculate the cost of the scalar and vector calls.
       SmallVector<Type *, 4> ScalarTys;
       for (unsigned op = 0, opc = CI->getNumArgOperands(); op != opc; ++op)
         ScalarTys.push_back(CI->getArgOperand(op)->getType());
 
       FastMathFlags FMF;
       if (auto *FPMO = dyn_cast<FPMathOperator>(CI))
         FMF = FPMO->getFastMathFlags();
 
       int ScalarEltCost =
           TTI->getIntrinsicInstrCost(ID, ScalarTy, ScalarTys, FMF);
       if (NeedToShuffleReuses) {
-        ReuseShuffleCost -= (ReuseShuffleNumbers - VL.size()) * ScalarEltCost;
+        ReuseShuffleCost -= (ReuseShuffleNumbers - IL.size()) * ScalarEltCost;
       }
       int ScalarCallCost = VecTy->getNumElements() * ScalarEltCost;
 
       SmallVector<Value *, 4> Args(CI->arg_operands());
       int VecCallCost = TTI->getIntrinsicInstrCost(ID, CI->getType(), Args, FMF,
                                                    VecTy->getNumElements());
 
       LLVM_DEBUG(dbgs() << "SLP: Call cost " << VecCallCost - ScalarCallCost
                         << " (" << VecCallCost << "-" << ScalarCallCost << ")"
                         << " for " << *CI << "\n");
 
       return ReuseShuffleCost + VecCallCost - ScalarCallCost;
     }
     case Instruction::ShuffleVector: {
-      assert(S.isAltShuffle() &&
-             ((Instruction::isBinaryOp(S.getOpcode()) &&
-               Instruction::isBinaryOp(S.getAltOpcode())) ||
-              (Instruction::isCast(S.getOpcode()) &&
-               Instruction::isCast(S.getAltOpcode()))) &&
+      assert(IL[0]->isAltShuffle() &&
+             ((Instruction::isBinaryOp(IL[0]->MainOpcode) &&
+             Instruction::isBinaryOp(IL[0]->AltOpcode)) ||
+             (Instruction::isCast(IL[0]->MainOpcode) &&
+             Instruction::isCast(IL[0]->AltOpcode))) &&
              "Invalid Shuffle Vector Operand");
       int ScalarCost = 0;
       if (NeedToShuffleReuses) {
-        for (unsigned Idx : E->ReuseShuffleIndices) {
-          Instruction *I = cast<Instruction>(VL[Idx]);
+        for (unsigned Idx : E->ReuseShuffleIndices)
           ReuseShuffleCost -= TTI->getInstructionCost(
-              I, TargetTransformInfo::TCK_RecipThroughput);
-        }
-        for (Value *V : VL) {
-          Instruction *I = cast<Instruction>(V);
+              IL[Idx]->OpInst, TargetTransformInfo::TCK_RecipThroughput);
+        for (auto *IOP : IL)
           ReuseShuffleCost += TTI->getInstructionCost(
-              I, TargetTransformInfo::TCK_RecipThroughput);
-        }
+              IOP->OpInst, TargetTransformInfo::TCK_RecipThroughput);
       }
-      for (Value *i : VL) {
-        Instruction *I = cast<Instruction>(i);
-        assert(S.isOpcodeOrAlt(I) && "Unexpected main/alternate opcode");
+      for (auto *IOP : IL)
         ScalarCost += TTI->getInstructionCost(
-            I, TargetTransformInfo::TCK_RecipThroughput);
-      }
+            IOP->OpInst, TargetTransformInfo::TCK_RecipThroughput);
       // VecCost is equal to sum of the cost of creating 2 vectors
       // and the cost of creating shuffle.
       int VecCost = 0;
-      if (Instruction::isBinaryOp(S.getOpcode())) {
-        VecCost = TTI->getArithmeticInstrCost(S.getOpcode(), VecTy);
-        VecCost += TTI->getArithmeticInstrCost(S.getAltOpcode(), VecTy);
+      if (Instruction::isBinaryOp(IL[0]->MainOpcode)) {
+        VecCost = TTI->getArithmeticInstrCost(IL[0]->MainOpcode, VecTy);
+        VecCost += TTI->getArithmeticInstrCost(IL[0]->AltOpcode, VecTy);
       } else {
-        Type *Src0SclTy = S.MainOp->getOperand(0)->getType();
-        Type *Src1SclTy = S.AltOp->getOperand(0)->getType();
-        VectorType *Src0Ty = VectorType::get(Src0SclTy, VL.size());
-        VectorType *Src1Ty = VectorType::get(Src1SclTy, VL.size());
-        VecCost = TTI->getCastInstrCost(S.getOpcode(), VecTy, Src0Ty);
-        VecCost += TTI->getCastInstrCost(S.getAltOpcode(), VecTy, Src1Ty);
+        Type *Src0SclTy = IL[0]->MainOp->getOperand(0)->getType();
+        Type *Src1SclTy = IL[0]->AltOp->getOperand(0)->getType();
+        VectorType *Src0Ty = VectorType::get(Src0SclTy, IL.size());
+        VectorType *Src1Ty = VectorType::get(Src1SclTy, IL.size());
+        VecCost = TTI->getCastInstrCost(IL[0]->MainOpcode, VecTy, Src0Ty);
+        VecCost += TTI->getCastInstrCost(IL[0]->AltOpcode, VecTy,
+                                         Src1Ty);
       }
       VecCost += TTI->getShuffleCost(TargetTransformInfo::SK_Select, VecTy, 0);
       return ReuseShuffleCost + VecCost - ScalarCost;
     }
     default:
       llvm_unreachable("Unknown instruction");
   }
 }
 
 bool BoUpSLP::isFullyVectorizableTinyTree() {
   LLVM_DEBUG(dbgs() << "SLP: Check whether the tree with height "
                     << VectorizableTree.size() << " is fully vectorizable .\n");
 
   // We only handle trees of heights 1 and 2.
   if (VectorizableTree.size() == 1 && !VectorizableTree[0].NeedToGather)
     return true;
 
   if (VectorizableTree.size() != 2)
     return false;
 
   // Handle splat and all-constants stores.
   if (!VectorizableTree[0].NeedToGather &&
       (allConstant(VectorizableTree[1].Scalars) ||
-       isSplat(VectorizableTree[1].Scalars)))
     return true;
 
   // Gathering cost would be too much for tiny trees.
   if (VectorizableTree[0].NeedToGather || VectorizableTree[1].NeedToGather)
     return false;
 
   return true;
 }
 
+Instruction *BoUpSLP::propagateMetadata(Instruction *I,
+                                        ArrayRef<InstructionOrPseudo *> IL) {
+  ValueList VL;
+  for (auto *IOP: IL)
+    VL.push_back(IOP->Op);
+  return llvm::propagateMetadata(I, VL);
+}
+
 bool BoUpSLP::isTreeTinyAndNotFullyVectorizable() {
   // We can vectorize the tree if its size is greater than or equal to the
   // minimum size specified by the MinTreeSize command line option.
   if (VectorizableTree.size() >= MinTreeSize)
     return false;
 
   // If we have a tiny tree (a tree whose size is less than MinTreeSize), we
   // can vectorize it if we can prove it fully vectorizable.
int BoUpSLP::getSpillCost() {
   // (for example, if spills and fills are required).
   unsigned BundleWidth = VectorizableTree.front().Scalars.size();
   int Cost = 0;
 
   SmallPtrSet<Instruction*, 4> LiveValues;
   Instruction *PrevInst = nullptr;
 
   for (const auto &N : VectorizableTree) {
-    Instruction *Inst = dyn_cast<Instruction>(N.Scalars[0]);
+    Instruction *Inst = N.Scalars[0]->MainOp;
     if (!Inst)
       continue;
 
     if (!PrevInst) {
       PrevInst = Inst;
       continue;
     }
 
     // Update LiveValues.
     LiveValues.erase(PrevInst);
     for (auto &J : PrevInst->operands()) {
-      if (isa<Instruction>(&*J) && getTreeEntry(&*J))
-        LiveValues.insert(cast<Instruction>(&*J));
+      auto *I = dyn_cast<Instruction>(&*J);
+      if (I && getTreeEntry(I))
+        LiveValues.insert(I);
     }
 
     LLVM_DEBUG({
       dbgs() << "SLP: #LV: " << LiveValues.size();
       for (auto *X : LiveValues)
         dbgs() << " " << X->getName();
       dbgs() << ", Looking at ";
       Inst->dump();
if (TE.NeedToGather &&
         std::any_of(std::next(VectorizableTree.begin(), I + 1),
                     VectorizableTree.end(), [TE](TreeEntry &Entry) {
                       return Entry.NeedToGather && Entry.isSame(TE.Scalars);
                     }))
       continue;
 
     int C = getEntryCost(&TE);
     LLVM_DEBUG(dbgs() << "SLP: Adding cost " << C
-                      << " for bundle that starts with " << *TE.Scalars[0]
+                      << " for bundle that starts with " << *TE.Scalars[0]->Op
                       << ".\n");
     Cost += C;
   }
 
   SmallPtrSet<Value *, 16> ExtractCostCalculated;
   int ExtractCost = 0;
   for (ExternalUser &EU : ExternalUses) {
     // We only add extract cost once for the same scalar.
     if (!ExtractCostCalculated.insert(EU.Scalar).second)
       continue;
 
     // Uses by ephemeral values are free (because the ephemeral value will be
     // removed prior to code generation, and so the extraction will be
     // removed as well).
     if (EphValues.count(EU.User))
       continue;
 
     // If we plan to rewrite the tree in a smaller type, we will need to sign
     // extend the extracted value back to the original type. Here, we account
     // for the extract and the added cost of the sign extend if needed.
     auto *VecTy = VectorType::get(EU.Scalar->getType(), BundleWidth);
-    auto *ScalarRoot = VectorizableTree[0].Scalars[0];
+    auto *ScalarRoot = VectorizableTree[0].Scalars[0]->Op;
     if (MinBWs.count(ScalarRoot)) {
       auto *MinTy = IntegerType::get(F->getContext(), MinBWs[ScalarRoot].first);
       auto Extend =
           MinBWs[ScalarRoot].second ? Instruction::SExt : Instruction::ZExt;
       VecTy = VectorType::get(MinTy, BundleWidth);
       ExtractCost += TTI->getExtractWithExtendCost(Extend, EU.Scalar->getType(),
                                                    VecTy, EU.Lane);
     } else {
int BoUpSLP::getGatherCost(Type *Ty,
   for (unsigned i = 0, e = cast<VectorType>(Ty)->getNumElements(); i < e; ++i)
     if (!ShuffledIndices.count(i))
       Cost += TTI->getVectorInstrCost(Instruction::InsertElement, Ty, i);
   if (!ShuffledIndices.empty())
       Cost += TTI->getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, Ty);
   return Cost;
 }
 
-int BoUpSLP::getGatherCost(ArrayRef<Value *> VL) {
-  // Find the type of the operands in VL.
-  Type *ScalarTy = VL[0]->getType();
-  if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
+int BoUpSLP::getGatherCost(ArrayRef<InstructionOrPseudo *> IL) {
+  // Find the type of the operands in IL.
+  Type *ScalarTy = IL[0]->Op->getType();
+  if (StoreInst *SI = dyn_cast<StoreInst>(IL[0]->Op))
     ScalarTy = SI->getValueOperand()->getType();
-  VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
+  VectorType *VecTy = VectorType::get(ScalarTy, IL.size());
   // Find the cost of inserting/extracting values from the vector.
   // Check if the same elements are inserted several times and count them as
   // shuffle candidates.
   DenseSet<unsigned> ShuffledElements;
   DenseSet<Value *> UniqueElements;
   // Iterate in reverse order to consider insert elements with the high cost.
-  for (unsigned I = VL.size(); I > 0; --I) {
+  for (unsigned I = IL.size(); I > 0; --I) {
     unsigned Idx = I - 1;
-    if (!UniqueElements.insert(VL[Idx]).second)
+    if (!UniqueElements.insert(IL[Idx]->Op).second)
       ShuffledElements.insert(Idx);
   }
   return getGatherCost(VecTy, ShuffledElements);
 }
 
 // Reorder commutative operations in alternate shuffle if the resulting vectors
 // are consecutive loads. This would allow us to vectorize the tree.
 // If we have something like-
 // load a[0] - load b[0]
 // load b[1] + load a[1]
 // load a[2] - load b[2]
 // load a[3] + load b[3]
 // Reordering the second load b[1]  load a[1] would allow us to vectorize this
 // code.
-void BoUpSLP::reorderAltShuffleOperands(const InstructionsState &S,
-                                        ArrayRef<Value *> VL,
+void BoUpSLP::reorderAltShuffleOperands(ArrayRef<InstructionOrPseudo *> IL,
                                         SmallVectorImpl<Value *> &Left,
                                         SmallVectorImpl<Value *> &Right) {
   // Push left and right operands of binary operation into Left and Right
-  for (Value *V : VL) {
-    auto *I = cast<Instruction>(V);
-    assert(S.isOpcodeOrAlt(I) && "Incorrect instruction in vector");
-    Left.push_back(I->getOperand(0));
-    Right.push_back(I->getOperand(1));
+  for (auto *Inst : IL) {
+    if (Inst->isOpcodeOrAlt()) {
+      Left.push_back(Inst->OpInst->getOperand(0));
+      Right.push_back(Inst->OpInst->getOperand(1));
+    } else {
+      Left.push_back(Inst->OpInst);
+      Right.push_back(ConstantExpr::getBinOpIdentity(Inst->MainOpcode,
+                                                     Inst->OpInst->getType(),
+                                                     true));
+    }
   }
 
   // Reorder if we have a commutative operation and consecutive access
   // are on either side of the alternate instructions.
-  for (unsigned j = 0; j < VL.size() - 1; ++j) {
+  for (unsigned j = 0; j < IL.size() - 1; ++j) {
     if (LoadInst *L = dyn_cast<LoadInst>(Left[j])) {
       if (LoadInst *L1 = dyn_cast<LoadInst>(Right[j + 1])) {
-        Instruction *VL1 = cast<Instruction>(VL[j]);
-        Instruction *VL2 = cast<Instruction>(VL[j + 1]);
+        Instruction *VL1 = IL[j]->OpInst;
+        Instruction *VL2 = IL[j + 1]->OpInst;
         if (VL1->isCommutative() && isConsecutiveAccess(L, L1, *DL, *SE)) {
           std::swap(Left[j], Right[j]);
           continue;
         } else if (VL2->isCommutative() &&
                    isConsecutiveAccess(L, L1, *DL, *SE)) {
           std::swap(Left[j + 1], Right[j + 1]);
           continue;
         }
         // else unchanged
       }
     }
     if (LoadInst *L = dyn_cast<LoadInst>(Right[j])) {
       if (LoadInst *L1 = dyn_cast<LoadInst>(Left[j + 1])) {
-        Instruction *VL1 = cast<Instruction>(VL[j]);
-        Instruction *VL2 = cast<Instruction>(VL[j + 1]);
+        Instruction *VL1 = IL[j]->OpInst;
+        Instruction *VL2 = IL[j + 1]->OpInst;
         if (VL1->isCommutative() && isConsecutiveAccess(L, L1, *DL, *SE)) {
           std::swap(Left[j], Right[j]);
           continue;
         } else if (VL2->isCommutative() &&
                    isConsecutiveAccess(L, L1, *DL, *SE)) {
           std::swap(Left[j + 1], Right[j + 1]);
           continue;
         }
         // else unchanged
       }
     }
   }
 }
 
 // Return true if I should be commuted before adding it's left and right
 // operands to the arrays Left and Right.
 //
 // The vectorizer is trying to either have all elements one side being
 // instruction with the same opcode to enable further vectorization, or having
 // a splat to lower the vectorizing cost.
 static bool shouldReorderOperands(
     int i, unsigned Opcode, Instruction &I, ArrayRef<Value *> Left,
     ArrayRef<Value *> Right, bool AllSameOpcodeLeft, bool AllSameOpcodeRight,
     bool SplatLeft, bool SplatRight, Value *&VLeft, Value *&VRight) {
+  if (I.getOpcode() == Opcode) {
     VLeft = I.getOperand(0);
     VRight = I.getOperand(1);
+  } else {
+    VLeft = &I;
+    VRight = ConstantExpr::getBinOpIdentity(Opcode, I.getType(), true);
+  }
   // If we have "SplatRight", try to see if commuting is needed to preserve it.
   if (SplatRight) {
     if (VRight == Right[i - 1])
       // Preserve SplatRight
       return false;
     if (VLeft == Right[i - 1]) {
       // Commuting would preserve SplatRight, but we don't want to break
       // SplatLeft either, i.e. preserve the original order if possible.
if (ILeft && LeftPrevOpcode == ILeft->getOpcode())
       return false;
     if (IRight && LeftPrevOpcode == IRight->getOpcode())
       return true;
   }
   return false;
 }
 
 void BoUpSLP::reorderInputsAccordingToOpcode(unsigned Opcode,
-                                             ArrayRef<Value *> VL,
+                                             ArrayRef<InstructionOrPseudo *> IL,
                                              SmallVectorImpl<Value *> &Left,
                                              SmallVectorImpl<Value *> &Right) {
-  if (!VL.empty()) {
+  if (!IL.empty()) {
     // Peel the first iteration out of the loop since there's nothing
     // interesting to do anyway and it simplifies the checks in the loop.
-    auto *I = cast<Instruction>(VL[0]);
-    Value *VLeft = I->getOperand(0);
-    Value *VRight = I->getOperand(1);
+    Value *VLeft;
+    Value *VRight;
+    if (IL[0]->OpInst->getOpcode() == Opcode) {
+      VLeft = IL[0]->OpInst->getOperand(0);
+      VRight = IL[0]->OpInst->getOperand(1);
+    } else {
+      VLeft = IL[0]->OpInst;
+      VRight = ConstantExpr::getBinOpIdentity(Opcode, IL[0]->OpInst->getType(),
+                                              true);
+    }
     if (!isa<Instruction>(VRight) && isa<Instruction>(VLeft))
       // Favor having instruction to the right. FIXME: why?
       std::swap(VLeft, VRight);
     Left.push_back(VLeft);
     Right.push_back(VRight);
   }
 
   // Keep track if we have instructions with all the same opcode on one side.
   bool AllSameOpcodeLeft = isa<Instruction>(Left[0]);
   bool AllSameOpcodeRight = isa<Instruction>(Right[0]);
   // Keep track if we have one side with all the same value (broadcast).
   bool SplatLeft = true;
   bool SplatRight = true;
 
-  for (unsigned i = 1, e = VL.size(); i != e; ++i) {
-    Instruction *I = cast<Instruction>(VL[i]);
+  for (unsigned i = 1, e = IL.size(); i != e; ++i) {
+    Instruction *I = IL[i]->OpInst;
     assert(((I->getOpcode() == Opcode && I->isCommutative()) ||
             (I->getOpcode() != Opcode && Instruction::isCommutative(Opcode))) &&
            "Can only process commutative instruction");
     // Commute to favor either a splat or maximizing having the same opcodes on
     // one side.
     Value *VLeft;
     Value *VRight;
     if (shouldReorderOperands(i, Opcode, *I, Left, Right, AllSameOpcodeLeft,
void BoUpSLP::reorderInputsAccordingToOpcode(unsigned Opcode,
   // Reordering the second load b[1]  load a[1] would allow us to vectorize
   // this code and we still retain AllSameOpcode property.
   // FIXME: This load reordering might break AllSameOpcode in some rare cases
   // such as-
   // add a[0],c[0]  load b[0]
   // add a[1],c[2]  load b[1]
   // b[2]           load b[2]
   // add a[3],c[3]  load b[3]
-  for (unsigned j = 0, e = VL.size() - 1; j < e; ++j) {
+  for (unsigned j = 0, e = IL.size() - 1; j < e; ++j) {
     if (LoadInst *L = dyn_cast<LoadInst>(Left[j])) {
       if (LoadInst *L1 = dyn_cast<LoadInst>(Right[j + 1])) {
         if (isConsecutiveAccess(L, L1, *DL, *SE)) {
           std::swap(Left[j + 1], Right[j + 1]);
           continue;
         }
       }
     }
     if (LoadInst *L = dyn_cast<LoadInst>(Right[j])) {
       if (LoadInst *L1 = dyn_cast<LoadInst>(Left[j + 1])) {
         if (isConsecutiveAccess(L, L1, *DL, *SE)) {
           std::swap(Left[j + 1], Right[j + 1]);
           continue;
         }
       }
     }
     // else unchanged
   }
 }
 
-void BoUpSLP::setInsertPointAfterBundle(ArrayRef<Value *> VL,
-                                        const InstructionsState &S) {
+void BoUpSLP::setInsertPointAfterBundle(ArrayRef<InstructionOrPseudo *> IL) {
   // Get the basic block this bundle is in. All instructions in the bundle
   // should be in this block.
-  auto *Front = cast<Instruction>(S.OpValue);
+  auto *Front = IL[0]->MainOp;
   auto *BB = Front->getParent();
-  assert(llvm::all_of(make_range(VL.begin(), VL.end()), [=](Value *V) -> bool {
-    auto *I = cast<Instruction>(V);
-    return !S.isOpcodeOrAlt(I) || I->getParent() == BB;
+  assert(llvm::all_of(
+      make_range(IL.begin(), IL.end()), [=](InstructionOrPseudo *I) -> bool {
+        return !I->isOpcodeOrAlt() || I->OpInst->getParent() == BB;
       }));
 
   // The last instruction in the bundle in program order.
   Instruction *LastInst = nullptr;
 
-  // Find the last instruction. The common case should be that BB has been
-  // scheduled, and the last instruction is VL.back(). So we start with
-  // VL.back() and iterate over schedule data until we reach the end of the
-  // bundle. The end of the bundle is marked by null ScheduleData.
+  // Find the last instruction. If the bundle is not scheduled then
+  // the first in the bundle is the last one in BB, because we discover
+  // bundles in backward walk.
   if (BlocksSchedules.count(BB)) {
-    auto *Bundle =
-        BlocksSchedules[BB]->getScheduleData(isOneOf(S, VL.back()));
+    BlockScheduling *BS = BlocksSchedules[BB].get();
+    auto *Bundle = BS->getScheduleData(IL[0]->MainOp, IL[0]->getKey());
     if (Bundle && Bundle->isPartOfBundle())
-      for (; Bundle; Bundle = Bundle->NextInBundle)
-        if (Bundle->OpValue == Bundle->Inst)
-          LastInst = Bundle->Inst;
+      LastInst = Bundle->FirstInBundle->Inst;
   }
 
   // LastInst can still be null at this point if there's either not an entry
   // for BB in BlocksSchedules or there's no ScheduleData available for
   // VL.back(). This can be the case if buildTree_rec aborts for various
   // reasons (e.g., the maximum recursion depth is reached, the maximum region
   // size is reached, etc.). ScheduleData is initialized in the scheduling
   // "dry-run".
   //
   // If this happens, we can still find the last instruction by brute force. We
   // iterate forwards from Front (inclusive) until we either see all
   // instructions in the bundle or reach the end of the block. If Front is the
   // last instruction in program order, LastInst will be set to Front, and we
   // will visit all the remaining instructions in the block.
   //
   // One of the reasons we exit early from buildTree_rec is to place an upper
   // bound on compile-time. Thus, taking an additional compile-time hit here is
   // not ideal. However, this should be exceedingly rare since it requires that
   // we both exit early from buildTree_rec and that the bundle be out-of-order
   // (causing us to iterate all the way to the end of the block).
   if (!LastInst) {
-    SmallPtrSet<Value *, 16> Bundle(VL.begin(), VL.end());
+    SmallPtrSet<Value *, 16> Bundle;
+    for (auto *IOP : IL)
+      Bundle.insert(IOP->Op);
     for (auto &I : make_range(BasicBlock::iterator(Front), BB->end())) {
-      if (Bundle.erase(&I) && S.isOpcodeOrAlt(&I))
+      if (Bundle.erase(&I))
         LastInst = &I;
       if (Bundle.empty())
         break;
     }
   }
 
   // Set the insertion point after the last instruction in the bundle. Set the
   // debug location to Front.
   Builder.SetInsertPoint(BB, ++LastInst->getIterator());
   Builder.SetCurrentDebugLocation(Front->getDebugLoc());
 }
 
-Value *BoUpSLP::Gather(ArrayRef<Value *> VL, VectorType *Ty) {
+Value *BoUpSLP::Gather(ArrayRef<InstructionOrPseudo *> IL, VectorType *Ty) {
   Value *Vec = UndefValue::get(Ty);
   // Generate the 'InsertElement' instruction.
   for (unsigned i = 0; i < Ty->getNumElements(); ++i) {
-    Vec = Builder.CreateInsertElement(Vec, VL[i], Builder.getInt32(i));
+    Vec = Builder.CreateInsertElement(Vec, IL[i]->Op, Builder.getInt32(i));
     if (Instruction *Insrt = dyn_cast<Instruction>(Vec)) {
       GatherSeq.insert(Insrt);
       CSEBlocks.insert(Insrt->getParent());
 
       // Add to our 'need-to-extract' list.
-      if (TreeEntry *E = getTreeEntry(VL[i])) {
+      if (TreeEntry *E = getTreeEntry(IL[i]->OpInst)) {
         // Find which lane we need to extract.
         int FoundLane = -1;
         for (unsigned Lane = 0, LE = E->Scalars.size(); Lane != LE; ++Lane) {
           // Is this the lane of the scalar that we are looking for ?
-          if (E->Scalars[Lane] == VL[i]) {
+          if (E->Scalars[Lane]->Op == IL[i]->Op) {
             FoundLane = Lane;
             break;
           }
         }
         assert(FoundLane >= 0 && "Could not find the correct lane");
         if (!E->ReuseShuffleIndices.empty()) {
           FoundLane =
               std::distance(E->ReuseShuffleIndices.begin(),
                             llvm::find(E->ReuseShuffleIndices, FoundLane));
         }
-        ExternalUses.push_back(ExternalUser(VL[i], Insrt, FoundLane));
+        ExternalUses.push_back(ExternalUser(IL[i]->Op, Insrt, FoundLane));
       }
     }
   }
 
   return Vec;
 }
 
-Value *BoUpSLP::vectorizeTree(ArrayRef<Value *> VL) {
+Value *BoUpSLP::vectorizeTree(ArrayRef<Value *> VL, Value *Parent) {
+  IOPList IL;
   InstructionsState S = getSameOpcode(VL);
-  if (S.getOpcode()) {
-    if (TreeEntry *E = getTreeEntry(S.OpValue)) {
-      if (E->isSame(VL)) {
+  for (Value *V: VL)
+    IL.push_back(allocInstructionOrPseudo(V, Parent, S));
+
+  if (IL[0]->MainOp) {
+    TreeEntry *E = getTreeEntry(IL[0]->MainOp);
+    if (E && E->isSame(IL)) {
         Value *V = vectorizeTree(E);
         if (VL.size() == E->Scalars.size() && !E->ReuseShuffleIndices.empty()) {
           // We need to get the vectorized value but without shuffle.
           if (auto *SV = dyn_cast<ShuffleVectorInst>(V)) {
             V = SV->getOperand(0);
           } else {
             // Reshuffle to get only unique values.
             SmallVector<unsigned, 4> UniqueIdxs;
             SmallSet<unsigned, 4> UsedIdxs;
             for(unsigned Idx : E->ReuseShuffleIndices)
               if (UsedIdxs.insert(Idx).second)
                 UniqueIdxs.emplace_back(Idx);
             V = Builder.CreateShuffleVector(V, UndefValue::get(V->getType()),
                                             UniqueIdxs);
           }
         }
         return V;
-      }
-    }
-  }
 
   Type *ScalarTy = S.OpValue->getType();
   if (StoreInst *SI = dyn_cast<StoreInst>(S.OpValue))
     ScalarTy = SI->getValueOperand()->getType();
 
   // Check that every instruction appears once in this bundle.
   SmallVector<unsigned, 4> ReuseShuffleIndicies;
-  SmallVector<Value *, 4> UniqueValues;
-  if (VL.size() > 2) {
+  IOPList UniqueValues;
+  if (IL.size() > 2) {
     DenseMap<Value *, unsigned> UniquePositions;
-    for (Value *V : VL) {
-      auto Res = UniquePositions.try_emplace(V, UniqueValues.size());
+    for (auto *I : IL) {
+      auto Res = UniquePositions.try_emplace(I->Op, UniqueValues.size());
       ReuseShuffleIndicies.emplace_back(Res.first->second);
-      if (Res.second || isa<Constant>(V))
-        UniqueValues.emplace_back(V);
+      if (Res.second || isa<Constant>(I->Op))
+        UniqueValues.emplace_back(I);
     }
     // Do not shuffle single element or if number of unique values is not power
     // of 2.
-    if (UniqueValues.size() == VL.size() || UniqueValues.size() <= 1 ||
+    if (UniqueValues.size() == IL.size() || UniqueValues.size() <= 1 ||
         !llvm::isPowerOf2_32(UniqueValues.size()))
       ReuseShuffleIndicies.clear();
     else
-      VL = UniqueValues;
+      IL = UniqueValues;
   }
-  VectorType *VecTy = VectorType::get(ScalarTy, VL.size());
+  VectorType *VecTy = VectorType::get(ScalarTy, IL.size());
 
-  Value *V = Gather(VL, VecTy);
+  Value *V = Gather(IL, VecTy);
   if (!ReuseShuffleIndicies.empty()) {
     V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                     ReuseShuffleIndicies, "shuffle");
     if (auto *I = dyn_cast<Instruction>(V)) {
       GatherSeq.insert(I);
       CSEBlocks.insert(I->getParent());
     }
   }
   return V;
 }
 
 static void inversePermutation(ArrayRef<unsigned> Indices,
                                SmallVectorImpl<unsigned> &Mask) {
   Mask.clear();
   const unsigned E = Indices.size();
   Mask.resize(E);
   for (unsigned I = 0; I < E; ++I)
     Mask[Indices[I]] = I;
 }
 
 Value *BoUpSLP::vectorizeTree(TreeEntry *E) {
   IRBuilder<>::InsertPointGuard Guard(Builder);
 
   if (E->VectorizedValue) {
-    LLVM_DEBUG(dbgs() << "SLP: Diamond merged for " << *E->Scalars[0] << ".\n");
+    LLVM_DEBUG(dbgs() << "SLP: Diamond merged for "
+                      << *E->Scalars[0]->MainOp << ".\n");
     return E->VectorizedValue;
   }
 
-  InstructionsState S = getSameOpcode(E->Scalars);
-  Instruction *VL0 = cast<Instruction>(S.OpValue);
+  auto *VL0 = cast<Instruction>(E->Scalars[0]->MainOp);
   Type *ScalarTy = VL0->getType();
   if (StoreInst *SI = dyn_cast<StoreInst>(VL0))
     ScalarTy = SI->getValueOperand()->getType();
   VectorType *VecTy = VectorType::get(ScalarTy, E->Scalars.size());
 
   bool NeedToShuffleReuses = !E->ReuseShuffleIndices.empty();
 
   if (E->NeedToGather) {
-    setInsertPointAfterBundle(E->Scalars, S);
+    setInsertPointAfterBundle(E->Scalars);
     auto *V = Gather(E->Scalars, VecTy);
     if (NeedToShuffleReuses) {
       V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                       E->ReuseShuffleIndices, "shuffle");
       if (auto *I = dyn_cast<Instruction>(V)) {
         GatherSeq.insert(I);
         CSEBlocks.insert(I->getParent());
       }
     }
     E->VectorizedValue = V;
     return V;
   }
 
-  unsigned ShuffleOrOp = S.isAltShuffle() ?
-           (unsigned) Instruction::ShuffleVector : S.getOpcode();
+  unsigned ShuffleOrOp = E->Scalars[0]->isAltShuffle() ?
+           (unsigned) Instruction::ShuffleVector : E->Scalars[0]->MainOpcode;
   switch (ShuffleOrOp) {
     case Instruction::PHI: {
       PHINode *PH = dyn_cast<PHINode>(VL0);
       Builder.SetInsertPoint(PH->getParent()->getFirstNonPHI());
       Builder.SetCurrentDebugLocation(PH->getDebugLoc());
       PHINode *NewPhi = Builder.CreatePHI(VecTy, PH->getNumIncomingValues());
       Value *V = NewPhi;
       if (NeedToShuffleReuses) {
case Instruction::PHI: {
         BasicBlock *IBB = PH->getIncomingBlock(i);
 
         if (!VisitedBBs.insert(IBB).second) {
           NewPhi->addIncoming(NewPhi->getIncomingValueForBlock(IBB), IBB);
           continue;
         }
 
         // Prepare the operand vector.
-        for (Value *V : E->Scalars)
-          Operands.push_back(cast<PHINode>(V)->getIncomingValueForBlock(IBB));
+        for (auto *IOP : E->Scalars)
+          Operands.push_back(
+              cast<PHINode>(IOP->Op)->getIncomingValueForBlock(IBB));
 
         Builder.SetInsertPoint(IBB->getTerminator());
         Builder.SetCurrentDebugLocation(PH->getDebugLoc());
-        Value *Vec = vectorizeTree(Operands);
+        Value *Vec = vectorizeTree(Operands, E->Scalars[0]->MainOp);
         NewPhi->addIncoming(Vec, IBB);
       }
 
       assert(NewPhi->getNumIncomingValues() == PH->getNumIncomingValues() &&
              "Invalid number of incoming values");
       return V;
     }
 
case Instruction::ExtractElement: {
           if (E->ReorderIndices.empty())
             Builder.SetInsertPoint(VL0);
           V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                           E->ReuseShuffleIndices, "shuffle");
         }
         E->VectorizedValue = V;
         return V;
       }
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
       auto *V = Gather(E->Scalars, VecTy);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
         if (auto *I = dyn_cast<Instruction>(V)) {
           GatherSeq.insert(I);
           CSEBlocks.insert(I->getParent());
         }
case Instruction::ExtractValue: {
         if (NeedToShuffleReuses) {
           // TODO: Merge this shuffle with the ReorderShuffleMask.
           NewV = Builder.CreateShuffleVector(
               NewV, UndefValue::get(VecTy), E->ReuseShuffleIndices, "shuffle");
         }
         E->VectorizedValue = NewV;
         return NewV;
       }
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
       auto *V = Gather(E->Scalars, VecTy);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
         if (auto *I = dyn_cast<Instruction>(V)) {
           GatherSeq.insert(I);
           CSEBlocks.insert(I->getParent());
         }
switch (ShuffleOrOp) {
     case Instruction::PtrToInt:
     case Instruction::IntToPtr:
     case Instruction::SIToFP:
     case Instruction::UIToFP:
     case Instruction::Trunc:
     case Instruction::FPTrunc:
     case Instruction::BitCast: {
       ValueList INVL;
-      for (Value *V : E->Scalars)
-        INVL.push_back(cast<Instruction>(V)->getOperand(0));
+      for (auto *IOP : E->Scalars)
+        INVL.push_back(IOP->OpInst->getOperand(0));
 
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
 
-      Value *InVec = vectorizeTree(INVL);
+      Value *InVec = vectorizeTree(INVL, E->Scalars[0]->MainOp);
 
       if (E->VectorizedValue) {
         LLVM_DEBUG(dbgs() << "SLP: Diamond merged for " << *VL0 << ".\n");
         return E->VectorizedValue;
       }
 
       CastInst *CI = dyn_cast<CastInst>(VL0);
       Value *V = Builder.CreateCast(CI->getOpcode(), InVec, VecTy);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
       ++NumVectorInstructions;
       return V;
     }
     case Instruction::FCmp:
     case Instruction::ICmp: {
       ValueList LHSV, RHSV;
-      for (Value *V : E->Scalars) {
-        LHSV.push_back(cast<Instruction>(V)->getOperand(0));
-        RHSV.push_back(cast<Instruction>(V)->getOperand(1));
+      for (auto *IOP : E->Scalars) {
+        LHSV.push_back(IOP->OpInst->getOperand(0));
+        RHSV.push_back(IOP->OpInst->getOperand(1));
       }
 
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
 
-      Value *L = vectorizeTree(LHSV);
-      Value *R = vectorizeTree(RHSV);
+      Value *L = vectorizeTree(LHSV, E->Scalars[0]->MainOp);
+      Value *R = vectorizeTree(RHSV, E->Scalars[0]->MainOp);
 
       if (E->VectorizedValue) {
         LLVM_DEBUG(dbgs() << "SLP: Diamond merged for " << *VL0 << ".\n");
         return E->VectorizedValue;
       }
 
       CmpInst::Predicate P0 = cast<CmpInst>(VL0)->getPredicate();
       Value *V;
-      if (S.getOpcode() == Instruction::FCmp)
+      if (E->Scalars[0]->MainOpcode == Instruction::FCmp)
         V = Builder.CreateFCmp(P0, L, R);
       else
         V = Builder.CreateICmp(P0, L, R);
 
-      propagateIRFlags(V, E->Scalars, VL0);
+      ValueList VL;
+      for (auto *IOP: E->Scalars)
+        VL.push_back(IOP->Op);
+      propagateIRFlags(V, VL, VL0);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
       ++NumVectorInstructions;
       return V;
     }
     case Instruction::Select: {
       ValueList TrueVec, FalseVec, CondVec;
-      for (Value *V : E->Scalars) {
-        CondVec.push_back(cast<Instruction>(V)->getOperand(0));
-        TrueVec.push_back(cast<Instruction>(V)->getOperand(1));
-        FalseVec.push_back(cast<Instruction>(V)->getOperand(2));
+      for (auto *IOP : E->Scalars) {
+        CondVec.push_back(IOP->OpInst->getOperand(0));
+        TrueVec.push_back(IOP->OpInst->getOperand(1));
+        FalseVec.push_back(IOP->OpInst->getOperand(2));
       }
 
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
 
-      Value *Cond = vectorizeTree(CondVec);
-      Value *True = vectorizeTree(TrueVec);
-      Value *False = vectorizeTree(FalseVec);
+      Value *Cond = vectorizeTree(CondVec, E->Scalars[0]->MainOp);
+      Value *True = vectorizeTree(TrueVec, E->Scalars[0]->MainOp);
+      Value *False = vectorizeTree(FalseVec, E->Scalars[0]->MainOp);
 
       if (E->VectorizedValue) {
         LLVM_DEBUG(dbgs() << "SLP: Diamond merged for " << *VL0 << ".\n");
         return E->VectorizedValue;
       }
 
       Value *V = Builder.CreateSelect(Cond, True, False);
       if (NeedToShuffleReuses) {
switch (ShuffleOrOp) {
     case Instruction::FRem:
     case Instruction::Shl:
     case Instruction::LShr:
     case Instruction::AShr:
     case Instruction::And:
     case Instruction::Or:
     case Instruction::Xor: {
       ValueList LHSVL, RHSVL;
-      if (isa<BinaryOperator>(VL0) && VL0->isCommutative())
-        reorderInputsAccordingToOpcode(S.getOpcode(), E->Scalars, LHSVL,
-                                       RHSVL);
-      else
-        for (Value *V : E->Scalars) {
-          auto *I = cast<Instruction>(V);
-          LHSVL.push_back(I->getOperand(0));
-          RHSVL.push_back(I->getOperand(1));
+      if (isa<BinaryOperator>(VL0) && VL0->isCommutative()) {
+        reorderInputsAccordingToOpcode(E->Scalars[0]->MainOpcode,
+                                       E->Scalars, LHSVL, RHSVL);
+      } else
+        for (auto *IOP : E->Scalars) {
+          if (IOP->OpInst->getOpcode() == E->Scalars[0]->MainOpcode) {
+            LHSVL.push_back(IOP->OpInst->getOperand(0));
+            RHSVL.push_back(IOP->OpInst->getOperand(1));
+          } else {
+            LHSVL.push_back(IOP->Op);
+            RHSVL.push_back(
+                ConstantExpr::getBinOpIdentity(E->Scalars[0]->MainOpcode,
+                                               IOP->Op->getType(), true));
+          }
         }
 
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
 
-      Value *LHS = vectorizeTree(LHSVL);
-      Value *RHS = vectorizeTree(RHSVL);
+      Value *LHS = vectorizeTree(LHSVL, E->Scalars[0]->MainOp);
+      Value *RHS = vectorizeTree(RHSVL, E->Scalars[0]->MainOp);
 
       if (E->VectorizedValue) {
         LLVM_DEBUG(dbgs() << "SLP: Diamond merged for " << *VL0 << ".\n");
         return E->VectorizedValue;
       }
 
       Value *V = Builder.CreateBinOp(
-          static_cast<Instruction::BinaryOps>(S.getOpcode()), LHS, RHS);
-      propagateIRFlags(V, E->Scalars, VL0);
+          static_cast<Instruction::BinaryOps>(VL0->getOpcode()), LHS, RHS);
+      ValueList VL;
+      for (auto *IOP: E->Scalars)
+        VL.push_back(IOP->Op);
+      propagateIRFlags(V, VL, VL0);
       if (auto *I = dyn_cast<Instruction>(V))
         V = propagateMetadata(I, E->Scalars);
 
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
       ++NumVectorInstructions;
 
       return V;
     }
     case Instruction::Load: {
       // Loads are inserted at the head of the tree because we don't want to
       // sink them all the way down past store instructions.
       bool IsReorder = !E->ReorderIndices.empty();
+      ValueList VL;
+      for (auto *IOP: E->Scalars)
+        VL.push_back(IOP->Op);
       if (IsReorder) {
-        S = getSameOpcode(E->Scalars, E->ReorderIndices.front());
+        InstructionsState S = getSameOpcode(VL,
+                                            E->ReorderIndices.front());
         VL0 = cast<Instruction>(S.OpValue);
-      }
-      setInsertPointAfterBundle(E->Scalars, S);
+        setInsertPointAfterBundle(E->Scalars);
+      } else
+      setInsertPointAfterBundle(E->Scalars);
 
       LoadInst *LI = cast<LoadInst>(VL0);
       Type *ScalarLoadTy = LI->getType();
       unsigned AS = LI->getPointerAddressSpace();
 
       Value *VecPtr = Builder.CreateBitCast(LI->getPointerOperand(),
                                             VecTy->getPointerTo(AS));
 
       // The pointer operand uses an in-tree scalar so we add the new BitCast to
       // ExternalUses list to make sure that an extract will be generated in the
       // future.
       Value *PO = LI->getPointerOperand();
-      if (getTreeEntry(PO))
+      if (getTreeEntry(dyn_cast<Instruction>(PO)))
         ExternalUses.push_back(ExternalUser(PO, cast<User>(VecPtr), 0));
 
       unsigned Alignment = LI->getAlignment();
       LI = Builder.CreateLoad(VecPtr);
       if (!Alignment) {
         Alignment = DL->getABITypeAlignment(ScalarLoadTy);
       }
       LI->setAlignment(Alignment);
case Instruction::Load: {
       return V;
     }
     case Instruction::Store: {
       StoreInst *SI = cast<StoreInst>(VL0);
       unsigned Alignment = SI->getAlignment();
       unsigned AS = SI->getPointerAddressSpace();
 
       ValueList ScalarStoreValues;
-      for (Value *V : E->Scalars)
-        ScalarStoreValues.push_back(cast<StoreInst>(V)->getValueOperand());
+      for (auto *IOP : E->Scalars)
+        ScalarStoreValues.push_back(
+            cast<StoreInst>(IOP->OpInst)->getValueOperand());
 
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
 
-      Value *VecValue = vectorizeTree(ScalarStoreValues);
+      Value *VecValue = vectorizeTree(ScalarStoreValues, E->Scalars[0]->MainOp);
       Value *ScalarPtr = SI->getPointerOperand();
       Value *VecPtr = Builder.CreateBitCast(ScalarPtr, VecTy->getPointerTo(AS));
       StoreInst *ST = Builder.CreateStore(VecValue, VecPtr);
 
       // The pointer operand uses an in-tree scalar, so add the new BitCast to
       // ExternalUses to make sure that an extract will be generated in the
       // future.
-      if (getTreeEntry(ScalarPtr))
+      if (getTreeEntry(dyn_cast<Instruction>(ScalarPtr)))
         ExternalUses.push_back(ExternalUser(ScalarPtr, cast<User>(VecPtr), 0));
 
       if (!Alignment)
         Alignment = DL->getABITypeAlignment(SI->getValueOperand()->getType());
 
       ST->setAlignment(Alignment);
       Value *V = propagateMetadata(ST, E->Scalars);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
       ++NumVectorInstructions;
       return V;
     }
     case Instruction::GetElementPtr: {
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
 
       ValueList Op0VL;
-      for (Value *V : E->Scalars)
-        Op0VL.push_back(cast<GetElementPtrInst>(V)->getOperand(0));
+      for (auto *IOP : E->Scalars)
+        Op0VL.push_back(cast<GetElementPtrInst>(IOP->OpInst)->getOperand(0));
 
-      Value *Op0 = vectorizeTree(Op0VL);
+      Value *Op0 = vectorizeTree(Op0VL, E->Scalars[0]->MainOp);
 
       std::vector<Value *> OpVecs;
       for (int j = 1, e = cast<GetElementPtrInst>(VL0)->getNumOperands(); j < e;
            ++j) {
         ValueList OpVL;
-        for (Value *V : E->Scalars)
-          OpVL.push_back(cast<GetElementPtrInst>(V)->getOperand(j));
+        for (auto *IOP : E->Scalars)
+          OpVL.push_back(cast<GetElementPtrInst>(IOP->OpInst)->getOperand(j));
 
-        Value *OpVec = vectorizeTree(OpVL);
+        Value *OpVec = vectorizeTree(OpVL, E->Scalars[0]->MainOp);
         OpVecs.push_back(OpVec);
       }
 
       Value *V = Builder.CreateGEP(
           cast<GetElementPtrInst>(VL0)->getSourceElementType(), Op0, OpVecs);
       if (Instruction *I = dyn_cast<Instruction>(V))
         V = propagateMetadata(I, E->Scalars);
 
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
       ++NumVectorInstructions;
 
       return V;
     }
     case Instruction::Call: {
       CallInst *CI = cast<CallInst>(VL0);
-      setInsertPointAfterBundle(E->Scalars, S);
+      setInsertPointAfterBundle(E->Scalars);
       Function *FI;
       Intrinsic::ID IID  = Intrinsic::not_intrinsic;
       Value *ScalarArg = nullptr;
       if (CI && (FI = CI->getCalledFunction())) {
         IID = FI->getIntrinsicID();
       }
       std::vector<Value *> OpVecs;
       for (int j = 0, e = CI->getNumArgOperands(); j < e; ++j) {
         ValueList OpVL;
         // ctlz,cttz and powi are special intrinsics whose second argument is
         // a scalar. This argument should not be vectorized.
         if (hasVectorInstrinsicScalarOpd(IID, 1) && j == 1) {
           CallInst *CEI = cast<CallInst>(VL0);
           ScalarArg = CEI->getArgOperand(j);
           OpVecs.push_back(CEI->getArgOperand(j));
           continue;
         }
-        for (Value *V : E->Scalars) {
-          CallInst *CEI = cast<CallInst>(V);
+        for (auto *IOP : E->Scalars) {
+          CallInst *CEI = cast<CallInst>(IOP->OpInst);
           OpVL.push_back(CEI->getArgOperand(j));
         }
 
-        Value *OpVec = vectorizeTree(OpVL);
+        Value *OpVec = vectorizeTree(OpVL, E->Scalars[0]->MainOp);
         LLVM_DEBUG(dbgs() << "SLP: OpVec[" << j << "]: " << *OpVec << "\n");
         OpVecs.push_back(OpVec);
       }
 
       Module *M = F->getParent();
       Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
       Type *Tys[] = { VectorType::get(CI->getType(), E->Scalars.size()) };
       Function *CF = Intrinsic::getDeclaration(M, ID, Tys);
       SmallVector<OperandBundleDef, 1> OpBundles;
       CI->getOperandBundlesAsDefs(OpBundles);
       Value *V = Builder.CreateCall(CF, OpVecs, OpBundles);
 
       // The scalar argument uses an in-tree scalar so we add the new vectorized
       // call to ExternalUses list to make sure that an extract will be
       // generated in the future.
-      if (ScalarArg && getTreeEntry(ScalarArg))
+      if (ScalarArg && getTreeEntry(dyn_cast<Instruction>(ScalarArg)))
         ExternalUses.push_back(ExternalUser(ScalarArg, cast<User>(V), 0));
 
-      propagateIRFlags(V, E->Scalars, VL0);
+      ValueList VL;
+      for (auto *IOP: E->Scalars)
+        VL.push_back(IOP->Op);
+      propagateIRFlags(V, VL, VL0);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
       ++NumVectorInstructions;
       return V;
     }
     case Instruction::ShuffleVector: {
       ValueList LHSVL, RHSVL;
-      assert(S.isAltShuffle() &&
-             ((Instruction::isBinaryOp(S.getOpcode()) &&
-               Instruction::isBinaryOp(S.getAltOpcode())) ||
-              (Instruction::isCast(S.getOpcode()) &&
-               Instruction::isCast(S.getAltOpcode()))) &&
+      assert(E->Scalars[0]->isAltShuffle() &&
+             ((Instruction::isBinaryOp(E->Scalars[0]->MainOpcode) &&
+             Instruction::isBinaryOp(E->Scalars[0]->AltOpcode)) ||
+             (Instruction::isCast(E->Scalars[0]->MainOpcode) &&
+             Instruction::isCast(E->Scalars[0]->AltOpcode))) &&
              "Invalid Shuffle Vector Operand");
 
       Value *LHS, *RHS;
-      if (Instruction::isBinaryOp(S.getOpcode())) {
-        reorderAltShuffleOperands(S, E->Scalars, LHSVL, RHSVL);
-        setInsertPointAfterBundle(E->Scalars, S);
-        LHS = vectorizeTree(LHSVL);
-        RHS = vectorizeTree(RHSVL);
+      if (Instruction::isBinaryOp(E->Scalars[0]->MainOpcode)) {
+        reorderAltShuffleOperands(E->Scalars, LHSVL, RHSVL);
+        setInsertPointAfterBundle(E->Scalars);
+        LHS = vectorizeTree(LHSVL, E->Scalars[0]->MainOp);
+        RHS = vectorizeTree(RHSVL, E->Scalars[0]->MainOp);
       } else {
         ValueList INVL;
-        for (Value *V : E->Scalars)
-          INVL.push_back(cast<Instruction>(V)->getOperand(0));
-        setInsertPointAfterBundle(E->Scalars, S);
-        LHS = vectorizeTree(INVL);
+        for (auto *IOP : E->Scalars)
+          INVL.push_back(IOP->OpInst->getOperand(0));
+        setInsertPointAfterBundle(E->Scalars);
+        LHS = vectorizeTree(INVL, E->Scalars[0]->MainOp);
       }
 
       if (E->VectorizedValue) {
         LLVM_DEBUG(dbgs() << "SLP: Diamond merged for " << *VL0 << ".\n");
         return E->VectorizedValue;
       }
 
       Value *V0, *V1;
-      if (Instruction::isBinaryOp(S.getOpcode())) {
+      if (Instruction::isBinaryOp(E->Scalars[0]->MainOpcode)) {
         V0 = Builder.CreateBinOp(
-          static_cast<Instruction::BinaryOps>(S.getOpcode()), LHS, RHS);
+          static_cast<Instruction::BinaryOps>(E->Scalars[0]->MainOpcode), LHS,
+          RHS);
         V1 = Builder.CreateBinOp(
-          static_cast<Instruction::BinaryOps>(S.getAltOpcode()), LHS, RHS);
+          static_cast<Instruction::BinaryOps>(E->Scalars[0]->AltOpcode), LHS,
+          RHS);
       } else {
         V0 = Builder.CreateCast(
-            static_cast<Instruction::CastOps>(S.getOpcode()), LHS, VecTy);
+            static_cast<Instruction::CastOps>(E->Scalars[0]->MainOpcode), LHS,
+            VecTy);
         V1 = Builder.CreateCast(
-            static_cast<Instruction::CastOps>(S.getAltOpcode()), LHS, VecTy);
+            static_cast<Instruction::CastOps>(E->Scalars[0]->AltOpcode), LHS,
+            VecTy);
       }
 
       // Create shuffle to take alternate operations from the vector.
       // Also, gather up main and alt scalar ops to propagate IR flags to
       // each vector operation.
       ValueList OpScalars, AltScalars;
       unsigned e = E->Scalars.size();
       SmallVector<Constant *, 8> Mask(e);
       for (unsigned i = 0; i < e; ++i) {
-        auto *OpInst = cast<Instruction>(E->Scalars[i]);
-        assert(S.isOpcodeOrAlt(OpInst) && "Unexpected main/alternate opcode");
-        if (OpInst->getOpcode() == S.getAltOpcode()) {
+        auto *OpInst = E->Scalars[i]->OpInst;
+        if (OpInst->getOpcode() == E->Scalars[0]->AltOpcode) {
           Mask[i] = Builder.getInt32(e + i);
-          AltScalars.push_back(E->Scalars[i]);
+          AltScalars.push_back(E->Scalars[i]->Op);
         } else {
           Mask[i] = Builder.getInt32(i);
-          OpScalars.push_back(E->Scalars[i]);
+          OpScalars.push_back(E->Scalars[i]->Op);
         }
       }
 
       Value *ShuffleMask = ConstantVector::get(Mask);
-      propagateIRFlags(V0, OpScalars);
-      propagateIRFlags(V1, AltScalars);
+      InstructionsState S = getSameOpcode(OpScalars);
+      propagateIRFlags(V0, OpScalars, S.OpValue);
+      S = getSameOpcode(AltScalars);
+      propagateIRFlags(V1, AltScalars, S.OpValue);
 
       Value *V = Builder.CreateShuffleVector(V0, V1, ShuffleMask);
       if (Instruction *I = dyn_cast<Instruction>(V))
         V = propagateMetadata(I, E->Scalars);
       if (NeedToShuffleReuses) {
         V = Builder.CreateShuffleVector(V, UndefValue::get(VecTy),
                                         E->ReuseShuffleIndices, "shuffle");
       }
       E->VectorizedValue = V;
BoUpSLP::vectorizeTree(ExtraValueToDebugLocsMap &ExternallyUsedValues) {
   }
 
   Builder.SetInsertPoint(&F->getEntryBlock().front());
   auto *VectorRoot = vectorizeTree(&VectorizableTree[0]);
 
   // If the vectorized tree can be rewritten in a smaller type, we truncate the
   // vectorized root. InstCombine will then rewrite the entire expression. We
   // sign extend the extracted values below.
-  auto *ScalarRoot = VectorizableTree[0].Scalars[0];
+  auto *ScalarRoot = VectorizableTree[0].Scalars[0]->MainOp;
   if (MinBWs.count(ScalarRoot)) {
     if (auto *I = dyn_cast<Instruction>(VectorRoot))
       Builder.SetInsertPoint(&*++BasicBlock::iterator(I));
     auto BundleWidth = VectorizableTree[0].Scalars.size();
     auto *MinTy = IntegerType::get(F->getContext(), MinBWs[ScalarRoot].first);
     auto *VecTy = VectorType::get(MinTy, BundleWidth);
     auto *Trunc = Builder.CreateTrunc(VectorRoot, VecTy);
     VectorizableTree[0].VectorizedValue = Trunc;
BoUpSLP::vectorizeTree(ExtraValueToDebugLocsMap &ExternallyUsedValues) {
   for (const auto &ExternalUse : ExternalUses) {
     Value *Scalar = ExternalUse.Scalar;
     llvm::User *User = ExternalUse.User;
 
     // Skip users that we already RAUW. This happens when one instruction
     // has multiple uses of the same value.
     if (User && !is_contained(Scalar->users(), User))
       continue;
-    TreeEntry *E = getTreeEntry(Scalar);
+    TreeEntry *E = getTreeEntry(dyn_cast<Instruction>(Scalar));
     assert(E && "Invalid scalar");
     assert(!E->NeedToGather && "Extracting from a gather list");
 
     Value *Vec = E->VectorizedValue;
     assert(Vec && "Can't find vectorizable value");
 
     Value *Lane = Builder.getInt32(ExternalUse.Lane);
     // If User == nullptr, the Scalar is used as extra arg. Generate
for (const auto &ExternalUse : ExternalUses) {
     LLVM_DEBUG(dbgs() << "SLP: Replaced:" << *User << ".\n");
   }
 
   // For each vectorized value:
   for (TreeEntry &EIdx : VectorizableTree) {
     TreeEntry *Entry = &EIdx;
 
     // No need to handle users of gathered values.
     if (Entry->NeedToGather)
       continue;
 
     assert(Entry->VectorizedValue && "Can't find vectorizable value");
 
     // For each lane:
     for (int Lane = 0, LE = Entry->Scalars.size(); Lane != LE; ++Lane) {
-      Value *Scalar = Entry->Scalars[Lane];
+      Value *Scalar = Entry->Scalars[Lane]->Op;
+
+      if (!Entry->Scalars[Lane]->isOpcodeOrAlt())
+        continue;
 
       Type *Ty = Scalar->getType();
       if (!Ty->isVoidTy()) {
 #ifndef NDEBUG
         for (User *U : Scalar->users()) {
           LLVM_DEBUG(dbgs() << "SLP: \tvalidating user:" << *U << ".\n");
 
           // It is legal to replace users in the ignorelist by undef.
-          assert((getTreeEntry(U) || is_contained(UserIgnoreList, U)) &&
+          assert((getTreeEntry(dyn_cast<Instruction>(U)) ||
+                  is_contained(UserIgnoreList, U)) &&
                  "Replacing out-of-tree value with undef");
         }
 #endif
         Value *Undef = UndefValue::get(Ty);
         Scalar->replaceAllUsesWith(Undef);
       }
       LLVM_DEBUG(dbgs() << "SLP: \tErasing scalar:" << *Scalar << ".\n");
-      eraseInstruction(cast<Instruction>(Scalar));
+      eraseInstruction(Entry->Scalars[Lane]->OpInst);
     }
   }
 
   Builder.ClearInsertionPoint();
 
   return VectorizableTree[0].VectorizedValue;
 }
 
for (auto I = CSEWorkList.begin(), E = CSEWorkList.end(); I != E; ++I) {
     }
   }
   CSEBlocks.clear();
   GatherSeq.clear();
 }
 
 // Groups the instructions to a bundle (which is then a single scheduling entity)
 // and schedules instructions until the bundle gets ready.
-bool BoUpSLP::BlockScheduling::tryScheduleBundle(ArrayRef<Value *> VL,
-                                                 BoUpSLP *SLP,
-                                                 const InstructionsState &S) {
-  if (isa<PHINode>(S.OpValue))
+bool BoUpSLP::BlockScheduling::tryScheduleBundle(
+    ArrayRef<InstructionOrPseudo *> IL, BoUpSLP *SLP) {
+  if (isa<PHINode>(IL[0]->MainOp))
     return true;
 
   // Initialize the instruction bundle.
   Instruction *OldScheduleEnd = ScheduleEnd;
   ScheduleData *PrevInBundle = nullptr;
   ScheduleData *Bundle = nullptr;
   bool ReSchedule = false;
-  LLVM_DEBUG(dbgs() << "SLP:  bundle: " << *S.OpValue << "\n");
+  bool IsPseudo = false;
+  LLVM_DEBUG(dbgs() << "SLP:  bundle: " << *IL[0]->MainOp << "\n");
 
   // Make sure that the scheduling region contains all
   // instructions of the bundle.
-  for (Value *V : VL) {
-    if (!extendSchedulingRegion(V, S))
+  for (InstructionOrPseudo *IOP : IL)
+    if (!extendSchedulingRegion(IOP))
       return false;
-  }
 
-  for (Value *V : VL) {
-    ScheduleData *BundleMember = getScheduleData(V);
+  for (InstructionOrPseudo *IOP : IL) {
+    ScheduleData *BundleMember = getScheduleData(IOP->OpInst);
+    if (BundleMember->isPartOfBundle()) {
+      BundleMember = getScheduleData(IOP->OpInst, IOP->getKey());
+      IsPseudo = true;
+    }
+    if (BundleMember->isPartOfBundle())
+      return false;
     assert(BundleMember &&
            "no ScheduleData for bundle member (maybe not in same basic block)");
+    assert(!BundleMember->isPartOfBundle() && "Already part of another bundle");
     if (BundleMember->IsScheduled) {
       // A bundle member was scheduled as single instruction before and now
       // needs to be scheduled as part of the bundle. We just get rid of the
       // existing schedule.
       LLVM_DEBUG(dbgs() << "SLP:  reset schedule because " << *BundleMember
                         << " was already scheduled\n");
       ReSchedule = true;
     }
     assert(BundleMember->isSchedulingEntity() &&
            "bundle member already part of other bundle");
     if (PrevInBundle) {
       PrevInBundle->NextInBundle = BundleMember;
     } else {
       Bundle = BundleMember;
     }
     BundleMember->UnscheduledDepsInBundle = 0;
+    BundleMember->Opcode = IOP->MainOpcode;
+    BundleMember->Parent = IOP->Parent;
     Bundle->UnscheduledDepsInBundle += BundleMember->UnscheduledDeps;
 
     // Group the instructions to a bundle.
     BundleMember->FirstInBundle = Bundle;
     PrevInBundle = BundleMember;
   }
-  if (ScheduleEnd != OldScheduleEnd) {
+  if (ScheduleEnd != OldScheduleEnd || IsPseudo) {
     // The scheduling region got new instructions at the lower end (or it is a
     // new region for the first bundle). This makes it necessary to
     // recalculate all dependencies.
     // It is seldom that this needs to be done a second time after adding the
     // initial bundle to the region.
     for (auto *I = ScheduleStart; I != ScheduleEnd; I = I->getNextNode()) {
       doForAllOpcodes(I, [](ScheduleData *SD) {
         SD->clearDependencies();
while (!Bundle->isReady() && !ReadyInsts.empty()) {
 
     ScheduleData *pickedSD = ReadyInsts.back();
     ReadyInsts.pop_back();
 
     if (pickedSD->isSchedulingEntity() && pickedSD->isReady()) {
       schedule(pickedSD, ReadyInsts);
     }
   }
   if (!Bundle->isReady()) {
-    cancelScheduling(VL, S.OpValue);
+    cancelScheduling(IL[0]);
+    // We have to clear all dependencies, since all values
+    // were calculated for the vectorized bundle.
+    for (auto *I = ScheduleStart; I != ScheduleEnd;
+      I = I->getNextNode()) {
+      doForAllOpcodes(I, [](ScheduleData *SD) {
+        SD->clearDependencies();
+      });
+    }
+    resetSchedule();
     return false;
   }
   return true;
 }
 
-void BoUpSLP::BlockScheduling::cancelScheduling(ArrayRef<Value *> VL,
-                                                Value *OpValue) {
-  if (isa<PHINode>(OpValue))
+void BoUpSLP::BlockScheduling::cancelScheduling(InstructionOrPseudo *IOP) {
+  if (!IOP->MainOp)
     return;
-
-  ScheduleData *Bundle = getScheduleData(OpValue);
+  if (isa<PHINode>(IOP->MainOp))
+    return;
+  auto Key = IOP->getKey();
+  ScheduleData *Bundle = getScheduleData(IOP->OpInst, Key)->FirstInBundle;
+  assert(Bundle && "Counld not find bundle");
   LLVM_DEBUG(dbgs() << "SLP:  cancel scheduling of " << *Bundle << "\n");
   assert(!Bundle->IsScheduled &&
          "Can't cancel bundle which is already scheduled");
   assert(Bundle->isSchedulingEntity() && Bundle->isPartOfBundle() &&
          "tried to unbundle something which is not a bundle");
 
   // Un-bundle: make single instructions out of the bundle.
   ScheduleData *BundleMember = Bundle;
   while (BundleMember) {
-    assert(BundleMember->FirstInBundle == Bundle && "corrupt bundle links");
+    assert(BundleMember->FirstInBundle == Bundle && "Corrupt bundle links");
+    assert(BundleMember->Parent == Key.first &&
+           BundleMember->Opcode == Key.second && "Corrupt bundle");
     BundleMember->FirstInBundle = BundleMember;
     ScheduleData *Next = BundleMember->NextInBundle;
     BundleMember->NextInBundle = nullptr;
     BundleMember->UnscheduledDepsInBundle = BundleMember->UnscheduledDeps;
+    if (BundleMember->isPseudo()) {
+      PseudoInstScheduleDataMap[BundleMember->Inst].erase(Key);
+      BundleMember->Opcode = 0;
+      BundleMember->Parent = nullptr;
+    } else {
+      BundleMember->Opcode = 0;
+      BundleMember->Parent = nullptr;
       if (BundleMember->UnscheduledDepsInBundle == 0) {
         ReadyInsts.insert(BundleMember);
       }
+    }
     BundleMember = Next;
   }
 }
 
-BoUpSLP::ScheduleData *BoUpSLP::BlockScheduling::allocateScheduleDataChunks() {
-  // Allocate a new ScheduleData for the instruction.
+BoUpSLP::ScheduleData *
+BoUpSLP::BlockScheduling::allocateScheduleDataChunks() {
+  // Allocate a new InstScheduleData for the instruction.
   if (ChunkPos >= ChunkSize) {
-    ScheduleDataChunks.push_back(llvm::make_unique<ScheduleData[]>(ChunkSize));
+    ScheduleDataChunks.push_back(
+        llvm::make_unique<ScheduleData[]>(ChunkSize));
     ChunkPos = 0;
   }
   return &(ScheduleDataChunks.back()[ChunkPos++]);
 }
 
-bool BoUpSLP::BlockScheduling::extendSchedulingRegion(Value *V,
-                                                      const InstructionsState &S) {
-  if (getScheduleData(V, isOneOf(S, V)))
+bool BoUpSLP::BlockScheduling::extendSchedulingRegion(
+    InstructionOrPseudo *IOP) {
+  if (getScheduleData(IOP->OpInst, IOP->getKey()))
     return true;
-  Instruction *I = dyn_cast<Instruction>(V);
-  assert(I && "bundle member must be an instruction");
-  assert(!isa<PHINode>(I) && "phi nodes don't need to be scheduled");
-  auto &&CheckSheduleForI = [this, &S](Instruction *I) -> bool {
+  assert(!isa<PHINode>(IOP->OpInst) && "phi nodes don't need to be scheduled");
+  auto &&CheckSheduleForI = [this, &IOP](Instruction *I) -> bool {
     ScheduleData *ISD = getScheduleData(I);
     if (!ISD)
       return false;
     assert(isInSchedulingRegion(ISD) &&
-           "ScheduleData not in scheduling region");
+           "InstScheduleData not in scheduling region");
+    if (ISD->isPartOfBundle()) {
       ScheduleData *SD = allocateScheduleDataChunks();
-    SD->Inst = I;
-    SD->init(SchedulingRegionID, S.OpValue);
-    ExtraScheduleDataMap[I][S.OpValue] = SD;
+      SD->init(SchedulingRegionID, ISD, IOP->Parent, IOP->MainOpcode);
+      PseudoInstScheduleDataMap[I][IOP->getKey()] = SD;
+    }
     return true;
   };
-  if (CheckSheduleForI(I))
+  if (CheckSheduleForI(IOP->OpInst))
     return true;
   if (!ScheduleStart) {
     // It's the first instruction in the new region.
-    initScheduleData(I, I->getNextNode(), nullptr, nullptr);
-    ScheduleStart = I;
-    ScheduleEnd = I->getNextNode();
-    if (isOneOf(S, I) != I)
-      CheckSheduleForI(I);
+    initScheduleData(IOP->OpInst, IOP->OpInst->getNextNode(), nullptr, nullptr);
+    ScheduleStart = IOP->OpInst;
+    ScheduleEnd = IOP->OpInst->getNextNode();
+    CheckSheduleForI(IOP->OpInst);
     assert(ScheduleEnd && "tried to vectorize a terminator?");
-    LLVM_DEBUG(dbgs() << "SLP:  initialize schedule region to " << *I << "\n");
+    LLVM_DEBUG(dbgs() << "SLP:  initialize schedule region to " << *IOP->Op
+               << "\n");
     return true;
   }
   // Search up and down at the same time, because we don't know if the new
   // instruction is above or below the existing scheduling region.
   BasicBlock::reverse_iterator UpIter =
       ++ScheduleStart->getIterator().getReverse();
   BasicBlock::reverse_iterator UpperEnd = BB->rend();
   BasicBlock::iterator DownIter = ScheduleEnd->getIterator();
   BasicBlock::iterator LowerEnd = BB->end();
   while (true) {
     if (++ScheduleRegionSize > ScheduleRegionSizeLimit) {
       LLVM_DEBUG(dbgs() << "SLP:  exceeded schedule region size limit\n");
       return false;
     }
 
     if (UpIter != UpperEnd) {
-      if (&*UpIter == I) {
-        initScheduleData(I, ScheduleStart, nullptr, FirstLoadStoreInRegion);
-        ScheduleStart = I;
-        if (isOneOf(S, I) != I)
-          CheckSheduleForI(I);
-        LLVM_DEBUG(dbgs() << "SLP:  extend schedule region start to " << *I
-                          << "\n");
+      if (&*UpIter == IOP->OpInst) {
+        initScheduleData(IOP->OpInst, ScheduleStart, nullptr,
+                         FirstLoadStoreInRegion);
+        ScheduleStart = IOP->OpInst;
+        CheckSheduleForI(IOP->OpInst);
+        LLVM_DEBUG(dbgs() << "SLP:  extend schedule region start to "
+                          << *IOP->Op << "\n");
         return true;
       }
       UpIter++;
     }
     if (DownIter != LowerEnd) {
-      if (&*DownIter == I) {
-        initScheduleData(ScheduleEnd, I->getNextNode(), LastLoadStoreInRegion,
-                         nullptr);
-        ScheduleEnd = I->getNextNode();
-        if (isOneOf(S, I) != I)
-          CheckSheduleForI(I);
+      if (&*DownIter == IOP->OpInst) {
+        initScheduleData(ScheduleEnd, IOP->OpInst->getNextNode(),
+                         LastLoadStoreInRegion, nullptr);
+        ScheduleEnd = IOP->OpInst->getNextNode();
+        CheckSheduleForI(IOP->OpInst);
         assert(ScheduleEnd && "tried to vectorize a terminator?");
-        LLVM_DEBUG(dbgs() << "SLP:  extend schedule region end to " << *I
+        LLVM_DEBUG(dbgs() << "SLP:  extend schedule region end to " << *IOP->Op
                           << "\n");
         return true;
       }
       DownIter++;
     }
     assert((UpIter != UpperEnd || DownIter != LowerEnd) &&
            "instruction not found in block");
   }
   return true;
 }
 
-void BoUpSLP::BlockScheduling::initScheduleData(Instruction *FromI,
-                                                Instruction *ToI,
-                                                ScheduleData *PrevLoadStore,
-                                                ScheduleData *NextLoadStore) {
+void BoUpSLP::BlockScheduling::initScheduleData(
+    Instruction *FromI, Instruction *ToI, ScheduleData *PrevLoadStore,
   ScheduleData *CurrentLoadStore = PrevLoadStore;
   for (Instruction *I = FromI; I != ToI; I = I->getNextNode()) {
-    ScheduleData *SD = ScheduleDataMap[I];
+    ScheduleData *SD = InstScheduleDataMap[I];
     if (!SD) {
       SD = allocateScheduleDataChunks();
-      ScheduleDataMap[I] = SD;
+      InstScheduleDataMap[I] = SD;
       SD->Inst = I;
     }
     assert(!isInSchedulingRegion(SD) &&
-           "new ScheduleData already in scheduling region");
-    SD->init(SchedulingRegionID, I);
+           "new InstScheduleData already in scheduling region");
+    SD->init(SchedulingRegionID);
 
     if (I->mayReadOrWriteMemory() &&
         (!isa<IntrinsicInst>(I) ||
          cast<IntrinsicInst>(I)->getIntrinsicID() != Intrinsic::sideeffect)) {
       // Update the linked list of memory accessing instructions.
       if (CurrentLoadStore) {
         CurrentLoadStore->NextLoadStore = SD;
       } else {
void BoUpSLP::BlockScheduling::calculateDependencies(ScheduleData *SD,
   SmallVector<ScheduleData *, 10> WorkList;
   WorkList.push_back(SD);
 
   while (!WorkList.empty()) {
     ScheduleData *SD = WorkList.back();
     WorkList.pop_back();
 
     ScheduleData *BundleMember = SD;
+    unsigned Opcode = BundleMember->Opcode;
+    Value *Parent = BundleMember->Parent;
     while (BundleMember) {
       assert(isInSchedulingRegion(BundleMember));
+      assert(BundleMember->Opcode == Opcode &&
+             BundleMember->Parent == Parent && "Corrupt bundle member");
+
       if (!BundleMember->hasValidDependencies()) {
 
         LLVM_DEBUG(dbgs() << "SLP:       update deps of " << *BundleMember
                           << "\n");
         BundleMember->Dependencies = 0;
         BundleMember->resetUnscheduledDeps();
 
         // Handle def-use chain dependencies.
-        if (BundleMember->OpValue != BundleMember->Inst) {
-          ScheduleData *UseSD = getScheduleData(BundleMember->Inst);
-          if (UseSD && isInSchedulingRegion(UseSD->FirstInBundle)) {
-            BundleMember->Dependencies++;
-            ScheduleData *DestBundle = UseSD->FirstInBundle;
-            if (!DestBundle->IsScheduled)
-              BundleMember->incrementUnscheduledDeps(1);
-            if (!DestBundle->hasValidDependencies())
-              WorkList.push_back(DestBundle);
-          }
-        } else {
-          for (User *U : BundleMember->Inst->users()) {
-            if (isa<Instruction>(U)) {
-              ScheduleData *UseSD = getScheduleData(U);
-              if (UseSD && isInSchedulingRegion(UseSD->FirstInBundle)) {
-                BundleMember->Dependencies++;
-                ScheduleData *DestBundle = UseSD->FirstInBundle;
-                if (!DestBundle->IsScheduled)
-                  BundleMember->incrementUnscheduledDeps(1);
-                if (!DestBundle->hasValidDependencies())
-                  WorkList.push_back(DestBundle);
-              }
-            } else {
-              // I'm not sure if this can ever happen. But we need to be safe.
-              // This lets the instruction/bundle never be scheduled and
-              // eventually disable vectorization.
-              BundleMember->Dependencies++;
-              BundleMember->incrementUnscheduledDeps(1);
-            }
-          }
-        }
+          if (auto *I = dyn_cast<Instruction>(U)) {
+            doForAllOpcodes(I, [&BundleMember, &WorkList](ScheduleData *UseSD) {
+            });
 
         // Handle the memory dependencies.
         ScheduleData *DepDest = BundleMember->NextLoadStore;
         if (DepDest) {
           Instruction *SrcInst = BundleMember->Inst;
           MemoryLocation SrcLoc = getLocation(SrcInst, SLP->AA);
-          bool SrcMayWrite = BundleMember->Inst->mayWriteToMemory();
+          bool SrcMayWrite = SrcInst->mayWriteToMemory();
           unsigned numAliased = 0;
           unsigned DistToSrc = 1;
 
           while (DepDest) {
             assert(isInSchedulingRegion(DepDest));
 
             // We have two limits to reduce the complexity:
             // 1) AliasedCheckLimit: It's a small limit to reduce calls to
             //    SLP->isAliased (which is the expensive part in this loop).
             // 2) MaxMemDepDistance: It's for very large blocks and it aborts
             //    the whole loop (even if the loop is fast, it's quadratic).
             //    It's important for the loop break condition (see below) to
             //    check this limit even between two read-only instructions.
             if (DistToSrc >= MaxMemDepDistance ||
                     ((SrcMayWrite || DepDest->Inst->mayWriteToMemory()) &&
                      (numAliased >= AliasedCheckLimit ||
                       SLP->isAliased(SrcLoc, SrcInst, DepDest->Inst)))) {
 
               // We increment the counter only if the locations are aliased
               // (instead of counting all alias checks). This gives a better
               // balance between reduced runtime and accurate dependencies.
               numAliased++;
 
+              // We don't want any duplicates in the set to have a correct
+              // dependancies.
+              doForAllOpcodes(DepDest->Inst, [&BundleMember, &WorkList](
+                                                      ScheduleData *DepDest) {
                   DepDest->MemoryDependencies.push_back(BundleMember);
                   BundleMember->Dependencies++;
                   ScheduleData *DestBundle = DepDest->FirstInBundle;
                   if (!DestBundle->IsScheduled) {
                     BundleMember->incrementUnscheduledDeps(1);
                   }
                   if (!DestBundle->hasValidDependencies()) {
                     WorkList.push_back(DestBundle);
                   }
+              });
             }
             DepDest = DepDest->NextLoadStore;
 
             // Example, explaining the loop break condition: Let's assume our
             // starting instruction is i0 and MaxMemDepDistance = 3.
             //
             //                      +--------v--v--v
             //             i0,i1,i2,i3,i4,i5,i6,i7,i8

 }
 
 void BoUpSLP::BlockScheduling::resetSchedule() {
   assert(ScheduleStart &&
          "tried to reset schedule on block which has not been scheduled");
   for (Instruction *I = ScheduleStart; I != ScheduleEnd; I = I->getNextNode()) {
     doForAllOpcodes(I, [&](ScheduleData *SD) {
       assert(isInSchedulingRegion(SD) &&
-             "ScheduleData not in scheduling region");
+             "InstScheduleData not in scheduling region");
       SD->IsScheduled = false;
       SD->resetUnscheduledDeps();
     });
   }
   ReadyInsts.clear();
 }
 
+void BoUpSLP::BlockScheduling::reorderBundles() {
+  SmallPtrSet<ScheduleData *, 2> Bundles;
+  DenseMap<ScheduleData *, ScheduleData *> ReorderMap;
+  for (auto I = PseudoInstScheduleDataMap.begin(),
+            E = PseudoInstScheduleDataMap.end();
+       I != E; ++I) {
+    doForAllOpcodes(I->first, [&Bundles](ScheduleData *SD) {
+      if (SD->isPartOfBundle())
+        Bundles.insert(SD->FirstInBundle);
+    });
+  }
+  // Walk backward in the BB to discover the last instruction
+  // for a bundle.
+  for (auto I = BB->rbegin(), E = BB->rend(); (I != E && Bundles.size() > 0);
+       ++I) {
+    doForAllOpcodes(&*I, [&ReorderMap, &Bundles](ScheduleData *SD) {
+      if (SD->isPartOfBundle() && Bundles.count(SD->FirstInBundle) != 0) {
+        ReorderMap[SD->FirstInBundle] = SD;
+        Bundles.erase(SD->FirstInBundle);
+      }
+    });
+  }
+  // Swap the last scheduled instruction with the first one in the bundle.
+  for (auto I = ReorderMap.begin(), E = ReorderMap.end(); I != E; ++I) {
+    ScheduleData *FirstSD = I->first;
+    ScheduleData *LastSD = I->second;
+    SmallVector<ScheduleData *, 4> Bundle;
+    unsigned LastPos = 0;
+    // The first instruction in the bundle is already the last one scheduled.
+    if (FirstSD == LastSD)
+      continue;
+    ScheduleData *SD = FirstSD;
+    while (SD) {
+      if (SD == LastSD)
+        LastPos = Bundle.size();
+      Bundle.push_back(SD);
+      SD = SD->NextInBundle;
+    }
+    std::swap(Bundle[0], Bundle[LastPos]);
+    for (ScheduleData *SD : Bundle)
+      SD->FirstInBundle = Bundle[0];
+    Bundle[0]->NextInBundle = Bundle[1];
+    Bundle[LastPos - 1]->NextInBundle = Bundle[LastPos];
+    if (LastPos == Bundle.size() - 1)
+      Bundle[LastPos]->NextInBundle = nullptr;
+    else
+      Bundle[LastPos]->NextInBundle = Bundle[LastPos + 1];
+  }
+}
+
 void BoUpSLP::scheduleBlock(BlockScheduling *BS) {
   if (!BS->ScheduleStart)
     return;
 
   LLVM_DEBUG(dbgs() << "SLP: schedule block " << BS->BB->getName() << "\n");
-
   BS->resetSchedule();
 
   // For the real scheduling we use a more sophisticated ready-list: it is
   // sorted by the original instruction location. This lets the final schedule
   // be as  close as possible to the original instruction order.
   struct ScheduleDataCompare {
     bool operator()(ScheduleData *SD1, ScheduleData *SD2) const {
       return SD2->SchedulingPriority < SD1->SchedulingPriority;
     }
   };
   std::set<ScheduleData *, ScheduleDataCompare> ReadyInsts;
 
   // Ensure that all dependency data is updated and fill the ready-list with
   // initial instructions.
   int Idx = 0;
   int NumToSchedule = 0;
   for (auto *I = BS->ScheduleStart; I != BS->ScheduleEnd;
        I = I->getNextNode()) {
     BS->doForAllOpcodes(I, [this, &Idx, &NumToSchedule, BS](ScheduleData *SD) {
+      std::pair<Value *, unsigned> Key = std::make_pair(SD->Parent, SD->Opcode);
       assert(SD->isPartOfBundle() ==
-                 (getTreeEntry(SD->Inst) != nullptr) &&
+                 (ScalarToTreeEntry.find(SD->Inst) != ScalarToTreeEntry.end() &&
+                  ScalarToTreeEntry[SD->Inst].find(Key) !=
+                  ScalarToTreeEntry[SD->Inst].end()) &&
              "scheduler and vectorizer bundle mismatch");
       SD->FirstInBundle->SchedulingPriority = Idx++;
       if (SD->isSchedulingEntity()) {
         BS->calculateDependencies(SD, false, this);
         NumToSchedule++;
       }
     });
   }
   BS->initialFillReadyList(ReadyInsts);
 
   Instruction *LastScheduledInst = BS->ScheduleEnd;
 
   // Do the "real" scheduling.
   while (!ReadyInsts.empty()) {
     ScheduleData *picked = *ReadyInsts.begin();
     ReadyInsts.erase(ReadyInsts.begin());
 
     // Move the scheduled instruction(s) to their dedicated places, if not
     // there yet.
     ScheduleData *BundleMember = picked;
+    unsigned Opcode = BundleMember->Opcode;
+    Value *Parent = BundleMember->Parent;
     while (BundleMember) {
-      Instruction *pickedInst = BundleMember->Inst;
-      if (LastScheduledInst->getNextNode() != pickedInst) {
-        BS->BB->getInstList().remove(pickedInst);
+      assert(Opcode == BundleMember->Opcode &&
+             Parent == BundleMember->Parent && "Corrupt bundle member");
+      Instruction *PickedInst = BundleMember->Inst;
+      if (LastScheduledInst != PickedInst) {
+        BS->BB->getInstList().remove(PickedInst);
         BS->BB->getInstList().insert(LastScheduledInst->getIterator(),
-                                     pickedInst);
+                                     PickedInst);
       }
-      LastScheduledInst = pickedInst;
+      LastScheduledInst = PickedInst;
       BundleMember = BundleMember->NextInBundle;
     }
-
     BS->schedule(picked, ReadyInsts);
     NumToSchedule--;
   }
+#ifndef NDEBUG
+  if (NumToSchedule != 0) {
+    for (BasicBlock::iterator I = BS->BB->begin(), E = BS->BB->end(); I != E;
+         ++I) {
+      BS->doForAllOpcodes(&*I, [](ScheduleData *SD) {
+        if (SD->isSchedulingEntity() && SD->UnscheduledDepsInBundle != 0)
+          LLVM_DEBUG(dbgs() << "SLP: Failed to schedule: " << *SD << ".\n");
+      });
+    }
+  }
+#endif
   assert(NumToSchedule == 0 && "could not schedule all instructions");
+  BS->reorderBundles();
 
   // Avoid duplicate scheduling of the block.
   BS->ScheduleStart = nullptr;
 }
 
 unsigned BoUpSLP::getVectorElementSize(Value *V) {
   // If V is a store, just return the width of the stored value without
   // traversing the expression tree. This is the common case.

 void BoUpSLP::computeMinimumValueSizes() {
   // If there are no external uses, the expression tree must be rooted by a
   // store. We can't demote in-memory values, so there is nothing to do here.
   if (ExternalUses.empty())
     return;
 
   // We only attempt to truncate integer expressions.
   auto &TreeRoot = VectorizableTree[0].Scalars;
-  auto *TreeRootIT = dyn_cast<IntegerType>(TreeRoot[0]->getType());
+  auto *TreeRootIT = dyn_cast<IntegerType>(TreeRoot[0]->Op->getType());
   if (!TreeRootIT)
     return;
 
   // If the expression is not rooted by a store, these roots should have
   // external uses. We will rely on InstCombine to rewrite the expression in
   // the narrower type. However, InstCombine only rewrites single-use values.
   // This means that if a tree entry other than a root is used externally, it
   // must have multiple uses and InstCombine will not rewrite it. The code
   // below ensures that only the roots are used externally.
-  SmallPtrSet<Value *, 32> Expr(TreeRoot.begin(), TreeRoot.end());
+  SmallPtrSet<Value *, 32> Expr;
+  for (auto *IOP : TreeRoot)
+    Expr.insert(IOP->Op);
   for (auto &EU : ExternalUses)
     if (!Expr.erase(EU.Scalar))
       return;
   if (!Expr.empty())
     return;
 
   // Collect the scalar values of the vectorizable expression. We will use this
   // context to determine which values can be demoted. If we see a truncation,
   // we mark it as seeding another demotion.
   for (auto &Entry : VectorizableTree)
-    Expr.insert(Entry.Scalars.begin(), Entry.Scalars.end());
+    for (auto *IOP: Entry.Scalars)
+      Expr.insert(IOP->Op);
 
   // Ensure the roots of the vectorizable tree don't form a cycle. They must
   // have a single external user that is not in the vectorizable tree.
   for (auto *Root : TreeRoot)
-    if (!Root->hasOneUse() || Expr.count(*Root->user_begin()))
+    if (!Root->Op->hasOneUse() || Expr.count(*Root->Op->user_begin()))
       return;
 
   // Conservatively determine if we can actually truncate the roots of the
   // expression. Collect the values that can be demoted in ToDemote and
   // additional roots that require investigating in Roots.
   SmallVector<Value *, 32> ToDemote;
   SmallVector<Value *, 4> Roots;
   for (auto *Root : TreeRoot)
-    if (!collectValuesToDemote(Root, Expr, ToDemote, Roots))
+    if (!collectValuesToDemote(Root->Op, Expr, ToDemote, Roots))
       return;
 
   // The maximum bit width required to represent all the values that can be
   // demoted without loss of precision. It would be safe to truncate the roots
   // of the expression to this width.
   auto MaxBitWidth = 8u;
 
   // We first check if all the bits of the roots are demanded. If they're not,
   // we can truncate the roots to this narrower type.
   for (auto *Root : TreeRoot) {
-    auto Mask = DB->getDemandedBits(cast<Instruction>(Root));
+    auto Mask = DB->getDemandedBits(Root->OpInst);
     MaxBitWidth = std::max<unsigned>(
         Mask.getBitWidth() - Mask.countLeadingZeros(), MaxBitWidth);
   }
 
   // True if the roots can be zero-extended back to their original type, rather
   // than sign-extended. We know that if the leading bits are not demanded, we
   // can safely zero-extend. So we initialize IsKnownPositive to True.
   bool IsKnownPositive = true;
 
   // If all the bits of the roots are demanded, we can try a little harder to
   // compute a narrower type. This can happen, for example, if the roots are
   // getelementptr indices. InstCombine promotes these indices to the pointer
   // width. Thus, all their bits are technically demanded even though the
   // address computation might be vectorized in a smaller type.
   //
   // We start by looking at each entry that can be demoted. We compute the
   // maximum bit width required to store the scalar by using ValueTracking to
   // compute the number of high-order bits we can truncate.
-  if (MaxBitWidth == DL->getTypeSizeInBits(TreeRoot[0]->getType()) &&
-      llvm::all_of(TreeRoot, [](Value *R) {
-        assert(R->hasOneUse() && "Root should have only one use!");
-        return isa<GetElementPtrInst>(R->user_back());
+  if (MaxBitWidth == DL->getTypeSizeInBits(TreeRoot[0]->Op->getType()) &&
+      llvm::all_of(TreeRoot, [](InstructionOrPseudo *R) {
+        assert(R->Op->hasOneUse() && "Root should have only one use!");
+        return isa<GetElementPtrInst>(R->Op->user_back());
       })) {
     MaxBitWidth = 8u;
 
     // Determine if the sign bit of all the roots is known to be zero. If not,
     // IsKnownPositive is set to False.
-    IsKnownPositive = llvm::all_of(TreeRoot, [&](Value *R) {
-      KnownBits Known = computeKnownBits(R, *DL);
+    IsKnownPositive = llvm::all_of(TreeRoot, [&](InstructionOrPseudo *R) {
+      KnownBits Known = computeKnownBits(R->Op, *DL);
       return Known.isNonNegative();
     });
 
     // Determine the maximum number of bits required to store the scalar
     // values.
     for (auto *Scalar : ToDemote) {
       auto NumSignBits = ComputeNumSignBits(Scalar, *DL, 0, AC, nullptr, DT);
       auto NumTypeBits = DL->getTypeSizeInBits(Scalar->getType());
bool SLPVectorizerPass::tryToVectorizeList(ArrayRef<Value *> VL, BoUpSLP &R,
   LLVM_DEBUG(dbgs() << "SLP: Trying to vectorize a list of length = "
                     << VL.size() << ".\n");
 
   // Check that all of the parts are scalar instructions of the same type,
   // we permit an alternate opcode via InstructionsState.
   InstructionsState S = getSameOpcode(VL);
   if (!S.getOpcode())
     return false;
+  for (Value *V : VL) {
+    auto *I = dyn_cast<Instruction>(V);
+    if (isOneOf(S, I) != I)
+      return false;
+  }
 
   Instruction *I0 = cast<Instruction>(S.OpValue);
   unsigned Sz = R.getVectorElementSize(I0);
   unsigned MinVF = std::max(2U, R.getMinVecRegSize() / Sz);
   unsigned MaxVF = std::max<unsigned>(PowerOf2Floor(VL.size()), MinVF);
   if (MaxVF < 2) {
     R.getORE()->emit([&]() {
       return OptimizationRemarkMissed(SV_NAME, "SmallVF", I0)
while (i < NumReducedVals - ReduxWidth + 1 && ReduxWidth > 2) {
 
       // Emit a reduction.
       Builder.SetInsertPoint(cast<Instruction>(ReductionRoot));
       Value *ReducedSubTree =
           emitReduction(VectorizedRoot, Builder, ReduxWidth, TTI);
       if (VectorizedTree) {
         Builder.SetCurrentDebugLocation(Loc);
         OperationData VectReductionData(ReductionData.getOpcode(),
                                         VectorizedTree, ReducedSubTree,
                                         ReductionData.getKind());
         VectorizedTree =
             VectReductionData.createOp(Builder, "op.rdx", ReductionOps);
       } else
         VectorizedTree = ReducedSubTree;
       i += ReduxWidth;
       ReduxWidth = PowerOf2Floor(NumReducedVals - i);
     }
 
     if (VectorizedTree) {
       // Finish the reduction.
       for (; i < NumReducedVals; ++i) {
         auto *I = cast<Instruction>(ReducedVals[i]);
         Builder.SetCurrentDebugLocation(I->getDebugLoc());
         OperationData VectReductionData(ReductionData.getOpcode(),
                                         VectorizedTree, I,
                                         ReductionData.getKind());
         VectorizedTree = VectReductionData.createOp(Builder, "", ReductionOps);
       }
       for (auto &Pair : ExternallyUsedValues) {
         // Add each externally used value to the final reduction.
         for (auto *I : Pair.second) {
           Builder.SetCurrentDebugLocation(I->getDebugLoc());
           OperationData VectReductionData(ReductionData.getOpcode(),
                                           VectorizedTree, Pair.first,
                                           ReductionData.getKind());
           VectorizedTree = VectReductionData.createOp(Builder, "op.extra", I);
         }
       }
       // Update users.
       ReductionRoot->replaceAllUsesWith(VectorizedTree);
     }
     return VectorizedTree != nullptr;
for (unsigned i = ReduxWidth / 2; i != 0; i >>= 1) {
           createRdxShuffleMask(ReduxWidth, i, true, false, Builder);
 
       Value *LeftShuf = Builder.CreateShuffleVector(
           TmpVec, UndefValue::get(TmpVec->getType()), LeftMask, "rdx.shuf.l");
       Value *RightShuf = Builder.CreateShuffleVector(
           TmpVec, UndefValue::get(TmpVec->getType()), (RightMask),
           "rdx.shuf.r");
       OperationData VectReductionData(ReductionData.getOpcode(), LeftShuf,
                                       RightShuf, ReductionData.getKind());
       TmpVec = VectReductionData.createOp(Builder, "op.rdx", ReductionOps);
     }
 
     // The result is in the first element of the vector.
     return Builder.CreateExtractElement(TmpVec, Builder.getInt32(0));
   }
 };
 

test/Transforms/SLPVectorizer/X86/cancel_scheduling.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -slp-vectorizer -S | FileCheck %s
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; This testcase shows the failure of scheduling bundles after calling
+; cancelScheduling() in tryScheduleBundle() and not cleaning all
+; dependencies. The dependency values are supposed to be cleared,
+; since everything was calculated before we cancel the bundle.
+
+define dso_local void @fn1() local_unnamed_addr #0 {
+; CHECK-LABEL: @fn1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 2
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 1
+; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 0
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 4
+; CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 3
+; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 17
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX5]], align 2
+; CHECK-NEXT:    [[ARRAYIDX6:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 16
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX6]], align 2
+; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 15
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX7]], align 2
+; CHECK-NEXT:    [[ARRAYIDX8:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 12
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX8]], align 2
+; CHECK-NEXT:    [[ARRAYIDX9:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 11
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX9]], align 2
+; CHECK-NEXT:    [[ARRAYIDX10:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 10
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX10]], align 2
+; CHECK-NEXT:    [[ARRAYIDX11:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 9
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX11]], align 2
+; CHECK-NEXT:    [[ARRAYIDX12:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 8
+; CHECK-NEXT:    store i16 7, i16* [[ARRAYIDX12]], align 2
+; CHECK-NEXT:    [[ARRAYIDX13:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 7
+; CHECK-NEXT:    [[ARRAYIDX14:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 6
+; CHECK-NEXT:    [[ARRAYIDX15:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 5
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[ARRAYIDX2]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> <i16 2, i16 2, i16 2, i16 0, i16 0, i16 7, i16 7, i16 7>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP1:%.*]] = load i8, i8* inttoptr (i64 1 to i8*), align 1
+; CHECK-NEXT:    [[ARRAYIDX17:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 18
+; CHECK-NEXT:    [[TMP2:%.*]] = lshr i8 [[TMP1]], 2
+; CHECK-NEXT:    [[ARRAYIDX21:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 19
+; CHECK-NEXT:    [[TMP3:%.*]] = and i8 [[TMP1]], 2
+; CHECK-NEXT:    [[ARRAYIDX26:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 20
+; CHECK-NEXT:    [[TMP4:%.*]] = load i8, i8* inttoptr (i64 2 to i8*), align 2
+; CHECK-NEXT:    [[TMP5:%.*]] = lshr i8 [[TMP4]], 4
+; CHECK-NEXT:    [[ARRAYIDX34:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 21
+; CHECK-NEXT:    [[TMP6:%.*]] = lshr i8 [[TMP4]], 1
+; CHECK-NEXT:    [[ARRAYIDX39:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 22
+; CHECK-NEXT:    [[ARRAYIDX44:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 23
+; CHECK-NEXT:    [[TMP7:%.*]] = load i8, i8* inttoptr (i64 3 to i8*), align 1
+; CHECK-NEXT:    [[TMP8:%.*]] = lshr i8 [[TMP7]], 3
+; CHECK-NEXT:    [[ARRAYIDX49:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 24
+; CHECK-NEXT:    [[ARRAYIDX54:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 25
+; CHECK-NEXT:    [[TMP9:%.*]] = load i8, i8* inttoptr (i64 4 to i8*), align 4
+; CHECK-NEXT:    [[TMP10:%.*]] = lshr i8 [[TMP9]], 4
+; CHECK-NEXT:    [[ARRAYIDX59:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 26
+; CHECK-NEXT:    [[TMP11:%.*]] = lshr i8 [[TMP9]], 1
+; CHECK-NEXT:    [[ARRAYIDX64:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 27
+; CHECK-NEXT:    [[ARRAYIDX69:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 28
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, i8* inttoptr (i64 5 to i8*), align 1
+; CHECK-NEXT:    [[TMP13:%.*]] = lshr i8 [[TMP12]], 3
+; CHECK-NEXT:    [[ARRAYIDX74:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 29
+; CHECK-NEXT:    [[ARRAYIDX79:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 30
+; CHECK-NEXT:    [[TMP14:%.*]] = load i8, i8* inttoptr (i64 6 to i8*), align 2
+; CHECK-NEXT:    [[ARRAYIDX83:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 31
+; CHECK-NEXT:    [[TMP15:%.*]] = lshr i8 [[TMP14]], 2
+; CHECK-NEXT:    [[ARRAYIDX88:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 32
+; CHECK-NEXT:    [[TMP16:%.*]] = shl i8 [[TMP14]], 1
+; CHECK-NEXT:    [[TMP17:%.*]] = or i8 [[TMP3]], 1
+; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <16 x i8> undef, i8 [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <16 x i8> [[TMP18]], i8 [[TMP2]], i32 1
+; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <16 x i8> [[TMP19]], i8 [[TMP17]], i32 2
+; CHECK-NEXT:    [[TMP21:%.*]] = insertelement <16 x i8> [[TMP20]], i8 [[TMP5]], i32 3
+; CHECK-NEXT:    [[TMP22:%.*]] = insertelement <16 x i8> [[TMP21]], i8 [[TMP6]], i32 4
+; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <16 x i8> [[TMP22]], i8 [[TMP4]], i32 5
+; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <16 x i8> [[TMP23]], i8 [[TMP8]], i32 6
+; CHECK-NEXT:    [[TMP25:%.*]] = insertelement <16 x i8> [[TMP24]], i8 [[TMP7]], i32 7
+; CHECK-NEXT:    [[TMP26:%.*]] = insertelement <16 x i8> [[TMP25]], i8 [[TMP10]], i32 8
+; CHECK-NEXT:    [[TMP27:%.*]] = insertelement <16 x i8> [[TMP26]], i8 [[TMP11]], i32 9
+; CHECK-NEXT:    [[TMP28:%.*]] = insertelement <16 x i8> [[TMP27]], i8 [[TMP9]], i32 10
+; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <16 x i8> [[TMP28]], i8 [[TMP13]], i32 11
+; CHECK-NEXT:    [[TMP30:%.*]] = insertelement <16 x i8> [[TMP29]], i8 [[TMP12]], i32 12
+; CHECK-NEXT:    [[TMP31:%.*]] = insertelement <16 x i8> [[TMP30]], i8 [[TMP14]], i32 13
+; CHECK-NEXT:    [[TMP32:%.*]] = insertelement <16 x i8> [[TMP31]], i8 [[TMP15]], i32 14
+; CHECK-NEXT:    [[TMP33:%.*]] = insertelement <16 x i8> [[TMP32]], i8 [[TMP16]], i32 15
+; CHECK-NEXT:    [[TMP34:%.*]] = ashr <16 x i8> [[TMP33]], <i8 7, i8 7, i8 0, i8 7, i8 7, i8 2, i8 7, i8 7, i8 7, i8 7, i8 2, i8 7, i8 7, i8 7, i8 7, i8 6>
+; CHECK-NEXT:    [[TMP35:%.*]] = and <16 x i8> [[TMP33]], <i8 7, i8 7, i8 0, i8 7, i8 7, i8 2, i8 7, i8 7, i8 7, i8 7, i8 2, i8 7, i8 7, i8 7, i8 7, i8 6>
+; CHECK-NEXT:    [[TMP36:%.*]] = shufflevector <16 x i8> [[TMP34]], <16 x i8> [[TMP35]], <16 x i32> <i32 0, i32 17, i32 2, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+; CHECK-NEXT:    [[TMP37:%.*]] = sext <16 x i8> [[TMP36]] to <16 x i16>
+; CHECK-NEXT:    [[TMP38:%.*]] = zext <16 x i8> [[TMP36]] to <16 x i16>
+; CHECK-NEXT:    [[TMP39:%.*]] = shufflevector <16 x i16> [[TMP37]], <16 x i16> [[TMP38]], <16 x i32> <i32 0, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+; CHECK-NEXT:    [[ARRAYIDX92:%.*]] = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 33
+; CHECK-NEXT:    [[TMP40:%.*]] = bitcast i16* [[ARRAYIDX17]] to <16 x i16>*
+; CHECK-NEXT:    store <16 x i16> [[TMP39]], <16 x i16>* [[TMP40]], align 2
+; CHECK-NEXT:    ret void
+;
+entry:
+  %arrayidx = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 2
+  store i16 2, i16* %arrayidx, align 2
+  %arrayidx1 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 1
+  store i16 2, i16* %arrayidx1, align 2
+  %arrayidx2 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 0
+  store i16 2, i16* %arrayidx2, align 2
+  %arrayidx3 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 4
+  store i16 0, i16* %arrayidx3, align 2
+  %arrayidx4 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 3
+  store i16 0, i16* %arrayidx4, align 2
+  %arrayidx5 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 17
+  store i16 7, i16* %arrayidx5, align 2
+  %arrayidx6 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 16
+  store i16 7, i16* %arrayidx6, align 2
+  %arrayidx7 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 15
+  store i16 7, i16* %arrayidx7, align 2
+  %arrayidx8 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 12
+  store i16 7, i16* %arrayidx8, align 2
+  %arrayidx9 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 11
+  store i16 7, i16* %arrayidx9, align 2
+  %arrayidx10 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 10
+  store i16 7, i16* %arrayidx10, align 2
+  %arrayidx11 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 9
+  store i16 7, i16* %arrayidx11, align 2
+  %arrayidx12 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 8
+  store i16 7, i16* %arrayidx12, align 2
+  %arrayidx13 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 7
+  store i16 7, i16* %arrayidx13, align 2
+  %arrayidx14 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 6
+  store i16 7, i16* %arrayidx14, align 2
+  %arrayidx15 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 5
+  store i16 7, i16* %arrayidx15, align 2
+  %0 = load i8, i8* inttoptr (i64 1 to i8*), align 1
+  %1 = ashr i8 %0, 7
+  %conv16 = sext i8 %1 to i16
+  %arrayidx17 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 18
+  store i16 %conv16, i16* %arrayidx17, align 2
+  %2 = lshr i8 %0, 2
+  %3 = and i8 %2, 7
+  %conv20 = zext i8 %3 to i16
+  %arrayidx21 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 19
+  store i16 %conv20, i16* %arrayidx21, align 2
+  %4 = and i8 %0, 2
+  %arrayidx26 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 20
+  %5 = or i8 %4, 1
+  %conv29 = zext i8 %5 to i16
+  store i16 %conv29, i16* %arrayidx26, align 2
+  %6 = load i8, i8* inttoptr (i64 2 to i8*), align 2
+  %7 = lshr i8 %6, 4
+  %8 = and i8 %7, 7
+  %conv33 = zext i8 %8 to i16
+  %arrayidx34 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 21
+  store i16 %conv33, i16* %arrayidx34, align 2
+  %9 = lshr i8 %6, 1
+  %10 = and i8 %9, 7
+  %conv38 = zext i8 %10 to i16
+  %arrayidx39 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 22
+  store i16 %conv38, i16* %arrayidx39, align 2
+  %11 = and i8 %6, 2
+  %conv43 = zext i8 %11 to i16
+  %arrayidx44 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 23
+  store i16 %conv43, i16* %arrayidx44, align 2
+  %12 = load i8, i8* inttoptr (i64 3 to i8*), align 1
+  %13 = lshr i8 %12, 3
+  %14 = and i8 %13, 7
+  %conv48 = zext i8 %14 to i16
+  %arrayidx49 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 24
+  store i16 %conv48, i16* %arrayidx49, align 2
+  %15 = and i8 %12, 7
+  %conv53 = zext i8 %15 to i16
+  %arrayidx54 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 25
+  store i16 %conv53, i16* %arrayidx54, align 2
+  %16 = load i8, i8* inttoptr (i64 4 to i8*), align 4
+  %17 = lshr i8 %16, 4
+  %18 = and i8 %17, 7
+  %conv58 = zext i8 %18 to i16
+  %arrayidx59 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 26
+  store i16 %conv58, i16* %arrayidx59, align 2
+  %19 = lshr i8 %16, 1
+  %20 = and i8 %19, 7
+  %conv63 = zext i8 %20 to i16
+  %arrayidx64 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 27
+  store i16 %conv63, i16* %arrayidx64, align 2
+  %21 = and i8 %16, 2
+  %conv68 = zext i8 %21 to i16
+  %arrayidx69 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 28
+  store i16 %conv68, i16* %arrayidx69, align 2
+  %22 = load i8, i8* inttoptr (i64 5 to i8*), align 1
+  %23 = lshr i8 %22, 3
+  %24 = and i8 %23, 7
+  %conv73 = zext i8 %24 to i16
+  %arrayidx74 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 29
+  store i16 %conv73, i16* %arrayidx74, align 2
+  %25 = and i8 %22, 7
+  %conv78 = zext i8 %25 to i16
+  %arrayidx79 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 30
+  store i16 %conv78, i16* %arrayidx79, align 2
+  %26 = load i8, i8* inttoptr (i64 6 to i8*), align 2
+  %27 = and i8 %26, 7
+  %conv82 = zext i8 %27 to i16
+  %arrayidx83 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 31
+  store i16 %conv82, i16* %arrayidx83, align 2
+  %28 = lshr i8 %26, 2
+  %29 = and i8 %28, 7
+  %conv87 = zext i8 %29 to i16
+  %arrayidx88 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 32
+  store i16 %conv87, i16* %arrayidx88, align 2
+  %30 = shl i8 %26, 1
+  %31 = and i8 %30, 6
+  %conv91 = zext i8 %31 to i16
+  %arrayidx92 = getelementptr inbounds [4 x i16], [4 x i16]* undef, i64 0, i64 33
+  store i16 %conv91, i16* %arrayidx92, align 2
+  ret void
+}
+
+attributes #0 = { nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="broadwell" "target-features"="+adx,+aes,+avx,+avx2,+bmi,+bmi2,+cx16,+f16c,+fma,+fsgsbase,+fxsr,+invpcid,+lzcnt,+mmx,+movbe,+pclmul,+popcnt,+prfchw,+rdrnd,+rdseed,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave,+xsaveopt" "unsafe-fp-math"="false" "use-soft-float"="false" }

test/Transforms/SLPVectorizer/X86/insert-after-multiple-bundle.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -slp-vectorizer -slp-vectorizer -mcpu=bdver1 < %s | FileCheck %s
+
+; Function Attrs: nounwind uwtable
+define dso_local void @fn1() local_unnamed_addr #0 {
+; CHECK-LABEL: @fn1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[TMP0:%.*]] = phi <4 x i32> [ undef, [[ENTRY:%.*]] ], [ [[TMP7:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* undef, align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i32> <i32 0, i32 undef>, i32 [[TMP1]], i32 1
+; CHECK-NEXT:    [[TMP3:%.*]] = add nsw <2 x i32> [[TMP2]], zeroinitializer
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i32> [[TMP3]], <2 x i32> undef, <4 x i32> <i32 0, i32 0, i32 1, i32 1>
+; CHECK-NEXT:    [[TMP4:%.*]] = shl nsw <4 x i32> [[SHUFFLE]], <i32 1, i32 2, i32 1, i32 2>
+; CHECK-NEXT:    [[TMP5:%.*]] = sub nsw <4 x i32> undef, [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = icmp slt <4 x i32> [[TMP0]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7]] = select <4 x i1> [[TMP6]], <4 x i32> [[TMP5]], <4 x i32> [[TMP0]]
+; CHECK-NEXT:    br label [[FOR_BODY]]
+;
+entry:
+  br label %for.body
+
+for.body:                                         ; preds = %for.body, %entry
+  %j.064 = phi i32 [ undef, %entry ], [ %spec.select, %for.body ]
+  %k.063 = phi i32 [ undef, %entry ], [ %k.1, %for.body ]
+  %l.062 = phi i32 [ undef, %entry ], [ %spec.select58, %for.body ]
+  %m.061 = phi i32 [ undef, %entry ], [ %m.1, %for.body ]
+  %conv = zext i8 undef to i32
+  %mul = shl nuw nsw i32 %conv, 1
+  %sub = sub nsw i32 undef, %mul
+  %mul4 = shl nuw nsw i32 %conv, 2
+  %sub5 = sub nsw i32 undef, %mul4
+  %conv8 = zext i8 undef to i32
+  %0 = load i32, i32* undef, align 4
+  %add = add nsw i32 %0, %conv8
+  %mul11 = shl nsw i32 %add, 1
+  %sub12 = sub nsw i32 undef, %mul11
+  %mul19 = shl nsw i32 %add, 2
+  %sub20 = sub nsw i32 undef, %mul19
+  %cmp = icmp slt i32 %j.064, %sub
+  %spec.select = select i1 %cmp, i32 %sub, i32 %j.064
+  %cmp22 = icmp slt i32 %k.063, %sub5
+  %k.1 = select i1 %cmp22, i32 %sub5, i32 %k.063
+  %cmp26 = icmp slt i32 %l.062, %sub12
+  %spec.select58 = select i1 %cmp26, i32 %sub12, i32 %l.062
+  %cmp30 = icmp slt i32 %m.061, %sub20
+  %m.1 = select i1 %cmp30, i32 %sub20, i32 %m.061
+  br label %for.body
+}
+
+; Function Attrs: nounwind uwtable
+define dso_local void @axis_to_quat() local_unnamed_addr #0 {
+; CHECK-LABEL: @axis_to_quat(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[CONV:%.*]] = fptrunc double undef to float
+; CHECK-NEXT:    [[MUL:%.*]] = fmul fast float undef, [[CONV]]
+; CHECK-NEXT:    store float [[MUL]], float* undef, align 4
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds float, float* undef, i64 1
+; CHECK-NEXT:    [[MUL2:%.*]] = fmul fast float undef, [[CONV]]
+; CHECK-NEXT:    store float [[MUL2]], float* [[ARRAYIDX1]], align 4
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds float, float* undef, i64 2
+; CHECK-NEXT:    [[MUL4:%.*]] = fmul fast float undef, [[CONV]]
+; CHECK-NEXT:    store float [[MUL4]], float* [[ARRAYIDX3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = tail call fast double @llvm.cos.f64(double 0x7FF8000000000000)
+; CHECK-NEXT:    [[CONV6:%.*]] = fptrunc double [[TMP0]] to float
+; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds float, float* undef, i64 3
+; CHECK-NEXT:    store float [[CONV6]], float* [[ARRAYIDX7]], align 4
+; CHECK-NEXT:    ret void
+;
+entry:
+  %conv = fptrunc double undef to float
+  %mul = fmul fast float undef, %conv
+  store float %mul, float* undef, align 4
+  %arrayidx1 = getelementptr inbounds float, float* undef, i64 1
+  %mul2 = fmul fast float undef, %conv
+  store float %mul2, float* %arrayidx1, align 4
+  %arrayidx3 = getelementptr inbounds float, float* undef, i64 2
+  %mul4 = fmul fast float undef, %conv
+  store float %mul4, float* %arrayidx3, align 4
+  %0 = tail call fast double @llvm.cos.f64(double 0x7FF8000000000000)
+  %conv6 = fptrunc double %0 to float
+  %arrayidx7 = getelementptr inbounds float, float* undef, i64 3
+  store float %conv6, float* %arrayidx7, align 4
+  ret void
+}
+
+; Function Attrs: nounwind readnone speculatable
+declare double @llvm.cos.f64(double) #1

test/Transforms/SLPVectorizer/X86/invoke.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -slp-vectorizer -slp-vectorizer -mcpu=bdver1 < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+$_Z19BuildCT_ELEMENT_ONEI1AIS0_IS0_IiP6vectorI1G1BIS2_EEES6_ES6_ES5_S9_EvT_T0_T1_ = comdat any
+
+@c = external dso_local global i32, align 4
+
+; Function Attrs: uwtable
+define dso_local void @_Z19BuildCT_ELEMENT_ONEI1AIS0_IS0_IiP6vectorI1G1BIS2_EEES6_ES6_ES5_S9_EvT_T0_T1_() local_unnamed_addr #0 comdat personality i32 (...)* @__gxx_personality_v0 {
+; CHECK-LABEL: @_Z19BuildCT_ELEMENT_ONEI1AIS0_IS0_IiP6vectorI1G1BIS2_EEES6_ES6_ES5_S9_EvT_T0_T1_(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    invoke void @_ZN6vectorI1II1G15compiletime_valIlLin1EEE1BIS4_EEixEl()
+; CHECK-NEXT:    to label [[INVOKE_CONT_I:%.*]] unwind label [[LPAD_I:%.*]]
+; CHECK:       invoke.cont.i:
+; CHECK-NEXT:    invoke void @_ZN1HC2EPi()
+; CHECK-NEXT:    to label [[DOTNOEXC_I:%.*]] unwind label [[LPAD_I]]
+; CHECK:       .noexc.i:
+; CHECK-NEXT:    [[CALL_I_I7_I:%.*]] = invoke i64 @_ZN1G13getInitialMinEv()
+; CHECK-NEXT:    to label [[CALL_I_I_NOEXC_I:%.*]] unwind label [[LPAD_I]]
+; CHECK:       call.i.i.noexc.i:
+; CHECK-NEXT:    [[ADD_I_I_I:%.*]] = add nsw i64 [[CALL_I_I7_I]], -1
+; CHECK-NEXT:    [[CALL_I13_I8_I:%.*]] = invoke i64 @_ZN1G13getInitialMaxEv()
+; CHECK-NEXT:    to label [[CALL_I13_I_NOEXC_I:%.*]] unwind label [[LPAD_I]]
+; CHECK:       call.i13.i.noexc.i:
+; CHECK-NEXT:    [[__B___A_I_I_I:%.*]] = select i1 undef, i64 [[ADD_I_I_I]], i64 0
+; CHECK-NEXT:    [[__B___A_I11_I_I:%.*]] = select i1 undef, i64 [[CALL_I13_I8_I]], i64 ptrtoint (void ()* @_Z10array_sizev to i64)
+; CHECK-NEXT:    [[CMP_I_I:%.*]] = icmp slt i64 [[__B___A_I_I_I]], [[__B___A_I11_I_I]]
+; CHECK-NEXT:    unreachable
+; CHECK:       lpad.i:
+; CHECK-NEXT:    [[TMP0:%.*]] = landingpad { i8*, i32 }
+; CHECK-NEXT:    cleanup
+; CHECK-NEXT:    resume { i8*, i32 } [[TMP0]]
+;
+entry:
+  invoke void @_ZN6vectorI1II1G15compiletime_valIlLin1EEE1BIS4_EEixEl()
+  to label %invoke.cont.i unwind label %lpad.i
+
+invoke.cont.i:                                    ; preds = %entry
+  invoke void @_ZN1HC2EPi()
+  to label %.noexc.i unwind label %lpad.i
+
+.noexc.i:                                         ; preds = %invoke.cont.i
+  %call.i.i7.i = invoke i64 @_ZN1G13getInitialMinEv()
+  to label %call.i.i.noexc.i unwind label %lpad.i
+
+call.i.i.noexc.i:                                 ; preds = %.noexc.i
+  %add.i.i.i = add nsw i64 %call.i.i7.i, -1
+  %call.i13.i8.i = invoke i64 @_ZN1G13getInitialMaxEv()
+  to label %call.i13.i.noexc.i unwind label %lpad.i
+
+call.i13.i.noexc.i:                               ; preds = %call.i.i.noexc.i
+  %__b.__a.i.i.i = select i1 undef, i64 %add.i.i.i, i64 0
+  %__b.__a.i11.i.i = select i1 undef, i64 %call.i13.i8.i, i64 ptrtoint (void ()* @_Z10array_sizev to i64)
+  %cmp.i.i = icmp slt i64 %__b.__a.i.i.i, %__b.__a.i11.i.i
+  unreachable
+
+lpad.i:                                           ; preds = %call.i.i.noexc.i, %.noexc.i, %invoke.cont.i, %entry
+  %0 = landingpad { i8*, i32 }
+  cleanup
+  resume { i8*, i32 } %0
+}
+
+declare dso_local void @_ZN6vectorI1II1G15compiletime_valIlLin1EEE1BIS4_EEixEl() local_unnamed_addr #1
+
+declare dso_local i32 @__gxx_personality_v0(...)
+
+declare dso_local void @_ZN1HC2EPi() unnamed_addr #1
+
+declare dso_local void @_Z10array_sizev() #1
+
+declare dso_local i64 @_ZN1G13getInitialMinEv() local_unnamed_addr #1
+
+declare dso_local i64 @_ZN1G13getInitialMaxEv() local_unnamed_addr #1
+

test/Transforms/SLPVectorizer/X86/memory-dep.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -slp-vectorizer -S | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+%struct.anon.1.2.3.4.87 = type { [6 x [6 x i16]], [6 x [6 x i32]], [0 x [4 x [4 x i32]]] }
+
+@f = external dso_local local_unnamed_addr global %struct.anon.1.2.3.4.87, align 4
+
+; Function Attrs: norecurse nounwind uwtable
+define dso_local void @itrans() local_unnamed_addr #0 {
+; CHECK-LABEL: @itrans(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    store i32 undef, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 0, i64 3), align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 0, i64 2), align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 0, i64 3), align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <4 x i32> undef, i32 undef, i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <4 x i32> [[TMP2]], i32 undef, i32 1
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <4 x i32> [[TMP3]], i32 [[TMP0]], i32 2
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <4 x i32> [[TMP4]], i32 [[TMP1]], i32 3
+; CHECK-NEXT:    [[TMP6:%.*]] = add nsw <4 x i32> <i32 2, i32 2, i32 2, i32 2>, [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = sub <4 x i32> [[TMP6]], undef
+; CHECK-NEXT:    [[TMP8:%.*]] = add nsw <4 x i32> undef, [[TMP7]]
+; CHECK-NEXT:    [[TMP9:%.*]] = shl <4 x i32> [[TMP8]], <i32 6, i32 6, i32 6, i32 6>
+; CHECK-NEXT:    [[TMP10:%.*]] = icmp slt <4 x i32> [[TMP9]], <i32 1, i32 1, i32 1, i32 1>
+; CHECK-NEXT:    [[TMP11:%.*]] = zext <4 x i1> [[TMP10]] to <4 x i32>
+; CHECK-NEXT:    [[TMP12:%.*]] = icmp slt <4 x i32> undef, [[TMP11]]
+; CHECK-NEXT:    [[TMP13:%.*]] = zext <4 x i1> [[TMP12]] to <4 x i32>
+; CHECK-NEXT:    store <4 x i32> [[TMP13]], <4 x i32>* bitcast (i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 3, i64 0) to <4 x i32>*), align 4
+; CHECK-NEXT:    ret void
+;
+entry:
+  %add8 = add nsw i32 undef, undef
+  store i32 undef, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 0, i64 3), align 4
+  %add15 = add nsw i32 undef, undef
+  %add26 = add nsw i32 %add8, 2
+  %sub27 = sub i32 %add26, undef
+  %add33 = add nsw i32 %sub27, undef
+  %shl = shl i32 %add33, 6
+  %cmp.i = icmp slt i32 %shl, 1
+  %conv.i = zext i1 %cmp.i to i32
+  %cmp1.i = icmp slt i32 undef, %conv.i
+  %conv2.i = zext i1 %cmp1.i to i32
+  store i32 %conv2.i, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 3, i64 0), align 4
+  %add26.1 = add nsw i32 %add15, 2
+  %sub27.1 = sub i32 %add26.1, undef
+  %add33.1 = add nsw i32 %sub27.1, undef
+  %shl.1 = shl i32 %add33.1, 6
+  %cmp.i.1 = icmp slt i32 %shl.1, 1
+  %conv.i.1 = zext i1 %cmp.i.1 to i32
+  %cmp1.i.1 = icmp slt i32 undef, %conv.i.1
+  %conv2.i.1 = zext i1 %cmp1.i.1 to i32
+  store i32 %conv2.i.1, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 3, i64 1), align 4
+  %0 = load i32, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 0, i64 2), align 4
+  %add26.2 = add nsw i32 %0, 2
+  %sub27.2 = sub i32 %add26.2, undef
+  %add33.2 = add nsw i32 %sub27.2, undef
+  %shl.2 = shl i32 %add33.2, 6
+  %cmp.i.2 = icmp slt i32 %shl.2, 1
+  %conv.i.2 = zext i1 %cmp.i.2 to i32
+  %cmp1.i.2 = icmp slt i32 undef, %conv.i.2
+  %conv2.i.2 = zext i1 %cmp1.i.2 to i32
+  store i32 %conv2.i.2, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 3, i64 2), align 4
+  %1 = load i32, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 0, i64 3), align 4
+  %add26.3 = add nsw i32 %1, 2
+  %sub27.3 = sub i32 %add26.3, undef
+  %add33.3 = add nsw i32 %sub27.3, undef
+  %shl.3 = shl i32 %add33.3, 6
+  %cmp.i.3 = icmp slt i32 %shl.3, 1
+  %conv.i.3 = zext i1 %cmp.i.3 to i32
+  %cmp1.i.3 = icmp slt i32 undef, %conv.i.3
+  %conv2.i.3 = zext i1 %cmp1.i.3 to i32
+  store i32 %conv2.i.3, i32* getelementptr inbounds (%struct.anon.1.2.3.4.87, %struct.anon.1.2.3.4.87* @f, i64 0, i32 1, i64 3, i64 3), align 4
+  ret void
+}

test/Transforms/SLPVectorizer/X86/pr35497.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -slp-vectorizer -S -mtriple=x86_64-unknown-linux-gnu < %s | FileCheck %s
 
 %class.1 = type { %class.2 }
 %class.2 = type { %"class.3" }
 %"class.3" = type { %"struct.1", i64 }
 %"struct.1" = type { [8 x i64] }
 
 $_ZN1C10SwitchModeEv = comdat any
 
 ; Function Attrs: uwtable
 define void @_ZN1C10SwitchModeEv() local_unnamed_addr #0 comdat align 2 {
 ; CHECK-LABEL: @_ZN1C10SwitchModeEv(
 ; CHECK-NEXT:  for.body.lr.ph.i:
-; CHECK-NEXT:    [[OR_1:%.*]] = or i64 undef, 1
-; CHECK-NEXT:    store i64 [[OR_1]], i64* undef, align 8
+; CHECK-NEXT:    [[BAR5:%.*]] = load i64, i64* undef, align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x i64> undef, i64 [[BAR5]], i32 1
+; CHECK-NEXT:    [[TMP1:%.*]] = or <2 x i64> [[TMP0]], <i64 1, i64 0>
+; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <2 x i64> [[TMP1]], i32 0
+; CHECK-NEXT:    store i64 [[TMP2]], i64* undef, align 8
 ; CHECK-NEXT:    [[FOO_1:%.*]] = getelementptr inbounds [[CLASS_1:%.*]], %class.1* undef, i64 0, i32 0, i32 0, i32 0, i32 0, i64 0
 ; CHECK-NEXT:    [[FOO_2:%.*]] = getelementptr inbounds [[CLASS_1]], %class.1* undef, i64 0, i32 0, i32 0, i32 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i64* [[FOO_1]] to <2 x i64>*
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x i64>, <2 x i64>* [[TMP0]], align 8
-; CHECK-NEXT:    [[BAR5:%.*]] = load i64, i64* undef, align 8
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x i64> undef, i64 [[OR_1]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x i64> [[TMP2]], i64 [[BAR5]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = and <2 x i64> [[TMP3]], [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i64* [[FOO_1]] to <2 x i64>*
+; CHECK-NEXT:    [[TMP4:%.*]] = load <2 x i64>, <2 x i64>* [[TMP3]], align 8
+; CHECK-NEXT:    [[TMP5:%.*]] = and <2 x i64> [[TMP1]], [[TMP4]]
 ; CHECK-NEXT:    [[BAR3:%.*]] = getelementptr inbounds [[CLASS_2:%.*]], %class.2* undef, i64 0, i32 0, i32 0, i32 0, i64 0
 ; CHECK-NEXT:    [[BAR4:%.*]] = getelementptr inbounds [[CLASS_2]], %class.2* undef, i64 0, i32 0, i32 0, i32 0, i64 1
-; CHECK-NEXT:    [[TMP5:%.*]] = bitcast i64* [[BAR3]] to <2 x i64>*
-; CHECK-NEXT:    store <2 x i64> [[TMP4]], <2 x i64>* [[TMP5]], align 8
+; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i64* [[BAR3]] to <2 x i64>*
+; CHECK-NEXT:    store <2 x i64> [[TMP5]], <2 x i64>* [[TMP6]], align 8
 ; CHECK-NEXT:    ret void
 ;
 for.body.lr.ph.i:
   %or.1 = or i64 undef, 1
   store i64 %or.1, i64* undef, align 8
   %foo.1 = getelementptr inbounds %class.1, %class.1* undef, i64 0, i32 0, i32 0, i32 0, i32 0, i64 0
   %foo.3 = load i64, i64* %foo.1, align 8
   %foo.2 = getelementptr inbounds %class.1, %class.1* undef, i64 0, i32 0, i32 0, i32 0, i32 0, i64 1

test/Transforms/SLPVectorizer/X86/rem-bundle.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -slp-vectorizer -slp-vectorizer -mcpu=bdver1 < %s | FileCheck %s
+
+%struct.anon.0.1.2.3.20 = type { i32, i32, i32, i32 }
+
+; This testcase shows the failure of combining any remainder operation
+; in a bundle with non-alternative operations.
+
+@b = external dso_local local_unnamed_addr global i32, align 4
+@c = external dso_local local_unnamed_addr global %struct.anon.0.1.2.3.20, align 4
+
+; Function Attrs: norecurse nounwind uwtable
+define dso_local void @fn1() local_unnamed_addr #0 {
+; CHECK-LABEL: @fn1(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* @b, align 4
+; CHECK-NEXT:    store i32 [[TMP0]], i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 3), align 4
+; CHECK-NEXT:    [[DIV:%.*]] = sdiv i32 undef, 600
+; CHECK-NEXT:    store i32 [[DIV]], i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 2), align 4
+; CHECK-NEXT:    [[DIV1:%.*]] = sdiv i32 undef, 60
+; CHECK-NEXT:    store i32 [[DIV1]], i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 1), align 4
+; CHECK-NEXT:    [[REM:%.*]] = srem i32 undef, 60
+; CHECK-NEXT:    store i32 [[REM]], i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 0), align 4
+; CHECK-NEXT:    ret void
+;
+entry:
+  %0 = load i32, i32* @b, align 4
+  store i32 %0, i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 3), align 4
+  %div = sdiv i32 undef, 600
+  store i32 %div, i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 2), align 4
+  %div1 = sdiv i32 undef, 60
+  store i32 %div1, i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 1), align 4
+  %rem = srem i32 undef, 60
+  store i32 %rem, i32* getelementptr inbounds (%struct.anon.0.1.2.3.20, %struct.anon.0.1.2.3.20* @c, i64 0, i32 0), align 4
+  ret void
+}

test/Transforms/SLPVectorizer/X86/schedule-bundle1.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -slp-vectorizer -slp-vectorizer -mcpu=bdver1 < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+%"struct.B::Node.1.4.7.10.13.16.19.64.100" = type { %class.A.0.3.6.9.12.15.18.63.99, float, float }
+%class.A.0.3.6.9.12.15.18.63.99 = type { [4 x float] }
+
+@f = external dso_local global %"struct.B::Node.1.4.7.10.13.16.19.64.100", align 4
+
+; Function Attrs: uwtable
+define dso_local void @_ZN1B5m_fn2Ev() local_unnamed_addr #0 align 2 {
+; CHECK-LABEL: @_ZN1B5m_fn2Ev(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[P1_SROA_2_8_VEC_EXTRACT_I:%.*]] = extractelement <2 x float> undef, i32 0
+; CHECK-NEXT:    [[MUL_I:%.*]] = fmul fast float undef, undef
+; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* getelementptr inbounds (%"struct.B::Node.1.4.7.10.13.16.19.64.100", %"struct.B::Node.1.4.7.10.13.16.19.64.100"* @f, i64 0, i32 0, i32 0, i64 2), align 4
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x float> <float 0.000000e+00, float undef>, float [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul fast <2 x float> [[TMP1]], <float 2.000000e+00, float 1.000000e+00>
+; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x float> [[TMP2]], i32 0
+; CHECK-NEXT:    [[MUL_I18:%.*]] = fmul fast float [[TMP3]], undef
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x float> undef, float [[P1_SROA_2_8_VEC_EXTRACT_I]], i32 0
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x float> [[TMP4]], float 1.000000e+00, i32 1
+; CHECK-NEXT:    [[TMP6:%.*]] = fmul fast <2 x float> [[TMP2]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> undef, float [[MUL_I]], i32 0
+; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x float> [[TMP7]], float 1.000000e+00, i32 1
+; CHECK-NEXT:    [[TMP9:%.*]] = fmul fast <2 x float> [[TMP6]], [[TMP8]]
+; CHECK-NEXT:    [[TMP10:%.*]] = fmul fast <2 x float> <float 2.000000e+00, float 2.000000e+00>, [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = fadd fast <2 x float> <float 1.000000e+00, float 1.000000e+00>, [[TMP10]]
+; CHECK-NEXT:    [[TMP12:%.*]] = extractelement <2 x float> [[TMP11]], i32 0
+; CHECK-NEXT:    [[TMP13:%.*]] = extractelement <2 x float> [[TMP11]], i32 1
+; CHECK-NEXT:    [[CMP_I:%.*]] = fcmp fast ogt float [[TMP12]], [[TMP13]]
+; CHECK-NEXT:    unreachable
+;
+entry:
+  %p1.sroa.2.8.vec.extract.i = extractelement <2 x float> undef, i32 0
+  %mul.i = fmul fast float undef, undef
+  %mul.i16 = fmul fast float 0.000000e+00, 2.000000e+00
+  %mul.i18 = fmul fast float %mul.i16, undef
+  %mul3.i20 = fmul fast float %mul.i16, %p1.sroa.2.8.vec.extract.i
+  %mul3.i.i = fmul fast float %mul3.i20, %mul.i
+  %mul.i18.i = fmul fast float %mul3.i.i, 2.000000e+00
+  %add.i19.i = fadd fast float %mul.i18.i, 1.000000e+00
+  %0 = load float, float* getelementptr inbounds (%"struct.B::Node.1.4.7.10.13.16.19.64.100", %"struct.B::Node.1.4.7.10.13.16.19.64.100"* @f, i64 0, i32 0, i32 0, i64 2), align 4
+  %mul.i16.i = fmul fast float %0, 2.000000e+00
+  %add.i.i = fadd fast float %mul.i16.i, 1.000000e+00
+  %cmp.i = fcmp fast ogt float %add.i19.i, %add.i.i
+  unreachable
+}

test/Transforms/SLPVectorizer/X86/scheduling_pseudo.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -S -slp-vectorizer -slp-vectorizer -mcpu=bdver1 < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+%class.D.2.7.12.17.332 = type { %class.C.1.6.11.16.331, %class.A.0.5.10.15.330 }
+%class.C.1.6.11.16.331 = type { [3 x %class.A.0.5.10.15.330], double, double, double, double, double, double }
+%class.A.0.5.10.15.330 = type { [4 x double], double, double }
+
+@__trans_tmp_1_1_0 = external dso_local local_unnamed_addr global double, align 8
+@__trans_tmp_3_1_2 = external dso_local local_unnamed_addr global double, align 8
+
+; Function Attrs: uwtable
+define dso_local void @_Z3fn1v() local_unnamed_addr #0 {
+; CHECK-LABEL: @_Z3fn1v(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load double, double* @__trans_tmp_3_1_2, align 8
+; CHECK-NEXT:    [[TMP1:%.*]] = load double, double* @__trans_tmp_1_1_0, align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> undef, double [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[TMP0]], i32 1
+; CHECK-NEXT:    [[TMP4:%.*]] = fptrunc <2 x double> [[TMP3]] to <2 x float>
+; CHECK-NEXT:    [[CALL2_I_I16:%.*]] = call double @_ZN1B5m_fn8Ev()
+; CHECK-NEXT:    [[CALL_I_I:%.*]] = call double @_ZN1F5m_fn9Ev()
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> undef, double [[CALL_I_I]], i32 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> [[TMP5]], double undef, i32 1
+; CHECK-NEXT:    [[TMP7:%.*]] = fdiv fast <2 x double> <double 2.000000e+00, double 2.000000e+00>, [[TMP6]]
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x double> [[TMP7]], i32 1
+; CHECK-NEXT:    [[TMP9:%.*]] = fmul fast <2 x double> undef, [[TMP7]]
+; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <2 x double> [[TMP9]], i32 1
+; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <2 x double> undef, double [[TMP8]], i32 0
+; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <2 x double> [[TMP11]], double [[TMP10]], i32 1
+; CHECK-NEXT:    [[TMP13:%.*]] = fmul fast <2 x double> <double undef, double 1.000000e+00>, [[TMP12]]
+; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <2 x double> [[TMP13]], i32 0
+; CHECK-NEXT:    [[ADD_I_I23:%.*]] = fadd fast double [[TMP14]], [[TMP8]]
+; CHECK-NEXT:    [[TMP15:%.*]] = fpext <2 x float> [[TMP4]] to <2 x double>
+; CHECK-NEXT:    [[TMP16:%.*]] = insertelement <2 x double> undef, double [[CALL2_I_I16]], i32 0
+; CHECK-NEXT:    [[TMP17:%.*]] = extractelement <2 x double> [[TMP7]], i32 0
+; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <2 x double> [[TMP16]], double [[TMP17]], i32 1
+; CHECK-NEXT:    [[TMP19:%.*]] = fmul fast <2 x double> [[TMP9]], [[TMP18]]
+; CHECK-NEXT:    [[TMP20:%.*]] = fadd fast <2 x double> [[TMP19]], undef
+; CHECK-NEXT:    [[TMP21:%.*]] = fmul fast <2 x double> [[TMP19]], undef
+; CHECK-NEXT:    [[TMP22:%.*]] = shufflevector <2 x double> [[TMP20]], <2 x double> [[TMP21]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP23:%.*]] = fadd fast <2 x double> undef, [[TMP15]]
+; CHECK-NEXT:    [[TMP24:%.*]] = extractelement <2 x double> [[TMP15]], i32 1
+; CHECK-NEXT:    [[TMP25:%.*]] = insertelement <2 x double> undef, double [[TMP24]], i32 0
+; CHECK-NEXT:    [[TMP26:%.*]] = insertelement <2 x double> [[TMP25]], double 1.000000e+00, i32 1
+; CHECK-NEXT:    [[TMP27:%.*]] = fmul fast <2 x double> [[TMP13]], [[TMP26]]
+; CHECK-NEXT:    [[TMP28:%.*]] = fadd fast <2 x double> [[TMP23]], [[TMP27]]
+; CHECK-NEXT:    [[AGG_TMP913_SROA_4_0__SROA_IDX201_I:%.*]] = getelementptr inbounds [[CLASS_D_2_7_12_17_332:%.*]], %class.D.2.7.12.17.332* undef, i64 0, i32 0, i32 0, i64 0, i32 0, i64 1
+; CHECK-NEXT:    [[AGG_TMP913_SROA_5_0__SROA_IDX203_I:%.*]] = getelementptr inbounds [[CLASS_D_2_7_12_17_332]], %class.D.2.7.12.17.332* undef, i64 0, i32 0, i32 0, i64 0, i32 0, i64 2
+; CHECK-NEXT:    [[TMP29:%.*]] = bitcast double* [[AGG_TMP913_SROA_4_0__SROA_IDX201_I]] to <2 x double>*
+; CHECK-NEXT:    store <2 x double> [[TMP22]], <2 x double>* [[TMP29]], align 8
+; CHECK-NEXT:    [[AGG_TMP1014_SROA_4_0__SROA_IDX256_I:%.*]] = getelementptr inbounds [[CLASS_D_2_7_12_17_332]], %class.D.2.7.12.17.332* undef, i64 0, i32 1, i32 0, i64 1
+; CHECK-NEXT:    [[AGG_TMP1014_SROA_5_0__SROA_IDX258_I:%.*]] = getelementptr inbounds [[CLASS_D_2_7_12_17_332]], %class.D.2.7.12.17.332* undef, i64 0, i32 1, i32 0, i64 2
+; CHECK-NEXT:    [[TMP30:%.*]] = bitcast double* [[AGG_TMP1014_SROA_4_0__SROA_IDX256_I]] to <2 x double>*
+; CHECK-NEXT:    store <2 x double> [[TMP28]], <2 x double>* [[TMP30]], align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  %0 = load double, double* @__trans_tmp_3_1_2, align 8
+  %conv = fptrunc double %0 to float
+  %1 = load double, double* @__trans_tmp_1_1_0, align 8
+  %conv1 = fptrunc double %1 to float
+  %div.i.i15 = fdiv fast double 2.000000e+00, undef
+  %call2.i.i16 = call double @_ZN1B5m_fn8Ev()
+  %mul.i.i18 = fmul fast double undef, %div.i.i15
+  %mul5.i.i20 = fmul fast double undef, %div.i.i15
+  %add.i.i23 = fadd fast double %mul5.i.i20, %div.i.i15
+  %conv6 = fpext float %conv1 to double
+  %conv7 = fpext float %conv to double
+  %call.i.i = call double @_ZN1F5m_fn9Ev()
+  %div.i.i = fdiv fast double 2.000000e+00, %call.i.i
+  %mul7.i.i = fmul fast double undef, %div.i.i
+  %mul.i25.i = fmul fast double %mul7.i.i, %call2.i.i16
+  %add.i27.i = fadd fast double %mul.i25.i, undef
+  %mul.i20.i = fmul fast double %div.i.i, %mul.i.i18
+  %mul6.i.i = fmul fast double %mul.i20.i, undef
+  %mul.i.i.i = fmul fast double %mul5.i.i20, %conv7
+  %add8.i.i.i = fadd fast double undef, %conv6
+  %add.i.i = fadd fast double %add8.i.i.i, %mul.i.i.i
+  %add16.i.i.i = fadd fast double undef, %conv7
+  %add10.i.i = fadd fast double %add16.i.i.i, %mul.i.i18
+  %agg.tmp913.sroa.4.0..sroa_idx201.i = getelementptr inbounds %class.D.2.7.12.17.332, %class.D.2.7.12.17.332* undef, i64 0, i32 0, i32 0, i64 0, i32 0, i64 1
+  store double %add.i27.i, double* %agg.tmp913.sroa.4.0..sroa_idx201.i, align 8
+  %agg.tmp913.sroa.5.0..sroa_idx203.i = getelementptr inbounds %class.D.2.7.12.17.332, %class.D.2.7.12.17.332* undef, i64 0, i32 0, i32 0, i64 0, i32 0, i64 2
+  store double %mul6.i.i, double* %agg.tmp913.sroa.5.0..sroa_idx203.i, align 8
+  %agg.tmp1014.sroa.4.0..sroa_idx256.i = getelementptr inbounds %class.D.2.7.12.17.332, %class.D.2.7.12.17.332* undef, i64 0, i32 1, i32 0, i64 1
+  store double %add.i.i, double* %agg.tmp1014.sroa.4.0..sroa_idx256.i, align 8
+  %agg.tmp1014.sroa.5.0..sroa_idx258.i = getelementptr inbounds %class.D.2.7.12.17.332, %class.D.2.7.12.17.332* undef, i64 0, i32 1, i32 0, i64 2
+  store double %add10.i.i, double* %agg.tmp1014.sroa.5.0..sroa_idx258.i, align 8
+  ret void
+}
+
+declare dso_local double @_ZN1F5m_fn9Ev() local_unnamed_addr #1
+
+declare dso_local double @_ZN1B5m_fn8Ev() local_unnamed_addr #1
+

test/Transforms/SLPVectorizer/X86/vect_copyable_in_binops.ll

entry:
   store i32 %add9, i32* %incdec.ptr7, align 4
   ret void
 }
 
 define void @add1(i32* noalias %dst, i32* noalias %src) {
 ; CHECK-LABEL: @add1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i32, i32* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[SRC]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i32, i32* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store i32 [[TMP0]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[ADD3:%.*]] = add nsw i32 [[TMP1]], 1
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
-; CHECK-NEXT:    store i32 [[ADD3]], i32* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR5:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[ADD6:%.*]] = add nsw i32 [[TMP2]], 2
 ; CHECK-NEXT:    [[INCDEC_PTR7:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
-; CHECK-NEXT:    store i32 [[ADD6]], i32* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[INCDEC_PTR5]], align 4
-; CHECK-NEXT:    [[ADD9:%.*]] = add nsw i32 [[TMP3]], 3
-; CHECK-NEXT:    store i32 [[ADD9]], i32* [[INCDEC_PTR7]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = add nsw <4 x i32> <i32 0, i32 1, i32 2, i32 3>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP2]], <4 x i32>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
   %0 = load i32, i32* %src, align 4
   %incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
   store i32 %0, i32* %dst, align 4
   %incdec.ptr2 = getelementptr inbounds i32, i32* %src, i64 2
entry:
   store i32 %add9, i32* %incdec.ptr7, align 4
   ret void
 }
 
 define void @sub0(i32* noalias %dst, i32* noalias %src) {
 ; CHECK-LABEL: @sub0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i32, i32* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[SRC]], align 4
-; CHECK-NEXT:    [[SUB:%.*]] = add nsw i32 [[TMP0]], -1
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i32, i32* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store i32 [[SUB]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[INCDEC_PTR]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR3:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
-; CHECK-NEXT:    store i32 [[TMP1]], i32* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[SUB5:%.*]] = add nsw i32 [[TMP2]], -2
 ; CHECK-NEXT:    [[INCDEC_PTR6:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
-; CHECK-NEXT:    store i32 [[SUB5]], i32* [[INCDEC_PTR3]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[SUB8:%.*]] = add nsw i32 [[TMP3]], -3
-; CHECK-NEXT:    store i32 [[SUB8]], i32* [[INCDEC_PTR6]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = add nsw <4 x i32> <i32 -1, i32 0, i32 -2, i32 -3>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP2]], <4 x i32>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
   %0 = load i32, i32* %src, align 4
   %sub = add nsw i32 %0, -1
   %incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
   store i32 %sub, i32* %dst, align 4
entry:
   store i32 %sub9, i32* %incdec.ptr7, align 4
   ret void
 }
 
 define void @addsub0(i32* noalias %dst, i32* noalias %src) {
 ; CHECK-LABEL: @addsub0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i32, i32* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[SRC]], align 4
-; CHECK-NEXT:    [[SUB:%.*]] = add nsw i32 [[TMP0]], -1
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i32, i32* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store i32 [[SUB]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[INCDEC_PTR]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR3:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
-; CHECK-NEXT:    store i32 [[TMP1]], i32* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[SUB5:%.*]] = add nsw i32 [[TMP2]], -2
 ; CHECK-NEXT:    [[INCDEC_PTR6:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
-; CHECK-NEXT:    store i32 [[SUB5]], i32* [[INCDEC_PTR3]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[SUB8:%.*]] = sub nsw i32 [[TMP3]], -3
-; CHECK-NEXT:    store i32 [[SUB8]], i32* [[INCDEC_PTR6]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 -1, i32 0, i32 -2, i32 -3>
+; CHECK-NEXT:    [[TMP3:%.*]] = sub nsw <4 x i32> [[TMP1]], <i32 -1, i32 0, i32 -2, i32 -3>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP3]], <4 x i32> <i32 0, i32 1, i32 2, i32 7>
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP4]], <4 x i32>* [[TMP5]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
   %0 = load i32, i32* %src, align 4
   %sub = add nsw i32 %0, -1
   %incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
   store i32 %sub, i32* %dst, align 4
entry:
   store i32 %sub8, i32* %incdec.ptr6, align 4
   ret void
 }
 
 define void @addsub1(i32* noalias %dst, i32* noalias %src) {
 ; CHECK-LABEL: @addsub1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i32, i32* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[SRC]], align 4
-; CHECK-NEXT:    [[SUB:%.*]] = add nsw i32 [[TMP0]], -1
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i32, i32* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store i32 [[SUB]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[SUB1:%.*]] = sub nsw i32 [[TMP1]], -1
 ; CHECK-NEXT:    [[INCDEC_PTR3:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
-; CHECK-NEXT:    store i32 [[SUB1]], i32* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[INCDEC_PTR2]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR6:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
-; CHECK-NEXT:    store i32 [[TMP2]], i32* [[INCDEC_PTR3]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[SUB8:%.*]] = sub nsw i32 [[TMP3]], -3
-; CHECK-NEXT:    store i32 [[SUB8]], i32* [[INCDEC_PTR6]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 -1, i32 -1, i32 0, i32 -3>
+; CHECK-NEXT:    [[TMP3:%.*]] = sub nsw <4 x i32> [[TMP1]], <i32 -1, i32 -1, i32 0, i32 -3>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP3]], <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP4]], <4 x i32>* [[TMP5]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
   %0 = load i32, i32* %src, align 4
   %sub = add nsw i32 %0, -1
   %incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
   store i32 %sub, i32* %dst, align 4
entry:
   store i32 %sub8, i32* %incdec.ptr6, align 4
   ret void
 }
 
 define void @mul(i32* noalias %dst, i32* noalias %src) {
 ; CHECK-LABEL: @mul(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i32, i32* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[SRC]], align 4
-; CHECK-NEXT:    [[MUL:%.*]] = mul nsw i32 [[TMP0]], 257
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i32, i32* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store i32 [[MUL]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[MUL3:%.*]] = mul nsw i32 [[TMP1]], -3
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
-; CHECK-NEXT:    store i32 [[MUL3]], i32* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR5:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[INCDEC_PTR2]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR7:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
-; CHECK-NEXT:    store i32 [[TMP2]], i32* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[INCDEC_PTR5]], align 4
-; CHECK-NEXT:    [[MUL9:%.*]] = mul nsw i32 [[TMP3]], -9
-; CHECK-NEXT:    store i32 [[MUL9]], i32* [[INCDEC_PTR7]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = mul nsw <4 x i32> <i32 257, i32 -3, i32 1, i32 -9>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP2]], <4 x i32>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
   %0 = load i32, i32* %src, align 4
   %mul = mul nsw i32 %0, 257
   %incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
   store i32 %mul, i32* %dst, align 4
entry:
   store i32 %mul9, i32* %incdec.ptr7, align 4
   ret void
 }
 
 define void @shl0(i32* noalias %dst, i32* noalias %src) {
 ; CHECK-LABEL: @shl0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds i32, i32* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[SRC]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds i32, i32* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store i32 [[TMP0]], i32* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[SHL:%.*]] = shl i32 [[TMP1]], 1
 ; CHECK-NEXT:    [[INCDEC_PTR3:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 2
-; CHECK-NEXT:    store i32 [[SHL]], i32* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds i32, i32* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[SHL5:%.*]] = shl i32 [[TMP2]], 2
 ; CHECK-NEXT:    [[INCDEC_PTR6:%.*]] = getelementptr inbounds i32, i32* [[DST]], i64 3
-; CHECK-NEXT:    store i32 [[SHL5]], i32* [[INCDEC_PTR3]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load i32, i32* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[SHL8:%.*]] = shl i32 [[TMP3]], 3
-; CHECK-NEXT:    store i32 [[SHL8]], i32* [[INCDEC_PTR6]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[SRC]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = shl <4 x i32> [[TMP1]], <i32 0, i32 1, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i32* [[DST]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP2]], <4 x i32>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds i32, i32* %src, i64 1
   %0 = load i32, i32* %src, align 4
   %incdec.ptr1 = getelementptr inbounds i32, i32* %dst, i64 1
   store i32 %0, i32* %dst, align 4
   %incdec.ptr2 = getelementptr inbounds i32, i32* %src, i64 2
entry:
   store float %add9, float* %incdec.ptr7, align 4
   ret void
 }
 
 define void @add1f(float* noalias %dst, float* noalias %src) {
 ; CHECK-LABEL: @add1f(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds float, float* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* [[SRC]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds float, float* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store float [[TMP0]], float* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[ADD3:%.*]] = fadd fast float [[TMP1]], 1.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds float, float* [[DST]], i64 2
-; CHECK-NEXT:    store float [[ADD3]], float* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR5:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[ADD6:%.*]] = fadd fast float [[TMP2]], 2.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR7:%.*]] = getelementptr inbounds float, float* [[DST]], i64 3
-; CHECK-NEXT:    store float [[ADD6]], float* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[INCDEC_PTR5]], align 4
-; CHECK-NEXT:    [[ADD9:%.*]] = fadd fast float [[TMP3]], 3.000000e+00
-; CHECK-NEXT:    store float [[ADD9]], float* [[INCDEC_PTR7]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast float* [[SRC]] to <4 x float>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, <4 x float>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd fast <4 x float> <float -0.000000e+00, float 1.000000e+00, float 2.000000e+00, float 3.000000e+00>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast float* [[DST]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP2]], <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds float, float* %src, i64 1
   %0 = load float, float* %src, align 4
   %incdec.ptr1 = getelementptr inbounds float, float* %dst, i64 1
   store float %0, float* %dst, align 4
   %incdec.ptr2 = getelementptr inbounds float, float* %src, i64 2
entry:
   store float %add9, float* %incdec.ptr7, align 4
   ret void
 }
 
 define void @sub0f(float* noalias %dst, float* noalias %src) {
 ; CHECK-LABEL: @sub0f(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds float, float* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* [[SRC]], align 4
-; CHECK-NEXT:    [[ADD:%.*]] = fadd fast float [[TMP0]], -1.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds float, float* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store float [[ADD]], float* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[INCDEC_PTR]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds float, float* [[DST]], i64 2
-; CHECK-NEXT:    store float [[TMP1]], float* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR5:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[ADD6:%.*]] = fadd fast float [[TMP2]], -2.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR7:%.*]] = getelementptr inbounds float, float* [[DST]], i64 3
-; CHECK-NEXT:    store float [[ADD6]], float* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[INCDEC_PTR5]], align 4
-; CHECK-NEXT:    [[ADD9:%.*]] = fadd fast float [[TMP3]], -3.000000e+00
-; CHECK-NEXT:    store float [[ADD9]], float* [[INCDEC_PTR7]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast float* [[SRC]] to <4 x float>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, <4 x float>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd fast <4 x float> <float -1.000000e+00, float -0.000000e+00, float -2.000000e+00, float -3.000000e+00>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast float* [[DST]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP2]], <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds float, float* %src, i64 1
   %0 = load float, float* %src, align 4
   %add = fadd fast float %0, -1.000000e+00
   %incdec.ptr1 = getelementptr inbounds float, float* %dst, i64 1
   store float %add, float* %dst, align 4
entry:
   store float %sub9, float* %incdec.ptr7, align 4
   ret void
 }
 
 define void @addsub0f(float* noalias %dst, float* noalias %src) {
 ; CHECK-LABEL: @addsub0f(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds float, float* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* [[SRC]], align 4
-; CHECK-NEXT:    [[SUB:%.*]] = fadd fast float [[TMP0]], -1.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds float, float* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store float [[SUB]], float* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[INCDEC_PTR]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR3:%.*]] = getelementptr inbounds float, float* [[DST]], i64 2
-; CHECK-NEXT:    store float [[TMP1]], float* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[SUB5:%.*]] = fadd fast float [[TMP2]], -2.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR6:%.*]] = getelementptr inbounds float, float* [[DST]], i64 3
-; CHECK-NEXT:    store float [[SUB5]], float* [[INCDEC_PTR3]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[SUB8:%.*]] = fsub fast float [[TMP3]], -3.000000e+00
-; CHECK-NEXT:    store float [[SUB8]], float* [[INCDEC_PTR6]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast float* [[SRC]] to <4 x float>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, <4 x float>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd fast <4 x float> [[TMP1]], <float -1.000000e+00, float -0.000000e+00, float -2.000000e+00, float -3.000000e+00>
+; CHECK-NEXT:    [[TMP3:%.*]] = fsub fast <4 x float> [[TMP1]], <float -1.000000e+00, float -0.000000e+00, float -2.000000e+00, float -3.000000e+00>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> [[TMP3]], <4 x i32> <i32 0, i32 1, i32 2, i32 7>
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast float* [[DST]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP4]], <4 x float>* [[TMP5]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds float, float* %src, i64 1
   %0 = load float, float* %src, align 4
   %sub = fadd fast float %0, -1.000000e+00
   %incdec.ptr1 = getelementptr inbounds float, float* %dst, i64 1
   store float %sub, float* %dst, align 4
entry:
   store float %sub8, float* %incdec.ptr6, align 4
   ret void
 }
 
 define void @addsub1f(float* noalias %dst, float* noalias %src) {
 ; CHECK-LABEL: @addsub1f(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds float, float* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* [[SRC]], align 4
-; CHECK-NEXT:    [[SUB:%.*]] = fadd fast float [[TMP0]], -1.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds float, float* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store float [[SUB]], float* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[SUB1:%.*]] = fsub fast float [[TMP1]], -1.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR3:%.*]] = getelementptr inbounds float, float* [[DST]], i64 2
-; CHECK-NEXT:    store float [[SUB1]], float* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[INCDEC_PTR2]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR6:%.*]] = getelementptr inbounds float, float* [[DST]], i64 3
-; CHECK-NEXT:    store float [[TMP2]], float* [[INCDEC_PTR3]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[SUB8:%.*]] = fsub fast float [[TMP3]], -3.000000e+00
-; CHECK-NEXT:    store float [[SUB8]], float* [[INCDEC_PTR6]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast float* [[SRC]] to <4 x float>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, <4 x float>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd fast <4 x float> [[TMP1]], <float -1.000000e+00, float -1.000000e+00, float -0.000000e+00, float -3.000000e+00>
+; CHECK-NEXT:    [[TMP3:%.*]] = fsub fast <4 x float> [[TMP1]], <float -1.000000e+00, float -1.000000e+00, float -0.000000e+00, float -3.000000e+00>
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> [[TMP3]], <4 x i32> <i32 0, i32 5, i32 2, i32 7>
+; CHECK-NEXT:    [[TMP5:%.*]] = bitcast float* [[DST]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP4]], <4 x float>* [[TMP5]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds float, float* %src, i64 1
   %0 = load float, float* %src, align 4
   %sub = fadd fast float %0, -1.000000e+00
   %incdec.ptr1 = getelementptr inbounds float, float* %dst, i64 1
   store float %sub, float* %dst, align 4
entry:
   store float %sub8, float* %incdec.ptr6, align 4
   ret void
 }
 
 define void @mulf(float* noalias %dst, float* noalias %src) {
 ; CHECK-LABEL: @mulf(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds float, float* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* [[SRC]], align 4
-; CHECK-NEXT:    [[SUB:%.*]] = fmul fast float [[TMP0]], 2.570000e+02
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds float, float* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store float [[SUB]], float* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[INCDEC_PTR]], align 4
-; CHECK-NEXT:    [[SUB3:%.*]] = fmul fast float [[TMP1]], -3.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds float, float* [[DST]], i64 2
-; CHECK-NEXT:    store float [[SUB3]], float* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR5:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[INCDEC_PTR2]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR7:%.*]] = getelementptr inbounds float, float* [[DST]], i64 3
-; CHECK-NEXT:    store float [[TMP2]], float* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[INCDEC_PTR5]], align 4
-; CHECK-NEXT:    [[SUB9:%.*]] = fmul fast float [[TMP3]], -9.000000e+00
-; CHECK-NEXT:    store float [[SUB9]], float* [[INCDEC_PTR7]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast float* [[SRC]] to <4 x float>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, <4 x float>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fmul fast <4 x float> <float 2.570000e+02, float -3.000000e+00, float 1.000000e+00, float -9.000000e+00>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast float* [[DST]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP2]], <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds float, float* %src, i64 1
   %0 = load float, float* %src, align 4
   %sub = fmul fast float %0, 2.570000e+02
   %incdec.ptr1 = getelementptr inbounds float, float* %dst, i64 1
   store float %sub, float* %dst, align 4
entry:
   store float %add9, float* %incdec.ptr7, align 4
   ret void
 }
 
 define void @sub0fn(float* noalias %dst, float* noalias %src) {
 ; CHECK-LABEL: @sub0fn(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[INCDEC_PTR:%.*]] = getelementptr inbounds float, float* [[SRC:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = load float, float* [[SRC]], align 4
-; CHECK-NEXT:    [[ADD:%.*]] = fadd fast float [[TMP0]], -1.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR1:%.*]] = getelementptr inbounds float, float* [[DST:%.*]], i64 1
-; CHECK-NEXT:    store float [[ADD]], float* [[DST]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR2:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 2
-; CHECK-NEXT:    [[TMP1:%.*]] = load float, float* [[INCDEC_PTR]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR4:%.*]] = getelementptr inbounds float, float* [[DST]], i64 2
-; CHECK-NEXT:    store float [[TMP1]], float* [[INCDEC_PTR1]], align 4
 ; CHECK-NEXT:    [[INCDEC_PTR5:%.*]] = getelementptr inbounds float, float* [[SRC]], i64 3
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, float* [[INCDEC_PTR2]], align 4
-; CHECK-NEXT:    [[ADD6:%.*]] = fadd float [[TMP2]], -2.000000e+00
 ; CHECK-NEXT:    [[INCDEC_PTR7:%.*]] = getelementptr inbounds float, float* [[DST]], i64 3
-; CHECK-NEXT:    store float [[ADD6]], float* [[INCDEC_PTR4]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = load float, float* [[INCDEC_PTR5]], align 4
-; CHECK-NEXT:    [[ADD9:%.*]] = fadd float [[TMP3]], -3.000000e+00
-; CHECK-NEXT:    store float [[ADD9]], float* [[INCDEC_PTR7]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast float* [[SRC]] to <4 x float>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x float>, <4 x float>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = fadd <4 x float> <float -1.000000e+00, float -0.000000e+00, float -2.000000e+00, float -3.000000e+00>, [[TMP1]]
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast float* [[DST]] to <4 x float>*
+; CHECK-NEXT:    store <4 x float> [[TMP2]], <4 x float>* [[TMP3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %incdec.ptr = getelementptr inbounds float, float* %src, i64 1
   %0 = load float, float* %src, align 4
   %add = fadd fast float %0, -1.000000e+00
   %incdec.ptr1 = getelementptr inbounds float, float* %dst, i64 1
   store float %add, float* %dst, align 4