[X86] Add pass to infer required-vector-width attribute based on size of function arguments and use of intrinsics

[craig.topper]

Summary

This patch adds a pass that determines if 512 bit registers are definitely needed based on the size of function arguments it detects. It explicitly ignores target independent intrinsics assuming that selection dag can gracefully split those.

I'm hoping what's here is enough to protect ABI, inline assembly, and intrinsic requirements. Anything else we need for correctness that I haven't thought of?

This doesn't help with any explicit vector code created in the original source code that get lowered to native IR without using in target specific intrinsics. We'll need to set the attribute from clang or an earlier IR pass to detect that. This pass is intended to get something testable so we can see if we can enable prefer vector width and get the legalizer to clip to 256 bit safely on real source code. Nothing will happen if -mprefer-vector-width=256 is not passed to clang.

[echristo]

I'm really not a fan in general of this path - can we step back and figure out what we'd like to do with this first? I don't really agree with the need for require-vector-width in the first place.

[craig.topper]

@echristo, I know we talked some. I know you don't like the attribute, are you also against the legalizer limit?

[craig.topper]

Fix the name of the attribute to be 'required-vector-width' to match what's already in X86TargetMachine.cpp

[lebedev.ri]

lib/Target/X86/X86.h

123 ↗

(On Diff #135322)

require*d*-vector-width

[craig.topper]

Fix the attribute name in one more place

[craig.topper]

Attempting to clarify some things.

Prior to any of my prefer vector width changes, the TTI interface informed the loop vectorizer about the maximum register width in hardware via the getRegisterBitWidth method. The loop vectorizer uses this information as a guideline, but not a strict limit of what types it can create. My understanding is that there are two modes, the default mode finds the largest scalar width of loads, stores, and phis within the loop. It then determines how many of this scalar type fit in the register width received from TTI. This determines the VF factor. The second mode, called maximize bandwidth, uses the smallest scalar width instead. I'm not sure if that's still restricted to loads, store, and phis or not.

So for both modes, if there are any larger scalar types in the loop, those types will be multiplied by the VF factor and cause the vectorizer to create types that are larger than the maximum hardware register size. These types will exist all the way to codegen, and it is up to the SelectionDAG type legalization process or target specific DAG combines to split these larger operations. In some cases the larger types are used to represent idioms that contain extends and truncates that we were able to combine to more complex instructions without doing a split. X86 probably has more pre type legalization target specific DAG combines here than most other targets. And we are doing our own splitting for type legalization in some of these combines. This prevents the type legalizer from needing to deal with target specific nodes. Though the type legalizer is capable of calling ReplaceNodeResults and LowerOperation for target specific nodes.

In addition to the vectorizer, I've believe I've seen evidence of InstCombine creating larger types when it canonicalizes getelementptrs to have pointer width types by inserting sign extends. This shows up in vector GEPs used by gather and scatter intrinsics. So if the GEP used a v16i32 index, InstCombine will turn it into v16i64. X86 tries to remove some of these extends with a DAG combine, but we're not great at finding a uniform base address for gathers which causes the extend to be hidden. So we end up spltting gathers when their result type is the maximum hardware register width, but their index is twice as large.

Based on this current behavior of IR passes with respect to hardware vector width and the fact that codegen already has to handle illegal types. So the first part of my changes(prefer-vector-width), make the vectorizer behavior similar between prefer-vector-width=256 on an avx512f target and an avx2 target by making getRegisterBitWidth return 256 in both cases. So under prefer-vector-width=256, the vectorizer will calculate the same VF factor it would for an AVX2 target, and it will still generate wider types for some things just like it does for AVX2. With just prefer-vector-width, when the IR gets to codegen we end up emitting 512 bit operations for these wider types because the type legalization process can't enforce prefer-vector-width. This is because codegen has no knowledge of the source of the vectors in the IR. They may have come from the vectorizer or they may have been present in the original source code. If they were in the original source code, they might be used by target specific intrinsics, or function parameters, or return types. In those cases we would need to ensure we emit 512 bit vectors or the intrinsics won't assemble or we could have mismatched ABI between two different translation units if one is compiled with prefer-vector-width and one isn't.

The patch presented here overcomes this limitation by providing a required-vector-width attribute that can be used to tell codegen that there is nothing in the IR that strictly requires wider vectors to avoid compilation failures or ABI mismatches. With this we can tell the type legalization process and the X86 DAG combines that only 256 bit vectors are legal. With this the type legalization process should be carried out in a very similar way to AVX2, splitting the wider types from the vectorizer in the same way. But we also gain the AVX512VL specific instructions like scatter and masked operations that aren't present in AVX2.

This patch isn't a perfect solution because it doesn't capture the source containing vectors that don't go through intrinsics or function arguments or return values. But we could emit the same function attribute earlier, in the frontend, to convey that information.

So based on my conversations with others there seem to be some questions about the baseline assumptions here.

-Should the vectorizer be creating wider vectors than what getRegisterBitWidth returns? Changing this likely means we would need to introduce splitting into the vectorizer to avoid hurting SSE/AVX/AVX2 vectorized code. But I don't think that removes the need for the backend to handle illegal types in IR so that would just be introducing another type legalization process.
-Some idioms require larger types to represent in IR, for example, average uses (trunc (add (add (zext X, zext Y), 1)) but we are able to do that in X86 with a single instruction. But the 512-bit version of that instruction requires the zexts be wider than 512 bits in IR.
-Should InstCombine be creating wide illegal vector types to canonicalize GEPs? Are there other places that do similar things? Other passes?
-Is X86 too aggressive about creating target specific nodes early requiring many places to do custom type legalization? The checks are getting centralized with SplitBinaryOpsAndApply so hopefully we won't have to think about all the different split sizes in multiple places going forward.

What I'm looking for is a way to guarantee that with prefer-vector-width=256, starting from scalar code and vectorizing it that we won't end up with 512 registers in the output. Even a few of them are enough to trigger than frequency reduction penalty I am trying to avoid. Given the current behavior of IR passes and their expectations of what codegen can handle, I don't see a way to make this guarantee from IR. And it would be easy to break in the future as new passes or passes are modified. The codegen type legalization process seems the best place to handle this. But we need to be able to communicate when it is safe to do so.

Would this patch be more acceptable in the near term if I do one of the following:
-Get ride of the required-vector-width attribute and have this pass force a prefer-vector-width attribute to be at least as large as any ABI or intrinsic constraints? Then make the X86 codegen use prefer-vector-width to control the type legalizer. This would mean some places in x86 codegen would create new 512-bit operations that weren't in the IR since we'd be overriding the preference.
-Rename the required-vector-width attribute to x86-required-vector-width so that it is clear it is x86 specific for now. And we could only given a generic name if we decide to emit it earlier than x86 codegen.

Either of these would allow people to start experimenting with prefer-vector-width=256. We would need to revisit and make changes before we could make prefer-vector-width=256 the default for SKX.

[craig.topper]

Ping

[echristo]

So for both modes, if there are any larger scalar types in the loop, those types will be multiplied by the VF factor and cause the vectorizer to create types that are larger than the maximum hardware register size. These types will exist all the way to codegen, and it is up to the SelectionDAG type legalization process or target specific DAG combines to split these larger operations. In some cases the larger types are used to represent idioms that contain extends and truncates that we were able to combine to more complex instructions without doing a split. X86 probably has more pre type legalization target specific DAG combines here than most other targets. And we are doing our own splitting for type legalization in some of these combines. This prevents the type legalizer from needing to deal with target specific nodes. Though the type legalizer is capable of calling ReplaceNodeResults and LowerOperation for target specific nodes.

This is interesting, and perhaps not what we should do.

In addition to the vectorizer, I've believe I've seen evidence of InstCombine creating larger types when it canonicalizes getelementptrs to have pointer width types by inserting sign extends. This shows up in vector GEPs used by gather and scatter intrinsics. So if the GEP used a v16i32 index, InstCombine will turn it into v16i64. X86 tries to remove some of these extends with a DAG combine, but we're not great at finding a uniform base address for gathers which causes the extend to be hidden. So we end up spltting gathers when their result type is the maximum hardware register width, but their index is twice as large.

This is perhaps also not what we should do with vector accessed operations.

Based on this current behavior of IR passes with respect to hardware vector width and the fact that codegen already has to handle illegal types. So the first part of my changes(prefer-vector-width), make the vectorizer behavior similar between prefer-vector-width=256 on an avx512f target and an avx2 target by making getRegisterBitWidth return 256 in both cases. So under prefer-vector-width=256, the vectorizer will calculate the same VF factor it would for an AVX2

The patch presented here overcomes this limitation by providing a required-vector-width attribute that can be used to tell codegen that there is nothing in the IR that strictly requires wider vectors to avoid compilation failures or ABI mismatches. With this we can tell the type legalization process and the X86 DAG combines that only 256 bit vectors are legal. With this the type legalization process should be carried out in a very similar way to AVX2, splitting the wider types from the vectorizer in the same way. But we also gain the AVX512VL specific instructions like scatter and masked operations that aren't present in AVX2.

I don't think this part is what we should do. We shouldn't change the size of a register, but rather change the optimizers to prefer smaller things. Legal versus preferred is a big difference.

What I'm looking for is a way to guarantee that with prefer-vector-width=256, starting from scalar code and vectorizing it that we won't end up with 512 registers in the output. Even a few of them are enough to trigger than frequency reduction penalty I am trying to avoid. Given the current behavior of IR passes and their expectations of what codegen can handle, I don't see a way to make this guarantee from IR. And it would be easy to break in the future as new passes or passes are modified. The codegen type legalization process seems the best place to handle this. But we need to be able to communicate when it is safe to do so.

I think it might be a lot of work, but possible. I think that we might abuse the legalizer here too much. For example, let's say I want to limit the vector size to 256, but in some specific code I use the intrinsics or other vector code to explicitly ask for 512-bit vectors. I should still be able to get that I think? Right now you'll say that the code isn't legal and try to lower it.

-eric

[chandlerc]

In D43306#1049842, @echristo wrote:

So for both modes, if there are any larger scalar types in the loop, those types will be multiplied by the VF factor and cause the vectorizer to create types that are larger than the maximum hardware register size. These types will exist all the way to codegen, and it is up to the SelectionDAG type legalization process or target specific DAG combines to split these larger operations. In some cases the larger types are used to represent idioms that contain extends and truncates that we were able to combine to more complex instructions without doing a split. X86 probably has more pre type legalization target specific DAG combines here than most other targets. And we are doing our own splitting for type legalization in some of these combines. This prevents the type legalizer from needing to deal with target specific nodes. Though the type legalizer is capable of calling ReplaceNodeResults and LowerOperation for target specific nodes.

This is interesting, and perhaps not what we should do.

FWIW, I really like the ability of passes to create larger-than-legal types and rely on common type legalization to split and fit them into useful vector registers. This proves very useful. That's only one part of what this paragraph talks about though, ond the rest I don't have a strong opinion about.

In addition to the vectorizer, I've believe I've seen evidence of InstCombine creating larger types when it canonicalizes getelementptrs to have pointer width types by inserting sign extends. This shows up in vector GEPs used by gather and scatter intrinsics. So if the GEP used a v16i32 index, InstCombine will turn it into v16i64. X86 tries to remove some of these extends with a DAG combine, but we're not great at finding a uniform base address for gathers which causes the extend to be hidden. So we end up spltting gathers when their result type is the maximum hardware register width, but their index is twice as large.

This is perhaps also not what we should do with vector accessed operations.

This does seem much more questionable. IMO, not because of the illegal vector types but because extending the scalar component seems much more troubling in a vector-gep context than a scalar-gep context, regardless of the arity of the vector.

Based on this current behavior of IR passes with respect to hardware vector width and the fact that codegen already has to handle illegal types. So the first part of my changes(prefer-vector-width), make the vectorizer behavior similar between prefer-vector-width=256 on an avx512f target and an avx2 target by making getRegisterBitWidth return 256 in both cases. So under prefer-vector-width=256, the vectorizer will calculate the same VF factor it would for an AVX2

The patch presented here overcomes this limitation by providing a required-vector-width attribute that can be used to tell codegen that there is nothing in the IR that strictly requires wider vectors to avoid compilation failures or ABI mismatches. With this we can tell the type legalization process and the X86 DAG combines that only 256 bit vectors are legal. With this the type legalization process should be carried out in a very similar way to AVX2, splitting the wider types from the vectorizer in the same way. But we also gain the AVX512VL specific instructions like scatter and masked operations that aren't present in AVX2.

I don't think this part is what we should do. We shouldn't change the size of a register, but rather change the optimizers to prefer smaller things. Legal versus preferred is a big difference.

I disagree here... I think type legalization is a necessary tool to meaningfully achieve a preferred type. I think we would end up building a mini legalizer elsewhere in the optimizer if we remove the ability of the current type legalizer logic to handle this, and that doesn't seem ideal. I also think there is a better way to split this hair....

What I'm looking for is a way to guarantee that with prefer-vector-width=256, starting from scalar code and vectorizing it that we won't end up with 512 registers in the output. Even a few of them are enough to trigger than frequency reduction penalty I am trying to avoid. Given the current behavior of IR passes and their expectations of what codegen can handle, I don't see a way to make this guarantee from IR. And it would be easy to break in the future as new passes or passes are modified. The codegen type legalization process seems the best place to handle this. But we need to be able to communicate when it is safe to do so.

I think this core requirement gets at the solution I would propose to this.

I think at the IR level we should simply replace the preferred width with required width. We should require the frontend to pick the width that it wants code generated for, and stick to it.

Then we can demand that the frontend, if it wants to explicitly generate 512-bit code through things like intrinsics, *increases* the required width for that function. This allows explicit AVX-512 _mm_* intrinsics to work correctly.

Then we teach the inliner how to cope with this intelligently. My guess is that this won't be trivial to do because of patterns like:

if (DynamicTestIfTargetHasAVX512()) {
  CallFunctionWithAVX512Instrinsics();
} else {
  CallFunctionWithLoopThatShouldBeAutoVectorizedTo256BitVectors();
}

Here, both naive solutions may prove unfortunate: increasing the callers required width in the IR to match the callees and allowing the inline might in theory regress the generated code for the other function, but blocking the inlining would almost certainly regress the code.

However, I think in practice inlining will be the correct call for the vast majority of cases. Most of the really interesting cases won't be inline candidates anyways due to different *subtargets* that already influences inlining decisions. So generally, I think it is safe to let the inliner take the max of the required widths and proceed to inline. If we find interesting cases that require more complex logic, we can address them once we have that test case.

One possible alternative is that we could make an explicit vs. implicit distinction and allow inlining to increase an implicit width but block inlining that would increase an explicitly narrow width. This might make it easier to write code like:

if (NumberOfItemsIsVeryLarge) {
  CallFunctionWithWideVectorsButLowerFreq();
} else {
  CallFunctionWithNarrowVectorsAndHigherFreq();
}

The critical result here is that if the user code *already contains* explicitly 512-bit width vector code, it seems reasonable for the vectorizer to alse produce 512-bit width vector code. So I don't think the IR needs to model a preference and a requirement independently in the IR for the purposes of the code generator. At most, it might want this for the purpose of controlling inlining and merging.

[chandlerc]

One thing I should clarify here:

What I'm imagining is reasonable to be per-subtarget and inform the legalizer are the *set of legal vector types*. This is subtly different from the *size of legal registers*. Does that still make sense?

[echristo]

In D43306#1050892, @chandlerc wrote:

One thing I should clarify here:

What I'm imagining is reasonable to be per-subtarget and inform the legalizer are the *set of legal vector types*. This is subtly different from the *size of legal registers*. Does that still make sense?

This split-hair makes sense to me and works for what I'm imagining we do.

[craig.topper]

Remove the required-vector-width attribute related code. Change legalization to be based on prefer-vector-width.

Add a pre-codegen IR pass to force the preferred vector width for any constructs that we know would fail isel if the prefer vector width is set to something smaller than required. The goal here is to protect correctness while I work on adding the attribute to clang's frontend. It will also protect from bad IR. I've also used it to experiment with running all x86 lit tests with prefer-vector-width forced to 256 to make sure nothing crashes or causes an isel failure.

I do wonder if we should have a some kind of disable flag for the legalization change independent of the preference. Is it useful to be able to test constraining TTI to a prefer vector width without constraining the legalizer? It could be useful if the IR pass added here doesn't catch everything that affects correctness.

Next steps:
-Make sure the TTI cost model understands whether 512 bits are legal based on the preference at the time the vectorizer runs. In particular it looks like X86TTIImpl::getCastInstrCost might not understand this correctly as most of its lookups aren't done with the result of getTypeLegalizationCost.
-Teach the inliner to merge the attribute.

[chandlerc]

Given the direction this is going, I'd suggest maybe renaming the attribute. I'm thinking something like legal_vetor_width or otherwise framing this as fundamentally a legalization constraint rather than anything more fundamental. I would also really suggest doing more than just one width shift in this patch because that will help ensure that the mechanism is working in the general way as intended.

I have a bunch of other nit-picky comments but can save those until this is more baked and you're looking to polish.

One last somewhat longer-term question is: have you thought about making this more targeted? Specifically, rather than making it a *type* legalization parameter, how crazy would it be to make this only apply to specific operations? I'm interested in potentially only preferring narrower vector widths for non-mov instructions so that loading and storing wider types can continue to occur. Thoughts?

lib/Target/X86/X86VectorWidthInfer.cpp

58–62 ↗

(On Diff #140358)

I think if you just delete this code, everything below already works for arbitrary vector widths?

116–117 ↗

(On Diff #140358)

You should be able to bound this (in both directions) by looking at the subtarget's register sizes. Which would somewhat illustrate that this is just parameterizing *legalization*, not the underlying physical machine description.

[craig.topper]

-Rename required-vector-width to legal-vector-width.
-Add a command line option to use prefer-vector-width for legal-vector-width if legal-vector-width is not already provided
-Simplify test to mostly just use the command line option so we don't have to repeat the test bodies with different attributes.

The intention here is to work towards removing prefer-vector-width in favor of legal-vector-width. This will require clang changes to set the legal-vector-width attribute. The command line option added here enables experimenting with the backend support while the clang changes are being developed.

lib/Target/X86/X86Subtarget.h

   /// features.
   unsigned PreferVectorWidth;
 
-  /// Required vector width from function attribute.
-  unsigned RequiredVectorWidth;
+  /// Legal vector width from function attribute.
+  unsigned LegalVectorWidthOverride;
+
+  /// Legal vector width from function attribute.
+  unsigned LegalVectorWidth;
 
   /// True if compiling for 64-bit, false for 16-bit or 32-bit.
   bool In64BitMode;

   X86Subtarget(const Triple &TT, StringRef CPU, StringRef FS,
                const X86TargetMachine &TM, unsigned StackAlignOverride,
                unsigned PreferVectorWidthOverride,
-               unsigned RequiredVectorWidth);
+               unsigned LegalVectorWidthOverride);
 
   const X86TargetLowering *getTargetLowering() const override {
     return &TLInfo;

   bool useRetpolineExternalThunk() const { return UseRetpolineExternalThunk; }
 
   unsigned getPreferVectorWidth() const { return PreferVectorWidth; }
-  unsigned getRequiredVectorWidth() const { return RequiredVectorWidth; }
 
   // Helper functions to determine when we should allow widening to 512-bit
   // during codegen.

   // If there are no 512-bit vectors and we prefer not to use 512-bit registers,
   // disable them in the legalizer.
   bool useAVX512Regs() const {
-    return hasAVX512() && (canExtendTo512DQ() || RequiredVectorWidth > 256);
+    return hasAVX512() && (canExtendTo512DQ() || LegalVectorWidth > 256);
   }
 
   bool useBWIRegs() const {

lib/Target/X86/X86Subtarget.cpp

 X86EarlyIfConv("x86-early-ifcvt", cl::Hidden,
                cl::desc("Enable early if-conversion on X86"));
 
+static cl::opt<bool>
+LegalizeToPreferVectorWidth("x86-experimental-legalize-to-prefer-vector-width",
+                            cl::desc("Legalize to the preferred vector width"),
+                            cl::init(false), cl::Hidden);
+
 
 /// Classify a blockaddress reference for the current subtarget according to how
 /// we should reference it in a non-pcrel context.

     PreferVectorWidth = PreferVectorWidthOverride;
   else if (Prefer256Bit)
     PreferVectorWidth = 256;
+
+  if (LegalVectorWidthOverride)
+    LegalVectorWidth = LegalVectorWidthOverride;
+  else if (LegalizeToPreferVectorWidth)
+    LegalVectorWidth = PreferVectorWidth;
 }
 
 void X86Subtarget::initializeEnvironment() {

   GatherOverhead = 1024;
   ScatterOverhead = 1024;
   PreferVectorWidth = UINT32_MAX;
+  LegalVectorWidth = UINT32_MAX;
   Prefer256Bit = false;
 }
 

                            const X86TargetMachine &TM,
                            unsigned StackAlignOverride,
                            unsigned PreferVectorWidthOverride,
-                           unsigned RequiredVectorWidth)
+                           unsigned LegalVectorWidthOverride)
     : X86GenSubtargetInfo(TT, CPU, FS), X86ProcFamily(Others),
       PICStyle(PICStyles::None), TM(TM), TargetTriple(TT),
       StackAlignOverride(StackAlignOverride),
       PreferVectorWidthOverride(PreferVectorWidthOverride),
-      RequiredVectorWidth(RequiredVectorWidth),
+      LegalVectorWidthOverride(LegalVectorWidthOverride),
       In64BitMode(TargetTriple.getArch() == Triple::x86_64),
       In32BitMode(TargetTriple.getArch() == Triple::x86 &&
                   TargetTriple.getEnvironment() != Triple::CODE16),

lib/Target/X86/X86TargetMachine.cpp

     }
   }
 
-  // Extract required-vector-width attribute.
-  unsigned RequiredVectorWidth = UINT32_MAX;
-  if (F.hasFnAttribute("required-vector-width")) {
-    StringRef Val = F.getFnAttribute("required-vector-width").getValueAsString();
+  // Extract legal-vector-width attribute.
+  unsigned LegalVectorWidthOverride = 0;
+  if (F.hasFnAttribute("legal-vector-width")) {
+    StringRef Val = F.getFnAttribute("legal-vector-width").getValueAsString();
     unsigned Width;
     if (!Val.getAsInteger(0, Width)) {
-      Key += ",required-vector-width=";
+      Key += ",legal-vector-width=";
       Key += Val;
-      RequiredVectorWidth = Width;
+      LegalVectorWidthOverride = Width;
     }
   }
 

     I = llvm::make_unique<X86Subtarget>(TargetTriple, CPU, FS, *this,
                                         Options.StackAlignmentOverride,
                                         PreferVectorWidthOverride,
-                                        RequiredVectorWidth);
+                                        LegalVectorWidthOverride);
   }
   return I.get();
 }

test/CodeGen/X86/legal-vector-width.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512vl,avx512bw,avx512dq,prefer-256-bit -x86-experimental-legalize-to-prefer-vector-width=true | FileCheck %s --check-prefix=CHECK --check-prefix=LEGAL256
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512vl,avx512bw,avx512dq,prefer-256-bit | FileCheck %s --check-prefix=CHECK --check-prefix=LEGAL512
+
+; This file primarily contains tests for specific places in X86ISelLowering.cpp that needed be made aware of the legalizer not allowing 512-bit vectors due to prefer-256-bit even though AVX512 is enabled.
+
+; The first two tests make sure the legal-vector-width attribute works. The remaining tests use a command lines to force different legalization configurations.
+
+define void @add256(<16 x i32>* %a, <16 x i32>* %b, <16 x i32>* %c) "legal-vector-width"="256" {
+; CHECK-LABEL: add256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovdqa (%rdi), %ymm0
+; CHECK-NEXT:    vmovdqa 32(%rdi), %ymm1
+; CHECK-NEXT:    vpaddd (%rsi), %ymm0, %ymm0
+; CHECK-NEXT:    vpaddd 32(%rsi), %ymm1, %ymm1
+; CHECK-NEXT:    vmovdqa %ymm1, 32(%rdx)
+; CHECK-NEXT:    vmovdqa %ymm0, (%rdx)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %d = load <16 x i32>, <16 x i32>* %a
+  %e = load <16 x i32>, <16 x i32>* %b
+  %f = add <16 x i32> %d, %e
+  store <16 x i32> %f, <16 x i32>* %c
+  ret void
+}
+
+define void @add512(<16 x i32>* %a, <16 x i32>* %b, <16 x i32>* %c) "legal-vector-width"="512" {
+; CHECK-LABEL: add512:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovdqa64 (%rdi), %zmm0
+; CHECK-NEXT:    vpaddd (%rsi), %zmm0, %zmm0
+; CHECK-NEXT:    vmovdqa64 %zmm0, (%rdx)
+; CHECK-NEXT:    vzeroupper
+; CHECK-NEXT:    retq
+  %d = load <16 x i32>, <16 x i32>* %a
+  %e = load <16 x i32>, <16 x i32>* %b
+  %f = add <16 x i32> %d, %e
+  store <16 x i32> %f, <16 x i32>* %c
+  ret void
+}
+
+define void @avg_v64i8(<64 x i8>* %a, <64 x i8>* %b) {
+; LEGAL256-LABEL: avg_v64i8:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vmovdqa (%rsi), %ymm0
+; LEGAL256-NEXT:    vmovdqa 32(%rsi), %ymm1
+; LEGAL256-NEXT:    vpavgb (%rdi), %ymm0, %ymm0
+; LEGAL256-NEXT:    vpavgb 32(%rdi), %ymm1, %ymm1
+; LEGAL256-NEXT:    vmovdqu %ymm1, (%rax)
+; LEGAL256-NEXT:    vmovdqu %ymm0, (%rax)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: avg_v64i8:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vmovdqa64 (%rsi), %zmm0
+; LEGAL512-NEXT:    vpavgb (%rdi), %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovdqu64 %zmm0, (%rax)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+  %1 = load <64 x i8>, <64 x i8>* %a
+  %2 = load <64 x i8>, <64 x i8>* %b
+  %3 = zext <64 x i8> %1 to <64 x i32>
+  %4 = zext <64 x i8> %2 to <64 x i32>
+  %5 = add nuw nsw <64 x i32> %3, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+  %6 = add nuw nsw <64 x i32> %5, %4
+  %7 = lshr <64 x i32> %6, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
+  %8 = trunc <64 x i32> %7 to <64 x i8>
+  store <64 x i8> %8, <64 x i8>* undef, align 4
+  ret void
+}
+
+define void @pmaddwd_32(<32 x i16>* %APtr, <32 x i16>* %BPtr, <16 x i32>* %CPtr) {
+; LEGAL256-LABEL: pmaddwd_32:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vmovdqa (%rdi), %ymm0
+; LEGAL256-NEXT:    vmovdqa 32(%rdi), %ymm1
+; LEGAL256-NEXT:    vpmaddwd (%rsi), %ymm0, %ymm0
+; LEGAL256-NEXT:    vpmaddwd 32(%rsi), %ymm1, %ymm1
+; LEGAL256-NEXT:    vmovdqa %ymm1, 32(%rdx)
+; LEGAL256-NEXT:    vmovdqa %ymm0, (%rdx)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: pmaddwd_32:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vmovdqa64 (%rdi), %zmm0
+; LEGAL512-NEXT:    vpmaddwd (%rsi), %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovdqa64 %zmm0, (%rdx)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+   %A = load <32 x i16>, <32 x i16>* %APtr
+   %B = load <32 x i16>, <32 x i16>* %BPtr
+   %a = sext <32 x i16> %A to <32 x i32>
+   %b = sext <32 x i16> %B to <32 x i32>
+   %m = mul nsw <32 x i32> %a, %b
+   %odd = shufflevector <32 x i32> %m, <32 x i32> undef, <16 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14, i32 16, i32 18, i32 20, i32 22, i32 24, i32 26, i32 28, i32 30>
+   %even = shufflevector <32 x i32> %m, <32 x i32> undef, <16 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15, i32 17, i32 19, i32 21, i32 23, i32 25, i32 27, i32 29, i32 31>
+   %ret = add <16 x i32> %odd, %even
+   store <16 x i32> %ret, <16 x i32>* %CPtr
+   ret void
+}
+
+define void @psubus_64i8_max(<64 x i8>* %xptr, <64 x i8>* %yptr, <64 x i8>* %zptr) {
+; LEGAL256-LABEL: psubus_64i8_max:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vmovdqa (%rdi), %ymm0
+; LEGAL256-NEXT:    vmovdqa 32(%rdi), %ymm1
+; LEGAL256-NEXT:    vpsubusb (%rsi), %ymm0, %ymm0
+; LEGAL256-NEXT:    vpsubusb 32(%rsi), %ymm1, %ymm1
+; LEGAL256-NEXT:    vmovdqa %ymm1, 32(%rdx)
+; LEGAL256-NEXT:    vmovdqa %ymm0, (%rdx)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: psubus_64i8_max:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vmovdqa64 (%rdi), %zmm0
+; LEGAL512-NEXT:    vpsubusb (%rsi), %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovdqa64 %zmm0, (%rdx)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+  %x = load <64 x i8>, <64 x i8>* %xptr
+  %y = load <64 x i8>, <64 x i8>* %yptr
+  %cmp = icmp ult <64 x i8> %x, %y
+  %max = select <64 x i1> %cmp, <64 x i8> %y, <64 x i8> %x
+  %res = sub <64 x i8> %max, %y
+  store <64 x i8> %res, <64 x i8>* %zptr
+  ret void
+}
+
+define i32 @_Z9test_charPcS_i(i8* nocapture readonly, i8* nocapture readonly, i32) {
+; LEGAL256-LABEL: _Z9test_charPcS_i:
+; LEGAL256:       # %bb.0: # %entry
+; LEGAL256-NEXT:    movl %edx, %eax
+; LEGAL256-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; LEGAL256-NEXT:    xorl %ecx, %ecx
+; LEGAL256-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; LEGAL256-NEXT:    vpxor %xmm2, %xmm2, %xmm2
+; LEGAL256-NEXT:    vpxor %xmm3, %xmm3, %xmm3
+; LEGAL256-NEXT:    .p2align 4, 0x90
+; LEGAL256-NEXT:  .LBB5_1: # %vector.body
+; LEGAL256-NEXT:    # =>This Inner Loop Header: Depth=1
+; LEGAL256-NEXT:    vpmovsxbw (%rdi,%rcx), %xmm4
+; LEGAL256-NEXT:    vpmovsxbw 8(%rdi,%rcx), %xmm5
+; LEGAL256-NEXT:    vpmovsxbw 16(%rdi,%rcx), %xmm6
+; LEGAL256-NEXT:    vpmovsxbw 24(%rdi,%rcx), %xmm8
+; LEGAL256-NEXT:    vpmovsxbw (%rsi,%rcx), %xmm7
+; LEGAL256-NEXT:    vpmaddwd %xmm4, %xmm7, %xmm4
+; LEGAL256-NEXT:    vpmovsxbw 8(%rsi,%rcx), %xmm7
+; LEGAL256-NEXT:    vpmaddwd %xmm5, %xmm7, %xmm5
+; LEGAL256-NEXT:    vpmovsxbw 16(%rsi,%rcx), %xmm7
+; LEGAL256-NEXT:    vpmaddwd %xmm6, %xmm7, %xmm6
+; LEGAL256-NEXT:    vpmovsxbw 24(%rsi,%rcx), %xmm7
+; LEGAL256-NEXT:    vpmaddwd %xmm8, %xmm7, %xmm7
+; LEGAL256-NEXT:    vpaddd %ymm3, %ymm7, %ymm3
+; LEGAL256-NEXT:    vpaddd %ymm2, %ymm6, %ymm2
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm5, %ymm1
+; LEGAL256-NEXT:    vpaddd %ymm0, %ymm4, %ymm0
+; LEGAL256-NEXT:    addq $32, %rcx
+; LEGAL256-NEXT:    cmpq %rcx, %rax
+; LEGAL256-NEXT:    jne .LBB5_1
+; LEGAL256-NEXT:  # %bb.2: # %middle.block
+; LEGAL256-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
+; LEGAL256-NEXT:    vpaddd %ymm3, %ymm1, %ymm1
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
+; LEGAL256-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
+; LEGAL256-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
+; LEGAL256-NEXT:    vphaddd %ymm0, %ymm0, %ymm0
+; LEGAL256-NEXT:    vmovd %xmm0, %eax
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: _Z9test_charPcS_i:
+; LEGAL512:       # %bb.0: # %entry
+; LEGAL512-NEXT:    movl %edx, %eax
+; LEGAL512-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; LEGAL512-NEXT:    xorl %ecx, %ecx
+; LEGAL512-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; LEGAL512-NEXT:    .p2align 4, 0x90
+; LEGAL512-NEXT:  .LBB5_1: # %vector.body
+; LEGAL512-NEXT:    # =>This Inner Loop Header: Depth=1
+; LEGAL512-NEXT:    vpmovsxbw (%rdi,%rcx), %zmm2
+; LEGAL512-NEXT:    vpmovsxbw (%rsi,%rcx), %zmm3
+; LEGAL512-NEXT:    vpmaddwd %zmm2, %zmm3, %zmm2
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm2, %zmm1
+; LEGAL512-NEXT:    addq $32, %rcx
+; LEGAL512-NEXT:    cmpq %rcx, %rax
+; LEGAL512-NEXT:    jne .LBB5_1
+; LEGAL512-NEXT:  # %bb.2: # %middle.block
+; LEGAL512-NEXT:    vpaddd %zmm0, %zmm1, %zmm0
+; LEGAL512-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[1,1,2,3]
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovd %xmm0, %eax
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+entry:
+  %3 = zext i32 %2 to i64
+  br label %vector.body
+
+vector.body:
+  %index = phi i64 [ %index.next, %vector.body ], [ 0, %entry ]
+  %vec.phi = phi <32 x i32> [ %11, %vector.body ], [ zeroinitializer, %entry ]
+  %4 = getelementptr inbounds i8, i8* %0, i64 %index
+  %5 = bitcast i8* %4 to <32 x i8>*
+  %wide.load = load <32 x i8>, <32 x i8>* %5, align 1
+  %6 = sext <32 x i8> %wide.load to <32 x i32>
+  %7 = getelementptr inbounds i8, i8* %1, i64 %index
+  %8 = bitcast i8* %7 to <32 x i8>*
+  %wide.load14 = load <32 x i8>, <32 x i8>* %8, align 1
+  %9 = sext <32 x i8> %wide.load14 to <32 x i32>
+  %10 = mul nsw <32 x i32> %9, %6
+  %11 = add nsw <32 x i32> %10, %vec.phi
+  %index.next = add i64 %index, 32
+  %12 = icmp eq i64 %index.next, %3
+  br i1 %12, label %middle.block, label %vector.body
+
+middle.block:
+  %rdx.shuf1 = shufflevector <32 x i32> %11, <32 x i32> undef, <32 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx1 = add <32 x i32> %11, %rdx.shuf1
+  %rdx.shuf = shufflevector <32 x i32> %bin.rdx1, <32 x i32> undef, <32 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx = add <32 x i32> %bin.rdx1, %rdx.shuf
+  %rdx.shuf15 = shufflevector <32 x i32> %bin.rdx, <32 x i32> undef, <32 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx32 = add <32 x i32> %bin.rdx, %rdx.shuf15
+  %rdx.shuf17 = shufflevector <32 x i32> %bin.rdx32, <32 x i32> undef, <32 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx18 = add <32 x i32> %bin.rdx32, %rdx.shuf17
+  %rdx.shuf19 = shufflevector <32 x i32> %bin.rdx18, <32 x i32> undef, <32 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx20 = add <32 x i32> %bin.rdx18, %rdx.shuf19
+  %13 = extractelement <32 x i32> %bin.rdx20, i32 0
+  ret i32 %13
+}
+
+@a = global [1024 x i8] zeroinitializer, align 16
+@b = global [1024 x i8] zeroinitializer, align 16
+
+define i32 @sad_16i8() {
+; LEGAL256-LABEL: sad_16i8:
+; LEGAL256:       # %bb.0: # %entry
+; LEGAL256-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; LEGAL256-NEXT:    movq $-1024, %rax # imm = 0xFC00
+; LEGAL256-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; LEGAL256-NEXT:    .p2align 4, 0x90
+; LEGAL256-NEXT:  .LBB6_1: # %vector.body
+; LEGAL256-NEXT:    # =>This Inner Loop Header: Depth=1
+; LEGAL256-NEXT:    vmovdqu a+1024(%rax), %xmm2
+; LEGAL256-NEXT:    vpsadbw b+1024(%rax), %xmm2, %xmm2
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm2, %ymm1
+; LEGAL256-NEXT:    addq $4, %rax
+; LEGAL256-NEXT:    jne .LBB6_1
+; LEGAL256-NEXT:  # %bb.2: # %middle.block
+; LEGAL256-NEXT:    vpaddd %ymm0, %ymm1, %ymm0
+; LEGAL256-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
+; LEGAL256-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
+; LEGAL256-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
+; LEGAL256-NEXT:    vphaddd %ymm0, %ymm0, %ymm0
+; LEGAL256-NEXT:    vmovd %xmm0, %eax
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: sad_16i8:
+; LEGAL512:       # %bb.0: # %entry
+; LEGAL512-NEXT:    vpxor %xmm0, %xmm0, %xmm0
+; LEGAL512-NEXT:    movq $-1024, %rax # imm = 0xFC00
+; LEGAL512-NEXT:    .p2align 4, 0x90
+; LEGAL512-NEXT:  .LBB6_1: # %vector.body
+; LEGAL512-NEXT:    # =>This Inner Loop Header: Depth=1
+; LEGAL512-NEXT:    vmovdqu a+1024(%rax), %xmm1
+; LEGAL512-NEXT:    vpsadbw b+1024(%rax), %xmm1, %xmm1
+; LEGAL512-NEXT:    vpaddd %zmm0, %zmm1, %zmm0
+; LEGAL512-NEXT:    addq $4, %rax
+; LEGAL512-NEXT:    jne .LBB6_1
+; LEGAL512-NEXT:  # %bb.2: # %middle.block
+; LEGAL512-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[1,1,2,3]
+; LEGAL512-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovd %xmm0, %eax
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+entry:
+  br label %vector.body
+
+vector.body:
+  %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
+  %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %10, %vector.body ]
+  %0 = getelementptr inbounds [1024 x i8], [1024 x i8]* @a, i64 0, i64 %index
+  %1 = bitcast i8* %0 to <16 x i8>*
+  %wide.load = load <16 x i8>, <16 x i8>* %1, align 4
+  %2 = zext <16 x i8> %wide.load to <16 x i32>
+  %3 = getelementptr inbounds [1024 x i8], [1024 x i8]* @b, i64 0, i64 %index
+  %4 = bitcast i8* %3 to <16 x i8>*
+  %wide.load1 = load <16 x i8>, <16 x i8>* %4, align 4
+  %5 = zext <16 x i8> %wide.load1 to <16 x i32>
+  %6 = sub nsw <16 x i32> %2, %5
+  %7 = icmp sgt <16 x i32> %6, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
+  %8 = sub nsw <16 x i32> zeroinitializer, %6
+  %9 = select <16 x i1> %7, <16 x i32> %6, <16 x i32> %8
+  %10 = add nsw <16 x i32> %9, %vec.phi
+  %index.next = add i64 %index, 4
+  %11 = icmp eq i64 %index.next, 1024
+  br i1 %11, label %middle.block, label %vector.body
+
+middle.block:
+  %.lcssa = phi <16 x i32> [ %10, %vector.body ]
+  %rdx.shuf = shufflevector <16 x i32> %.lcssa, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx = add <16 x i32> %.lcssa, %rdx.shuf
+  %rdx.shuf2 = shufflevector <16 x i32> %bin.rdx, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx2 = add <16 x i32> %bin.rdx, %rdx.shuf2
+  %rdx.shuf3 = shufflevector <16 x i32> %bin.rdx2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx3 = add <16 x i32> %bin.rdx2, %rdx.shuf3
+  %rdx.shuf4 = shufflevector <16 x i32> %bin.rdx3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+  %bin.rdx4 = add <16 x i32> %bin.rdx3, %rdx.shuf4
+  %12 = extractelement <16 x i32> %bin.rdx4, i32 0
+  ret i32 %12
+}
+
+define void @sbto16f32(<16 x i16> %a, <16 x float>* %res) {
+; LEGAL256-LABEL: sbto16f32:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL256-NEXT:    kshiftrw $8, %k0, %k1
+; LEGAL256-NEXT:    vpmovm2d %k1, %ymm0
+; LEGAL256-NEXT:    vcvtdq2ps %ymm0, %ymm0
+; LEGAL256-NEXT:    vpmovm2d %k0, %ymm1
+; LEGAL256-NEXT:    vcvtdq2ps %ymm1, %ymm1
+; LEGAL256-NEXT:    vmovaps %ymm1, (%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm0, 32(%rdi)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: sbto16f32:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL512-NEXT:    vpmovm2d %k0, %zmm0
+; LEGAL512-NEXT:    vcvtdq2ps %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovaps %zmm0, (%rdi)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+  %mask = icmp slt <16 x i16> %a, zeroinitializer
+  %1 = sitofp <16 x i1> %mask to <16 x float>
+  store <16 x float> %1, <16 x float>* %res
+  ret void
+}
+
+define void @sbto16f64(<16 x i16> %a, <16 x double>* %res)  {
+; LEGAL256-LABEL: sbto16f64:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL256-NEXT:    kshiftrw $8, %k0, %k1
+; LEGAL256-NEXT:    vpmovm2d %k1, %ymm0
+; LEGAL256-NEXT:    vcvtdq2pd %xmm0, %ymm1
+; LEGAL256-NEXT:    vextracti128 $1, %ymm0, %xmm0
+; LEGAL256-NEXT:    vcvtdq2pd %xmm0, %ymm0
+; LEGAL256-NEXT:    vpmovm2d %k0, %ymm2
+; LEGAL256-NEXT:    vcvtdq2pd %xmm2, %ymm3
+; LEGAL256-NEXT:    vextracti128 $1, %ymm2, %xmm2
+; LEGAL256-NEXT:    vcvtdq2pd %xmm2, %ymm2
+; LEGAL256-NEXT:    vmovaps %ymm2, 32(%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm3, (%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm0, 96(%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm1, 64(%rdi)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: sbto16f64:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL512-NEXT:    vpmovm2d %k0, %zmm0
+; LEGAL512-NEXT:    vcvtdq2pd %ymm0, %zmm1
+; LEGAL512-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
+; LEGAL512-NEXT:    vcvtdq2pd %ymm0, %zmm0
+; LEGAL512-NEXT:    vmovaps %zmm0, 64(%rdi)
+; LEGAL512-NEXT:    vmovaps %zmm1, (%rdi)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+  %mask = icmp slt <16 x i16> %a, zeroinitializer
+  %1 = sitofp <16 x i1> %mask to <16 x double>
+  store <16 x double> %1, <16 x double>* %res
+  ret void
+}
+
+define void @ubto16f32_256(<16 x i16> %a, <16 x float>* %res) {
+; LEGAL256-LABEL: ubto16f32_256:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL256-NEXT:    kshiftrw $8, %k0, %k1
+; LEGAL256-NEXT:    vpmovm2d %k1, %ymm0
+; LEGAL256-NEXT:    vpsrld $31, %ymm0, %ymm0
+; LEGAL256-NEXT:    vcvtdq2ps %ymm0, %ymm0
+; LEGAL256-NEXT:    vpmovm2d %k0, %ymm1
+; LEGAL256-NEXT:    vpsrld $31, %ymm1, %ymm1
+; LEGAL256-NEXT:    vcvtdq2ps %ymm1, %ymm1
+; LEGAL256-NEXT:    vmovaps %ymm1, (%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm0, 32(%rdi)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: ubto16f32_256:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL512-NEXT:    vpmovm2d %k0, %zmm0
+; LEGAL512-NEXT:    vpsrld $31, %zmm0, %zmm0
+; LEGAL512-NEXT:    vcvtdq2ps %zmm0, %zmm0
+; LEGAL512-NEXT:    vmovaps %zmm0, (%rdi)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+  %mask = icmp slt <16 x i16> %a, zeroinitializer
+  %1 = uitofp <16 x i1> %mask to <16 x float>
+  store <16 x float> %1, <16 x float>* %res
+  ret void
+}
+
+define void @ubto16f64(<16 x i16> %a, <16 x double>* %res) {
+; LEGAL256-LABEL: ubto16f64:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL256-NEXT:    kshiftrw $8, %k0, %k1
+; LEGAL256-NEXT:    vpmovm2d %k1, %ymm0
+; LEGAL256-NEXT:    vpsrld $31, %ymm0, %ymm0
+; LEGAL256-NEXT:    vcvtdq2pd %xmm0, %ymm1
+; LEGAL256-NEXT:    vextracti128 $1, %ymm0, %xmm0
+; LEGAL256-NEXT:    vcvtdq2pd %xmm0, %ymm0
+; LEGAL256-NEXT:    vpmovm2d %k0, %ymm2
+; LEGAL256-NEXT:    vpsrld $31, %ymm2, %ymm2
+; LEGAL256-NEXT:    vcvtdq2pd %xmm2, %ymm3
+; LEGAL256-NEXT:    vextracti128 $1, %ymm2, %xmm2
+; LEGAL256-NEXT:    vcvtdq2pd %xmm2, %ymm2
+; LEGAL256-NEXT:    vmovaps %ymm2, 32(%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm3, (%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm0, 96(%rdi)
+; LEGAL256-NEXT:    vmovaps %ymm1, 64(%rdi)
+; LEGAL256-NEXT:    vzeroupper
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: ubto16f64:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vpmovw2m %ymm0, %k0
+; LEGAL512-NEXT:    vpmovm2d %k0, %zmm0
+; LEGAL512-NEXT:    vpsrld $31, %zmm0, %zmm0
+; LEGAL512-NEXT:    vcvtdq2pd %ymm0, %zmm1
+; LEGAL512-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
+; LEGAL512-NEXT:    vcvtdq2pd %ymm0, %zmm0
+; LEGAL512-NEXT:    vmovaps %zmm0, 64(%rdi)
+; LEGAL512-NEXT:    vmovaps %zmm1, (%rdi)
+; LEGAL512-NEXT:    vzeroupper
+; LEGAL512-NEXT:    retq
+  %mask = icmp slt <16 x i16> %a, zeroinitializer
+  %1 = uitofp <16 x i1> %mask to <16 x double>
+  store <16 x double> %1, <16 x double>* %res
+  ret void
+}
+
+define <16 x i16> @test_16f32toub(<16 x float>* %ptr, <16 x i16> %passthru) {
+; LEGAL256-LABEL: test_16f32toub:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vcvttps2dq (%rdi), %ymm1
+; LEGAL256-NEXT:    vpmovdw %ymm1, %xmm1
+; LEGAL256-NEXT:    vcvttps2dq 32(%rdi), %ymm2
+; LEGAL256-NEXT:    vpmovdw %ymm2, %xmm2
+; LEGAL256-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
+; LEGAL256-NEXT:    vpsllw $15, %ymm1, %ymm1
+; LEGAL256-NEXT:    vpmovw2m %ymm1, %k1
+; LEGAL256-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: test_16f32toub:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vcvttps2dq (%rdi), %zmm1
+; LEGAL512-NEXT:    vpslld $31, %zmm1, %zmm1
+; LEGAL512-NEXT:    vpmovd2m %zmm1, %k1
+; LEGAL512-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
+; LEGAL512-NEXT:    retq
+  %a = load <16 x float>, <16 x float>* %ptr
+  %mask = fptoui <16 x float> %a to <16 x i1>
+  %select = select <16 x i1> %mask, <16 x i16> %passthru, <16 x i16> zeroinitializer
+  ret <16 x i16> %select
+}
+
+define <16 x i16> @test_16f32tosb(<16 x float>* %ptr, <16 x i16> %passthru) {
+; LEGAL256-LABEL: test_16f32tosb:
+; LEGAL256:       # %bb.0:
+; LEGAL256-NEXT:    vcvttps2dq (%rdi), %ymm1
+; LEGAL256-NEXT:    vpmovdw %ymm1, %xmm1
+; LEGAL256-NEXT:    vcvttps2dq 32(%rdi), %ymm2
+; LEGAL256-NEXT:    vpmovdw %ymm2, %xmm2
+; LEGAL256-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
+; LEGAL256-NEXT:    vpsllw $15, %ymm1, %ymm1
+; LEGAL256-NEXT:    vpmovw2m %ymm1, %k1
+; LEGAL256-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
+; LEGAL256-NEXT:    retq
+;
+; LEGAL512-LABEL: test_16f32tosb:
+; LEGAL512:       # %bb.0:
+; LEGAL512-NEXT:    vcvttps2dq (%rdi), %zmm1
+; LEGAL512-NEXT:    vpmovd2m %zmm1, %k1
+; LEGAL512-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
+; LEGAL512-NEXT:    retq
+  %a = load <16 x float>, <16 x float>* %ptr
+  %mask = fptosi <16 x float> %a to <16 x i1>
+  %select = select <16 x i1> %mask, <16 x i16> %passthru, <16 x i16> zeroinitializer
+  ret <16 x i16> %select
+}

test/CodeGen/X86/required-vector-width.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512vl,avx512bw,avx512dq,prefer-256-bit | FileCheck %s
-
-; This file primarily contains tests for specific places in X86ISelLowering.cpp that needed be made aware of the legalizer not allowing 512-bit vectors due to prefer-256-bit even though AVX512 is enabled.
-
-define void @add256(<16 x i32>* %a, <16 x i32>* %b, <16 x i32>* %c) "required-vector-width"="256" {
-; CHECK-LABEL: add256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa (%rdi), %ymm0
-; CHECK-NEXT:    vmovdqa 32(%rdi), %ymm1
-; CHECK-NEXT:    vpaddd (%rsi), %ymm0, %ymm0
-; CHECK-NEXT:    vpaddd 32(%rsi), %ymm1, %ymm1
-; CHECK-NEXT:    vmovdqa %ymm1, 32(%rdx)
-; CHECK-NEXT:    vmovdqa %ymm0, (%rdx)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %d = load <16 x i32>, <16 x i32>* %a
-  %e = load <16 x i32>, <16 x i32>* %b
-  %f = add <16 x i32> %d, %e
-  store <16 x i32> %f, <16 x i32>* %c
-  ret void
-}
-
-define void @add512(<16 x i32>* %a, <16 x i32>* %b, <16 x i32>* %c) "required-vector-width"="512" {
-; CHECK-LABEL: add512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa64 (%rdi), %zmm0
-; CHECK-NEXT:    vpaddd (%rsi), %zmm0, %zmm0
-; CHECK-NEXT:    vmovdqa64 %zmm0, (%rdx)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %d = load <16 x i32>, <16 x i32>* %a
-  %e = load <16 x i32>, <16 x i32>* %b
-  %f = add <16 x i32> %d, %e
-  store <16 x i32> %f, <16 x i32>* %c
-  ret void
-}
-
-define void @avg_v64i8_256(<64 x i8>* %a, <64 x i8>* %b) "required-vector-width"="256" {
-; CHECK-LABEL: avg_v64i8_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa (%rsi), %ymm0
-; CHECK-NEXT:    vmovdqa 32(%rsi), %ymm1
-; CHECK-NEXT:    vpavgb (%rdi), %ymm0, %ymm0
-; CHECK-NEXT:    vpavgb 32(%rdi), %ymm1, %ymm1
-; CHECK-NEXT:    vmovdqu %ymm1, (%rax)
-; CHECK-NEXT:    vmovdqu %ymm0, (%rax)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %1 = load <64 x i8>, <64 x i8>* %a
-  %2 = load <64 x i8>, <64 x i8>* %b
-  %3 = zext <64 x i8> %1 to <64 x i32>
-  %4 = zext <64 x i8> %2 to <64 x i32>
-  %5 = add nuw nsw <64 x i32> %3, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-  %6 = add nuw nsw <64 x i32> %5, %4
-  %7 = lshr <64 x i32> %6, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-  %8 = trunc <64 x i32> %7 to <64 x i8>
-  store <64 x i8> %8, <64 x i8>* undef, align 4
-  ret void
-}
-
-
-define void @avg_v64i8_512(<64 x i8>* %a, <64 x i8>* %b) "required-vector-width"="512" {
-; CHECK-LABEL: avg_v64i8_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa64 (%rsi), %zmm0
-; CHECK-NEXT:    vpavgb (%rdi), %zmm0, %zmm0
-; CHECK-NEXT:    vmovdqu64 %zmm0, (%rax)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %1 = load <64 x i8>, <64 x i8>* %a
-  %2 = load <64 x i8>, <64 x i8>* %b
-  %3 = zext <64 x i8> %1 to <64 x i32>
-  %4 = zext <64 x i8> %2 to <64 x i32>
-  %5 = add nuw nsw <64 x i32> %3, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-  %6 = add nuw nsw <64 x i32> %5, %4
-  %7 = lshr <64 x i32> %6, <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-  %8 = trunc <64 x i32> %7 to <64 x i8>
-  store <64 x i8> %8, <64 x i8>* undef, align 4
-  ret void
-}
-
-define void @pmaddwd_32_256(<32 x i16>* %APtr, <32 x i16>* %BPtr, <16 x i32>* %CPtr) "required-vector-width"="256" {
-; CHECK-LABEL: pmaddwd_32_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa (%rdi), %ymm0
-; CHECK-NEXT:    vmovdqa 32(%rdi), %ymm1
-; CHECK-NEXT:    vpmaddwd (%rsi), %ymm0, %ymm0
-; CHECK-NEXT:    vpmaddwd 32(%rsi), %ymm1, %ymm1
-; CHECK-NEXT:    vmovdqa %ymm1, 32(%rdx)
-; CHECK-NEXT:    vmovdqa %ymm0, (%rdx)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-   %A = load <32 x i16>, <32 x i16>* %APtr
-   %B = load <32 x i16>, <32 x i16>* %BPtr
-   %a = sext <32 x i16> %A to <32 x i32>
-   %b = sext <32 x i16> %B to <32 x i32>
-   %m = mul nsw <32 x i32> %a, %b
-   %odd = shufflevector <32 x i32> %m, <32 x i32> undef, <16 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14, i32 16, i32 18, i32 20, i32 22, i32 24, i32 26, i32 28, i32 30>
-   %even = shufflevector <32 x i32> %m, <32 x i32> undef, <16 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15, i32 17, i32 19, i32 21, i32 23, i32 25, i32 27, i32 29, i32 31>
-   %ret = add <16 x i32> %odd, %even
-   store <16 x i32> %ret, <16 x i32>* %CPtr
-   ret void
-}
-
-define void @pmaddwd_32_512(<32 x i16>* %APtr, <32 x i16>* %BPtr, <16 x i32>* %CPtr) "required-vector-width"="512" {
-; CHECK-LABEL: pmaddwd_32_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa64 (%rdi), %zmm0
-; CHECK-NEXT:    vpmaddwd (%rsi), %zmm0, %zmm0
-; CHECK-NEXT:    vmovdqa64 %zmm0, (%rdx)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-   %A = load <32 x i16>, <32 x i16>* %APtr
-   %B = load <32 x i16>, <32 x i16>* %BPtr
-   %a = sext <32 x i16> %A to <32 x i32>
-   %b = sext <32 x i16> %B to <32 x i32>
-   %m = mul nsw <32 x i32> %a, %b
-   %odd = shufflevector <32 x i32> %m, <32 x i32> undef, <16 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14, i32 16, i32 18, i32 20, i32 22, i32 24, i32 26, i32 28, i32 30>
-   %even = shufflevector <32 x i32> %m, <32 x i32> undef, <16 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15, i32 17, i32 19, i32 21, i32 23, i32 25, i32 27, i32 29, i32 31>
-   %ret = add <16 x i32> %odd, %even
-   store <16 x i32> %ret, <16 x i32>* %CPtr
-   ret void
-}
-
-define void @psubus_64i8_max_256(<64 x i8>* %xptr, <64 x i8>* %yptr, <64 x i8>* %zptr) "required-vector-width"="256" {
-; CHECK-LABEL: psubus_64i8_max_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa (%rdi), %ymm0
-; CHECK-NEXT:    vmovdqa 32(%rdi), %ymm1
-; CHECK-NEXT:    vpsubusb (%rsi), %ymm0, %ymm0
-; CHECK-NEXT:    vpsubusb 32(%rsi), %ymm1, %ymm1
-; CHECK-NEXT:    vmovdqa %ymm1, 32(%rdx)
-; CHECK-NEXT:    vmovdqa %ymm0, (%rdx)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %x = load <64 x i8>, <64 x i8>* %xptr
-  %y = load <64 x i8>, <64 x i8>* %yptr
-  %cmp = icmp ult <64 x i8> %x, %y
-  %max = select <64 x i1> %cmp, <64 x i8> %y, <64 x i8> %x
-  %res = sub <64 x i8> %max, %y
-  store <64 x i8> %res, <64 x i8>* %zptr
-  ret void
-}
-
-define void @psubus_64i8_max_512(<64 x i8>* %xptr, <64 x i8>* %yptr, <64 x i8>* %zptr) "required-vector-width"="512" {
-; CHECK-LABEL: psubus_64i8_max_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vmovdqa64 (%rdi), %zmm0
-; CHECK-NEXT:    vpsubusb (%rsi), %zmm0, %zmm0
-; CHECK-NEXT:    vmovdqa64 %zmm0, (%rdx)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %x = load <64 x i8>, <64 x i8>* %xptr
-  %y = load <64 x i8>, <64 x i8>* %yptr
-  %cmp = icmp ult <64 x i8> %x, %y
-  %max = select <64 x i1> %cmp, <64 x i8> %y, <64 x i8> %x
-  %res = sub <64 x i8> %max, %y
-  store <64 x i8> %res, <64 x i8>* %zptr
-  ret void
-}
-
-define i32 @_Z9test_charPcS_i_256(i8* nocapture readonly, i8* nocapture readonly, i32) "required-vector-width"="256" {
-; CHECK-LABEL: _Z9test_charPcS_i_256:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movl %edx, %eax
-; CHECK-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    xorl %ecx, %ecx
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; CHECK-NEXT:    vpxor %xmm3, %xmm3, %xmm3
-; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  .LBB8_1: # %vector.body
-; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vpmovsxbw (%rdi,%rcx), %xmm4
-; CHECK-NEXT:    vpmovsxbw 8(%rdi,%rcx), %xmm5
-; CHECK-NEXT:    vpmovsxbw 16(%rdi,%rcx), %xmm6
-; CHECK-NEXT:    vpmovsxbw 24(%rdi,%rcx), %xmm8
-; CHECK-NEXT:    vpmovsxbw (%rsi,%rcx), %xmm7
-; CHECK-NEXT:    vpmaddwd %xmm4, %xmm7, %xmm4
-; CHECK-NEXT:    vpmovsxbw 8(%rsi,%rcx), %xmm7
-; CHECK-NEXT:    vpmaddwd %xmm5, %xmm7, %xmm5
-; CHECK-NEXT:    vpmovsxbw 16(%rsi,%rcx), %xmm7
-; CHECK-NEXT:    vpmaddwd %xmm6, %xmm7, %xmm6
-; CHECK-NEXT:    vpmovsxbw 24(%rsi,%rcx), %xmm7
-; CHECK-NEXT:    vpmaddwd %xmm8, %xmm7, %xmm7
-; CHECK-NEXT:    vpaddd %ymm3, %ymm7, %ymm3
-; CHECK-NEXT:    vpaddd %ymm2, %ymm6, %ymm2
-; CHECK-NEXT:    vpaddd %ymm1, %ymm5, %ymm1
-; CHECK-NEXT:    vpaddd %ymm0, %ymm4, %ymm0
-; CHECK-NEXT:    addq $32, %rcx
-; CHECK-NEXT:    cmpq %rcx, %rax
-; CHECK-NEXT:    jne .LBB8_1
-; CHECK-NEXT:  # %bb.2: # %middle.block
-; CHECK-NEXT:    vpaddd %ymm2, %ymm0, %ymm0
-; CHECK-NEXT:    vpaddd %ymm3, %ymm1, %ymm1
-; CHECK-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
-; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; CHECK-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
-; CHECK-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
-; CHECK-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
-; CHECK-NEXT:    vphaddd %ymm0, %ymm0, %ymm0
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-entry:
-  %3 = zext i32 %2 to i64
-  br label %vector.body
-
-vector.body:
-  %index = phi i64 [ %index.next, %vector.body ], [ 0, %entry ]
-  %vec.phi = phi <32 x i32> [ %11, %vector.body ], [ zeroinitializer, %entry ]
-  %4 = getelementptr inbounds i8, i8* %0, i64 %index
-  %5 = bitcast i8* %4 to <32 x i8>*
-  %wide.load = load <32 x i8>, <32 x i8>* %5, align 1
-  %6 = sext <32 x i8> %wide.load to <32 x i32>
-  %7 = getelementptr inbounds i8, i8* %1, i64 %index
-  %8 = bitcast i8* %7 to <32 x i8>*
-  %wide.load14 = load <32 x i8>, <32 x i8>* %8, align 1
-  %9 = sext <32 x i8> %wide.load14 to <32 x i32>
-  %10 = mul nsw <32 x i32> %9, %6
-  %11 = add nsw <32 x i32> %10, %vec.phi
-  %index.next = add i64 %index, 32
-  %12 = icmp eq i64 %index.next, %3
-  br i1 %12, label %middle.block, label %vector.body
-
-middle.block:
-  %rdx.shuf1 = shufflevector <32 x i32> %11, <32 x i32> undef, <32 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx1 = add <32 x i32> %11, %rdx.shuf1
-  %rdx.shuf = shufflevector <32 x i32> %bin.rdx1, <32 x i32> undef, <32 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx = add <32 x i32> %bin.rdx1, %rdx.shuf
-  %rdx.shuf15 = shufflevector <32 x i32> %bin.rdx, <32 x i32> undef, <32 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx32 = add <32 x i32> %bin.rdx, %rdx.shuf15
-  %rdx.shuf17 = shufflevector <32 x i32> %bin.rdx32, <32 x i32> undef, <32 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx18 = add <32 x i32> %bin.rdx32, %rdx.shuf17
-  %rdx.shuf19 = shufflevector <32 x i32> %bin.rdx18, <32 x i32> undef, <32 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx20 = add <32 x i32> %bin.rdx18, %rdx.shuf19
-  %13 = extractelement <32 x i32> %bin.rdx20, i32 0
-  ret i32 %13
-}
-
-define i32 @_Z9test_charPcS_i_512(i8* nocapture readonly, i8* nocapture readonly, i32) "required-vector-width"="512" {
-; CHECK-LABEL: _Z9test_charPcS_i_512:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    movl %edx, %eax
-; CHECK-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    xorl %ecx, %ecx
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  .LBB9_1: # %vector.body
-; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vpmovsxbw (%rdi,%rcx), %zmm2
-; CHECK-NEXT:    vpmovsxbw (%rsi,%rcx), %zmm3
-; CHECK-NEXT:    vpmaddwd %zmm2, %zmm3, %zmm2
-; CHECK-NEXT:    vpaddd %zmm1, %zmm2, %zmm1
-; CHECK-NEXT:    addq $32, %rcx
-; CHECK-NEXT:    cmpq %rcx, %rax
-; CHECK-NEXT:    jne .LBB9_1
-; CHECK-NEXT:  # %bb.2: # %middle.block
-; CHECK-NEXT:    vpaddd %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[1,1,2,3]
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-entry:
-  %3 = zext i32 %2 to i64
-  br label %vector.body
-
-vector.body:
-  %index = phi i64 [ %index.next, %vector.body ], [ 0, %entry ]
-  %vec.phi = phi <32 x i32> [ %11, %vector.body ], [ zeroinitializer, %entry ]
-  %4 = getelementptr inbounds i8, i8* %0, i64 %index
-  %5 = bitcast i8* %4 to <32 x i8>*
-  %wide.load = load <32 x i8>, <32 x i8>* %5, align 1
-  %6 = sext <32 x i8> %wide.load to <32 x i32>
-  %7 = getelementptr inbounds i8, i8* %1, i64 %index
-  %8 = bitcast i8* %7 to <32 x i8>*
-  %wide.load14 = load <32 x i8>, <32 x i8>* %8, align 1
-  %9 = sext <32 x i8> %wide.load14 to <32 x i32>
-  %10 = mul nsw <32 x i32> %9, %6
-  %11 = add nsw <32 x i32> %10, %vec.phi
-  %index.next = add i64 %index, 32
-  %12 = icmp eq i64 %index.next, %3
-  br i1 %12, label %middle.block, label %vector.body
-
-middle.block:
-  %rdx.shuf1 = shufflevector <32 x i32> %11, <32 x i32> undef, <32 x i32> <i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx1 = add <32 x i32> %11, %rdx.shuf1
-  %rdx.shuf = shufflevector <32 x i32> %bin.rdx1, <32 x i32> undef, <32 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx = add <32 x i32> %bin.rdx1, %rdx.shuf
-  %rdx.shuf15 = shufflevector <32 x i32> %bin.rdx, <32 x i32> undef, <32 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx32 = add <32 x i32> %bin.rdx, %rdx.shuf15
-  %rdx.shuf17 = shufflevector <32 x i32> %bin.rdx32, <32 x i32> undef, <32 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx18 = add <32 x i32> %bin.rdx32, %rdx.shuf17
-  %rdx.shuf19 = shufflevector <32 x i32> %bin.rdx18, <32 x i32> undef, <32 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx20 = add <32 x i32> %bin.rdx18, %rdx.shuf19
-  %13 = extractelement <32 x i32> %bin.rdx20, i32 0
-  ret i32 %13
-}
-
-@a = global [1024 x i8] zeroinitializer, align 16
-@b = global [1024 x i8] zeroinitializer, align 16
-
-define i32 @sad_16i8_256() "required-vector-width"="256" {
-; CHECK-LABEL: sad_16i8_256:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    movq $-1024, %rax # imm = 0xFC00
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  .LBB10_1: # %vector.body
-; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmovdqu a+1024(%rax), %xmm2
-; CHECK-NEXT:    vpsadbw b+1024(%rax), %xmm2, %xmm2
-; CHECK-NEXT:    vpaddd %ymm1, %ymm2, %ymm1
-; CHECK-NEXT:    addq $4, %rax
-; CHECK-NEXT:    jne .LBB10_1
-; CHECK-NEXT:  # %bb.2: # %middle.block
-; CHECK-NEXT:    vpaddd %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; CHECK-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
-; CHECK-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
-; CHECK-NEXT:    vpaddd %ymm1, %ymm0, %ymm0
-; CHECK-NEXT:    vphaddd %ymm0, %ymm0, %ymm0
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-entry:
-  br label %vector.body
-
-vector.body:
-  %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
-  %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %10, %vector.body ]
-  %0 = getelementptr inbounds [1024 x i8], [1024 x i8]* @a, i64 0, i64 %index
-  %1 = bitcast i8* %0 to <16 x i8>*
-  %wide.load = load <16 x i8>, <16 x i8>* %1, align 4
-  %2 = zext <16 x i8> %wide.load to <16 x i32>
-  %3 = getelementptr inbounds [1024 x i8], [1024 x i8]* @b, i64 0, i64 %index
-  %4 = bitcast i8* %3 to <16 x i8>*
-  %wide.load1 = load <16 x i8>, <16 x i8>* %4, align 4
-  %5 = zext <16 x i8> %wide.load1 to <16 x i32>
-  %6 = sub nsw <16 x i32> %2, %5
-  %7 = icmp sgt <16 x i32> %6, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
-  %8 = sub nsw <16 x i32> zeroinitializer, %6
-  %9 = select <16 x i1> %7, <16 x i32> %6, <16 x i32> %8
-  %10 = add nsw <16 x i32> %9, %vec.phi
-  %index.next = add i64 %index, 4
-  %11 = icmp eq i64 %index.next, 1024
-  br i1 %11, label %middle.block, label %vector.body
-
-middle.block:
-  %.lcssa = phi <16 x i32> [ %10, %vector.body ]
-  %rdx.shuf = shufflevector <16 x i32> %.lcssa, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx = add <16 x i32> %.lcssa, %rdx.shuf
-  %rdx.shuf2 = shufflevector <16 x i32> %bin.rdx, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx2 = add <16 x i32> %bin.rdx, %rdx.shuf2
-  %rdx.shuf3 = shufflevector <16 x i32> %bin.rdx2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx3 = add <16 x i32> %bin.rdx2, %rdx.shuf3
-  %rdx.shuf4 = shufflevector <16 x i32> %bin.rdx3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx4 = add <16 x i32> %bin.rdx3, %rdx.shuf4
-  %12 = extractelement <16 x i32> %bin.rdx4, i32 0
-  ret i32 %12
-}
-
-define i32 @sad_16i8_512() "required-vector-width"="512" {
-; CHECK-LABEL: sad_16i8_512:
-; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; CHECK-NEXT:    movq $-1024, %rax # imm = 0xFC00
-; CHECK-NEXT:    .p2align 4, 0x90
-; CHECK-NEXT:  .LBB11_1: # %vector.body
-; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmovdqu a+1024(%rax), %xmm1
-; CHECK-NEXT:    vpsadbw b+1024(%rax), %xmm1, %xmm1
-; CHECK-NEXT:    vpaddd %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    addq $4, %rax
-; CHECK-NEXT:    jne .LBB11_1
-; CHECK-NEXT:  # %bb.2: # %middle.block
-; CHECK-NEXT:    vextracti64x4 $1, %zmm0, %ymm1
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm1
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[1,1,2,3]
-; CHECK-NEXT:    vpaddd %zmm1, %zmm0, %zmm0
-; CHECK-NEXT:    vmovd %xmm0, %eax
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-entry:
-  br label %vector.body
-
-vector.body:
-  %index = phi i64 [ 0, %entry ], [ %index.next, %vector.body ]
-  %vec.phi = phi <16 x i32> [ zeroinitializer, %entry ], [ %10, %vector.body ]
-  %0 = getelementptr inbounds [1024 x i8], [1024 x i8]* @a, i64 0, i64 %index
-  %1 = bitcast i8* %0 to <16 x i8>*
-  %wide.load = load <16 x i8>, <16 x i8>* %1, align 4
-  %2 = zext <16 x i8> %wide.load to <16 x i32>
-  %3 = getelementptr inbounds [1024 x i8], [1024 x i8]* @b, i64 0, i64 %index
-  %4 = bitcast i8* %3 to <16 x i8>*
-  %wide.load1 = load <16 x i8>, <16 x i8>* %4, align 4
-  %5 = zext <16 x i8> %wide.load1 to <16 x i32>
-  %6 = sub nsw <16 x i32> %2, %5
-  %7 = icmp sgt <16 x i32> %6, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
-  %8 = sub nsw <16 x i32> zeroinitializer, %6
-  %9 = select <16 x i1> %7, <16 x i32> %6, <16 x i32> %8
-  %10 = add nsw <16 x i32> %9, %vec.phi
-  %index.next = add i64 %index, 4
-  %11 = icmp eq i64 %index.next, 1024
-  br i1 %11, label %middle.block, label %vector.body
-
-middle.block:
-  %.lcssa = phi <16 x i32> [ %10, %vector.body ]
-  %rdx.shuf = shufflevector <16 x i32> %.lcssa, <16 x i32> undef, <16 x i32> <i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx = add <16 x i32> %.lcssa, %rdx.shuf
-  %rdx.shuf2 = shufflevector <16 x i32> %bin.rdx, <16 x i32> undef, <16 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx2 = add <16 x i32> %bin.rdx, %rdx.shuf2
-  %rdx.shuf3 = shufflevector <16 x i32> %bin.rdx2, <16 x i32> undef, <16 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx3 = add <16 x i32> %bin.rdx2, %rdx.shuf3
-  %rdx.shuf4 = shufflevector <16 x i32> %bin.rdx3, <16 x i32> undef, <16 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
-  %bin.rdx4 = add <16 x i32> %bin.rdx3, %rdx.shuf4
-  %12 = extractelement <16 x i32> %bin.rdx4, i32 0
-  ret i32 %12
-}
-
-define void @sbto16f32_256(<16 x i16> %a, <16 x float>* %res) "required-vector-width"="256" {
-; CHECK-LABEL: sbto16f32_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    kshiftrw $8, %k0, %k1
-; CHECK-NEXT:    vpmovm2d %k1, %ymm0
-; CHECK-NEXT:    vcvtdq2ps %ymm0, %ymm0
-; CHECK-NEXT:    vpmovm2d %k0, %ymm1
-; CHECK-NEXT:    vcvtdq2ps %ymm1, %ymm1
-; CHECK-NEXT:    vmovaps %ymm1, (%rdi)
-; CHECK-NEXT:    vmovaps %ymm0, 32(%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = sitofp <16 x i1> %mask to <16 x float>
-  store <16 x float> %1, <16 x float>* %res
-  ret void
-}
-
-define void @sbto16f32_512(<16 x i16> %a, <16 x float>* %res) "required-vector-width"="512" {
-; CHECK-LABEL: sbto16f32_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    vpmovm2d %k0, %zmm0
-; CHECK-NEXT:    vcvtdq2ps %zmm0, %zmm0
-; CHECK-NEXT:    vmovaps %zmm0, (%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = sitofp <16 x i1> %mask to <16 x float>
-  store <16 x float> %1, <16 x float>* %res
-  ret void
-}
-
-define void @sbto16f64_256(<16 x i16> %a, <16 x double>* %res)  "required-vector-width"="256" {
-; CHECK-LABEL: sbto16f64_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    kshiftrw $8, %k0, %k1
-; CHECK-NEXT:    vpmovm2d %k1, %ymm0
-; CHECK-NEXT:    vcvtdq2pd %xmm0, %ymm1
-; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm0
-; CHECK-NEXT:    vcvtdq2pd %xmm0, %ymm0
-; CHECK-NEXT:    vpmovm2d %k0, %ymm2
-; CHECK-NEXT:    vcvtdq2pd %xmm2, %ymm3
-; CHECK-NEXT:    vextracti128 $1, %ymm2, %xmm2
-; CHECK-NEXT:    vcvtdq2pd %xmm2, %ymm2
-; CHECK-NEXT:    vmovaps %ymm2, 32(%rdi)
-; CHECK-NEXT:    vmovaps %ymm3, (%rdi)
-; CHECK-NEXT:    vmovaps %ymm0, 96(%rdi)
-; CHECK-NEXT:    vmovaps %ymm1, 64(%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = sitofp <16 x i1> %mask to <16 x double>
-  store <16 x double> %1, <16 x double>* %res
-  ret void
-}
-
-define void @sbto16f64_512(<16 x i16> %a, <16 x double>* %res)  "required-vector-width"="512" {
-; CHECK-LABEL: sbto16f64_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    vpmovm2d %k0, %zmm0
-; CHECK-NEXT:    vcvtdq2pd %ymm0, %zmm1
-; CHECK-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
-; CHECK-NEXT:    vcvtdq2pd %ymm0, %zmm0
-; CHECK-NEXT:    vmovaps %zmm0, 64(%rdi)
-; CHECK-NEXT:    vmovaps %zmm1, (%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = sitofp <16 x i1> %mask to <16 x double>
-  store <16 x double> %1, <16 x double>* %res
-  ret void
-}
-
-define void @ubto16f32_256(<16 x i16> %a, <16 x float>* %res) "required-vector-width"="256" {
-; CHECK-LABEL: ubto16f32_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    kshiftrw $8, %k0, %k1
-; CHECK-NEXT:    vpmovm2d %k1, %ymm0
-; CHECK-NEXT:    vpsrld $31, %ymm0, %ymm0
-; CHECK-NEXT:    vcvtdq2ps %ymm0, %ymm0
-; CHECK-NEXT:    vpmovm2d %k0, %ymm1
-; CHECK-NEXT:    vpsrld $31, %ymm1, %ymm1
-; CHECK-NEXT:    vcvtdq2ps %ymm1, %ymm1
-; CHECK-NEXT:    vmovaps %ymm1, (%rdi)
-; CHECK-NEXT:    vmovaps %ymm0, 32(%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = uitofp <16 x i1> %mask to <16 x float>
-  store <16 x float> %1, <16 x float>* %res
-  ret void
-}
-
-define void @ubto16f32_512(<16 x i16> %a, <16 x float>* %res) "required-vector-width"="512" {
-; CHECK-LABEL: ubto16f32_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    vpmovm2d %k0, %zmm0
-; CHECK-NEXT:    vpsrld $31, %zmm0, %zmm0
-; CHECK-NEXT:    vcvtdq2ps %zmm0, %zmm0
-; CHECK-NEXT:    vmovaps %zmm0, (%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = uitofp <16 x i1> %mask to <16 x float>
-  store <16 x float> %1, <16 x float>* %res
-  ret void
-}
-
-define void @ubto16f64_256(<16 x i16> %a, <16 x double>* %res) "required-vector-width"="256" {
-; CHECK-LABEL: ubto16f64_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    kshiftrw $8, %k0, %k1
-; CHECK-NEXT:    vpmovm2d %k1, %ymm0
-; CHECK-NEXT:    vpsrld $31, %ymm0, %ymm0
-; CHECK-NEXT:    vcvtdq2pd %xmm0, %ymm1
-; CHECK-NEXT:    vextracti128 $1, %ymm0, %xmm0
-; CHECK-NEXT:    vcvtdq2pd %xmm0, %ymm0
-; CHECK-NEXT:    vpmovm2d %k0, %ymm2
-; CHECK-NEXT:    vpsrld $31, %ymm2, %ymm2
-; CHECK-NEXT:    vcvtdq2pd %xmm2, %ymm3
-; CHECK-NEXT:    vextracti128 $1, %ymm2, %xmm2
-; CHECK-NEXT:    vcvtdq2pd %xmm2, %ymm2
-; CHECK-NEXT:    vmovaps %ymm2, 32(%rdi)
-; CHECK-NEXT:    vmovaps %ymm3, (%rdi)
-; CHECK-NEXT:    vmovaps %ymm0, 96(%rdi)
-; CHECK-NEXT:    vmovaps %ymm1, 64(%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = uitofp <16 x i1> %mask to <16 x double>
-  store <16 x double> %1, <16 x double>* %res
-  ret void
-}
-
-define void @ubto16f64_512(<16 x i16> %a, <16 x double>* %res) "required-vector-width"="512" {
-; CHECK-LABEL: ubto16f64_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vpmovw2m %ymm0, %k0
-; CHECK-NEXT:    vpmovm2d %k0, %zmm0
-; CHECK-NEXT:    vpsrld $31, %zmm0, %zmm0
-; CHECK-NEXT:    vcvtdq2pd %ymm0, %zmm1
-; CHECK-NEXT:    vextracti64x4 $1, %zmm0, %ymm0
-; CHECK-NEXT:    vcvtdq2pd %ymm0, %zmm0
-; CHECK-NEXT:    vmovaps %zmm0, 64(%rdi)
-; CHECK-NEXT:    vmovaps %zmm1, (%rdi)
-; CHECK-NEXT:    vzeroupper
-; CHECK-NEXT:    retq
-  %mask = icmp slt <16 x i16> %a, zeroinitializer
-  %1 = uitofp <16 x i1> %mask to <16 x double>
-  store <16 x double> %1, <16 x double>* %res
-  ret void
-}
-
-define <16 x i16> @test_16f32toub_256(<16 x float>* %ptr, <16 x i16> %passthru) "required-vector-width"="256" {
-; CHECK-LABEL: test_16f32toub_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcvttps2dq (%rdi), %ymm1
-; CHECK-NEXT:    vpmovdw %ymm1, %xmm1
-; CHECK-NEXT:    vcvttps2dq 32(%rdi), %ymm2
-; CHECK-NEXT:    vpmovdw %ymm2, %xmm2
-; CHECK-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; CHECK-NEXT:    vpsllw $15, %ymm1, %ymm1
-; CHECK-NEXT:    vpmovw2m %ymm1, %k1
-; CHECK-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
-; CHECK-NEXT:    retq
-  %a = load <16 x float>, <16 x float>* %ptr
-  %mask = fptoui <16 x float> %a to <16 x i1>
-  %select = select <16 x i1> %mask, <16 x i16> %passthru, <16 x i16> zeroinitializer
-  ret <16 x i16> %select
-}
-
-define <16 x i16> @test_16f32toub_512(<16 x float>* %ptr, <16 x i16> %passthru) "required-vector-width"="512" {
-; CHECK-LABEL: test_16f32toub_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcvttps2dq (%rdi), %zmm1
-; CHECK-NEXT:    vpslld $31, %zmm1, %zmm1
-; CHECK-NEXT:    vpmovd2m %zmm1, %k1
-; CHECK-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
-; CHECK-NEXT:    retq
-  %a = load <16 x float>, <16 x float>* %ptr
-  %mask = fptoui <16 x float> %a to <16 x i1>
-  %select = select <16 x i1> %mask, <16 x i16> %passthru, <16 x i16> zeroinitializer
-  ret <16 x i16> %select
-}
-
-define <16 x i16> @test_16f32tosb_256(<16 x float>* %ptr, <16 x i16> %passthru) "required-vector-width"="256" {
-; CHECK-LABEL: test_16f32tosb_256:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcvttps2dq (%rdi), %ymm1
-; CHECK-NEXT:    vpmovdw %ymm1, %xmm1
-; CHECK-NEXT:    vcvttps2dq 32(%rdi), %ymm2
-; CHECK-NEXT:    vpmovdw %ymm2, %xmm2
-; CHECK-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; CHECK-NEXT:    vpsllw $15, %ymm1, %ymm1
-; CHECK-NEXT:    vpmovw2m %ymm1, %k1
-; CHECK-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
-; CHECK-NEXT:    retq
-  %a = load <16 x float>, <16 x float>* %ptr
-  %mask = fptosi <16 x float> %a to <16 x i1>
-  %select = select <16 x i1> %mask, <16 x i16> %passthru, <16 x i16> zeroinitializer
-  ret <16 x i16> %select
-}
-
-define <16 x i16> @test_16f32tosb_512(<16 x float>* %ptr, <16 x i16> %passthru) "required-vector-width"="512" {
-; CHECK-LABEL: test_16f32tosb_512:
-; CHECK:       # %bb.0:
-; CHECK-NEXT:    vcvttps2dq (%rdi), %zmm1
-; CHECK-NEXT:    vpmovd2m %zmm1, %k1
-; CHECK-NEXT:    vmovdqu16 %ymm0, %ymm0 {%k1} {z}
-; CHECK-NEXT:    retq
-  %a = load <16 x float>, <16 x float>* %ptr
-  %mask = fptosi <16 x float> %a to <16 x i1>
-  %select = select <16 x i1> %mask, <16 x i16> %passthru, <16 x i16> zeroinitializer
-  ret <16 x i16> %select
-}