[X86] Support Intel avxvnni

[LiuChen3]

Summary

This patch mainly made the following changes:

1. Support AVX-VNNI instructions;
2. Support more complex FeatureList. By default, ',' is prior than '|'. And parentheses is supported;
3. Introduce ExplicitVEXPrefix flag so that vpdpbusd/vpdpbusds/vpdpbusds/vpdpbusds instructions only use vex-encoding when user explicity add {vex} prefix.

[RKSimon]

Please can you add an entry to the 12.0 release notes?

clang/include/clang/Basic/BuiltinsNVPTX.def

46 ↗

(On Diff #297148)

Pull this out? This and the CodeGenFunction.* changes seem to be worth a separate patch

clang/lib/Headers/avxvnniintrin.h

32

Is having a commonvnniintrin.h header the approach gcc/icc are taking?

40

Please can you add doxygen descriptions to each intrinsic.

[LiuChen3]

Address comments.

[LiuChen3]

clang/include/clang/Basic/BuiltinsNVPTX.def

46 ↗

(On Diff #297148)

Thanks for your review.
I will separate this part. I want to keep this part here for now until another patch is merged in.

clang/lib/Headers/avxvnniintrin.h

32

GCC doesn't have commonvnniintrin.h headr, they only have avxvnniintrin.h. For ICC, I will check with this and give you answer later.

[LiuChen3]

clang/lib/Headers/avxvnniintrin.h

32

GCC and ICC doesn't break intrinsics into multiple headers.

[LiuChen3]

Rebase.

[RKSimon]

clang/lib/Headers/avxvnniintrin.h

32

Is there anyway that we can avoid this as well then? I get worried when header layouts start to diverge...

[craig.topper]

We also need to add avxvnni versions of the patterns from this commit along with appropriate testing. Maybe just split the 128/256 bit out of avx512vnni.ll and into a separate test with multiple RUN lines. That should all be done in a separate patch.

{code}
commit cc4b0596b1b0e58672e4151396c7b804eccaf273
Author: Craig Topper <[e-mail]>
Date: Sat Aug 24 23:14:57 2019 +0000

[X86] Add isel patterns to match vpdpwssd avx512vnni instruction from add+pmaddwd nodes.

{code}

[pengfei]

clang/lib/Headers/avxvnniintrin.h

32

Hi Simon, why we need to keep the same layout with GCC and ICC?
We always create new header files for new instructions. We don't allow user include these headers directly. So I think it doesn't matter if GCC and ICC create the same headers or not.

[craig.topper]

clang/lib/Headers/avxvnniintrin.h

32

As far as I know our header layout is largely the same as gcc for existing intrinsics. Can we just keep them the same?

clang/lib/Headers/commonvnniintrin.h

48 ↗

(On Diff #298920)

Arguments to macros shouldn't start with __

49 ↗

(On Diff #298920)

Because this is a macro this needs to have parentheses around S, A, and B.

[pengfei]

clang/lib/Headers/avxvnniintrin.h

33

Maybe we can move commonvnniintrin.h to immintrin.h. Then we don't have a different layout.

#if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__AVX512VNNI__) || (defined(__AVX512VL__) && defined(__AVX512VNNI__))
#include <commonvnniintrin.h>
#endif

[craig.topper]

clang/lib/Headers/avxvnniintrin.h

33

gcc appears to have just put the common intrinsics in avxvnniintrin.h. Why can't we do that?

[pengfei]

clang/lib/Headers/avxvnniintrin.h

33

I asked GCC guys. They don't use macro like defined(__AVXVNNI__). If we put it in avxvnniintrin.h, we need handle macro AVX512VNNI in avxvnniintrin.h. It's bit confused.

@LiuChen3 I meant __AVXVNNI__ for the first __AVX512VNNI__ in last comment.

[pengfei]

clang/lib/Headers/avxvnniintrin.h

33

Or maybe we can put the common intrinsics in avx512vlvnniintrin.h and not to handle macro __AVXVNNI__ int it.
I think it's reasonable. For Linux, we always include all headers. For Windows, we can only allow user use the _avx_ intrinscis for AVXVNNI, which is compatible with MSVC.

[craig.topper]

clang/lib/Headers/avxvnniintrin.h

33

Minor clarification. It's not Linux vs Windows. Its -ms-compatibility that controls it. You can use clang on Windows without enabling MSVC compatibility. In which case MSC_VER won't be defined.

[pengfei]

clang/lib/Headers/avxvnniintrin.h

33

Right. Thank @craig.topper

[RKSimon]

clang/test/CodeGen/avxvnni-builtins.c

1 ↗

(On Diff #298920)

Please can you move this test file under the X86 subdirectory?

[LiuChen3]

1. move the commonvnniintrin.h to the avx512vlvnniintrin.h.
2. move the testcase avxvnni-builtins.c to X86 subdirectory.

[craig.topper]

llvm/lib/Support/X86TargetParser.cpp

211

Add here too.

llvm/lib/Target/X86/X86.td

775

Please add this to Alderlake which was added to X86.td by @bkramer yesterday

[craig.topper]

Also need clang/test/Preprocessor/predefined-arch-macros.c updates for sapphirerapids and alderlake to check that AVXVNNI is defined.

[LiuChen3]

Adding avxvnni to Alder Lake.

[pengfei]

clang/test/Preprocessor/predefined-arch-macros.c

1637

You need to add to sapphirerapids too.

1708

You need to add to sapphirerapids too.

llvm/docs/ReleaseNotes.rst

124

You also need to add it to Clang release notes.

[LiuChen3]

Address comments.

[pengfei]

clang/lib/Headers/avxvnniintrin.h

31

I think we may still need add some information for intrinsics that can be used for AVXVNNI but not declared here. A proposal might be:

/* Below intrinsics defined in avx512vlvnniintrin.h can be used for */
/* AVXVNNI except used for compatibility with msvc.                 */
// #define _mm256_dpbusd_epi32(S, A, B)
// #define _mm256_dpbusds_epi32(S, A, B)
// #define _mm256_dpwssd_epi32(S, A, B)
...

[pengfei]

clang/lib/Headers/avxvnniintrin.h

31

I had a look at the Doxygen grammar. Maybe we can use like below:

/* Below intrinsics defined in avx512vlvnniintrin.h can be used for */
/* AVXVNNI except used for compatibility with msvc.                 */
/// \fn __m256i _mm256_dpbusd_epi32(__m256i __S, __m256i __A, __m256i __B)
/// \fn __m256i _mm256_dpbusds_epi32(__m256i __S, __m256i __A, __m256i __B)
/// \fn __m256i _mm256_dpwssd_epi32(__m256i __S, __m256i __A, __m256i __B)
...

[pengfei]

clang/lib/Headers/avxvnniintrin.h

31

Missing one line comment?

clang/lib/Headers/immintrin.h

144

Change back the spaces.

llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp

3851

Missing check for VEX2?

[LiuChen3]

Address comments

[pengfei]

LGTM with little nit:

llvm/test/CodeGen/X86/avx-vnni/avx_vnni-intrinsics.ll

1

Do we need new directory for the tests?

3

I don't see difference between AVX-X86 and AVX-X64, maybe you can use AVX for both, or AVX,AVX-X86 and AVX,AVX-X64 if they do have some differences. And remove CHECK since it never been used.

[LiuChen3]

1. Move the testcase from avx-vnni/ to test/CodeGen/X86/.
2. Refine the Run line in avx_vnni-intrinsics.ll

[pengfei]

LGTM. Thanks. Better to wait one day or two to see if others object.

[LiuChen3]

In D89105#2363927, @pengfei wrote:

LGTM. Thanks. Better to wait one day or two to see if others object.

Sure. Thanks for your review.

[RKSimon]

LGTM cheers

[craig.topper]

LGTM to me with a couple minors that can be fixed on commit.

clang/test/Preprocessor/predefined-arch-macros.c

1828

doesn't this NOT need to be above AVXNNI to be effective since that's where AVX512 would go alphabetically?

llvm/lib/Target/X86/MCTargetDesc/X86InstPrinterCommon.cpp

351

Put parentheses around the & parts. Similar to lock and notrack above. I don't want to think about operator precedence.

[LiuChen3]

Thanks for all of your review!

[RKSimon]

clang/lib/Headers/cpuid.h

199

@LiuChen3 https://github.com/llvm/llvm-project/issues/53392 reports this should be 0x00000010?

[craig.topper]

clang/lib/Headers/cpuid.h

199

Maybe we should start writing these as 1 << 4. This is not the first shifted by one issue we've had in this file.

[LiuChen3]

clang/lib/Headers/cpuid.h

199

It my fault. I have a fix here: https://reviews.llvm.org/D118103 .

clang/docs/ClangCommandLineReference.rst

 .. option:: -mavx512vl, -mno-avx512vl
 
 .. option:: -mavx512vnni, -mno-avx512vnni
 
 .. option:: -mavx512vp2intersect, -mno-avx512vp2intersect
 
 .. option:: -mavx512vpopcntdq, -mno-avx512vpopcntdq
 
+.. option:: -mavxvnni, -mno-avxvnni
+
 .. option:: -mbmi, -mno-bmi
 
 .. option:: -mbmi2, -mno-bmi2
 
 .. option:: -mcldemote, -mno-cldemote
 
 .. option:: -mclflushopt, -mno-clflushopt
 

clang/docs/ReleaseNotes.rst

- The -mtune command line option is no longer ignored for X86. This can be used
   to request microarchitectural optimizations independent on -march. -march=<cpu>
   implies -mtune=<cpu>. -mtune=generic is the default with no -march or -mtune
   specified.
 
 - Support for ``HRESET`` instructions has been added.
 
 - Support for ``UINTR`` instructions has been added.
 
+- Support for ``AVXVNNI`` instructions has been added.
+
 Internal API Changes
 --------------------
 
 These are major API changes that have happened since the 11.0.0 release of
 Clang. If upgrading an external codebase that uses Clang as a library,
 this section should help get you past the largest hurdles of upgrading.
 
 - ...

clang/include/clang/Basic/BuiltinsX86.def

 TARGET_BUILTIN(__builtin_ia32_alignd512, "V16iV16iV16iIi", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_alignd128, "V4iV4iV4iIi", "ncV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_alignd256, "V8iV8iV8iIi", "ncV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_alignq128, "V2OiV2OiV2OiIi", "ncV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_alignq256, "V4OiV4OiV4OiIi", "ncV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_extractf64x4_mask, "V4dV8dIiV4dUc", "ncV:512:", "avx512f")
 TARGET_BUILTIN(__builtin_ia32_extractf32x4_mask, "V4fV16fIiV4fUc", "ncV:512:", "avx512f")
 
-TARGET_BUILTIN(__builtin_ia32_vpdpbusd128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpbusd256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpbusd128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni|avxvnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpbusd256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni|avxvnni")
 TARGET_BUILTIN(__builtin_ia32_vpdpbusd512, "V16iV16iV16iV16i", "ncV:512:", "avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpbusds128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpbusds256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpbusds128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni|avxvnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpbusds256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni|avxvnni")
 TARGET_BUILTIN(__builtin_ia32_vpdpbusds512, "V16iV16iV16iV16i", "ncV:512:", "avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpwssd128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpwssd256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpwssd128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni|avxvnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpwssd256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni|avxvnni")
 TARGET_BUILTIN(__builtin_ia32_vpdpwssd512, "V16iV16iV16iV16i", "ncV:512:", "avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpwssds128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni")
-TARGET_BUILTIN(__builtin_ia32_vpdpwssds256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpwssds128, "V4iV4iV4iV4i", "ncV:128:", "avx512vl,avx512vnni|avxvnni")
+TARGET_BUILTIN(__builtin_ia32_vpdpwssds256, "V8iV8iV8iV8i", "ncV:256:", "avx512vl,avx512vnni|avxvnni")
 TARGET_BUILTIN(__builtin_ia32_vpdpwssds512, "V16iV16iV16iV16i", "ncV:512:", "avx512vnni")
 
 TARGET_BUILTIN(__builtin_ia32_gather3div2df, "V2dV2dvC*V2OiUcIi", "nV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_gather3div2di, "V2OiV2OivC*V2OiUcIi", "nV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_gather3div4df, "V4dV4dvC*V4OiUcIi", "nV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_gather3div4di, "V4OiV4OivC*V4OiUcIi", "nV:256:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_gather3div4sf, "V4fV4fvC*V2OiUcIi", "nV:128:", "avx512vl")
 TARGET_BUILTIN(__builtin_ia32_gather3div4si, "V4iV4ivC*V2OiUcIi", "nV:128:", "avx512vl")

clang/include/clang/Driver/Options.td

 def mavx512vl : Flag<["-"], "mavx512vl">, Group<m_x86_Features_Group>;
 def mno_avx512vl : Flag<["-"], "mno-avx512vl">, Group<m_x86_Features_Group>;
 def mavx512vnni : Flag<["-"], "mavx512vnni">, Group<m_x86_Features_Group>;
 def mno_avx512vnni : Flag<["-"], "mno-avx512vnni">, Group<m_x86_Features_Group>;
 def mavx512vpopcntdq : Flag<["-"], "mavx512vpopcntdq">, Group<m_x86_Features_Group>;
 def mno_avx512vpopcntdq : Flag<["-"], "mno-avx512vpopcntdq">, Group<m_x86_Features_Group>;
 def mavx512vp2intersect : Flag<["-"], "mavx512vp2intersect">, Group<m_x86_Features_Group>;
 def mno_avx512vp2intersect : Flag<["-"], "mno-avx512vp2intersect">, Group<m_x86_Features_Group>;
+def mavxvnni : Flag<["-"], "mavxvnni">, Group<m_x86_Features_Group>;
+def mno_avxvnni : Flag<["-"], "mno-avxvnni">, Group<m_x86_Features_Group>;
 def madx : Flag<["-"], "madx">, Group<m_x86_Features_Group>;
 def mno_adx : Flag<["-"], "mno-adx">, Group<m_x86_Features_Group>;
 def maes : Flag<["-"], "maes">, Group<m_x86_Features_Group>;
 def mno_aes : Flag<["-"], "mno-aes">, Group<m_x86_Features_Group>;
 def mbmi : Flag<["-"], "mbmi">, Group<m_x86_Features_Group>;
 def mno_bmi : Flag<["-"], "mno-bmi">, Group<m_x86_Features_Group>;
 def mbmi2 : Flag<["-"], "mbmi2">, Group<m_x86_Features_Group>;
 def mno_bmi2 : Flag<["-"], "mno-bmi2">, Group<m_x86_Features_Group>;

clang/lib/Basic/Targets/X86.h

class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo {
   bool HasMOVDIRI = false;
   bool HasMOVDIR64B = false;
   bool HasPTWRITE = false;
   bool HasINVPCID = false;
   bool HasENQCMD = false;
   bool HasKL = false;      // For key locker
   bool HasWIDEKL = false; // For wide key locker
   bool HasHRESET = false;
+  bool HasAVXVNNI = false;
   bool HasAMXTILE = false;
   bool HasAMXINT8 = false;
   bool HasAMXBF16 = false;
   bool HasSERIALIZE = false;
   bool HasTSXLDTRK = false;
   bool HasUINTR = false;
 
 protected:

clang/lib/Basic/Targets/X86.cpp

for (const auto &Feature : Features) {
     } else if (Feature == "+hreset") {
       HasHRESET = true;
     } else if (Feature == "+amx-bf16") {
       HasAMXBF16 = true;
     } else if (Feature == "+amx-int8") {
       HasAMXINT8 = true;
     } else if (Feature == "+amx-tile") {
       HasAMXTILE = true;
+    } else if (Feature == "+avxvnni") {
+      HasAVXVNNI = true;
     } else if (Feature == "+serialize") {
       HasSERIALIZE = true;
     } else if (Feature == "+tsxldtrk") {
       HasTSXLDTRK = true;
     } else if (Feature == "+uintr") {
       HasUINTR = true;
     }
 
void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
   if (HasHRESET)
     Builder.defineMacro("__HRESET__");
   if (HasAMXTILE)
     Builder.defineMacro("__AMXTILE__");
   if (HasAMXINT8)
     Builder.defineMacro("__AMXINT8__");
   if (HasAMXBF16)
     Builder.defineMacro("__AMXBF16__");
+  if (HasAVXVNNI)
+    Builder.defineMacro("__AVXVNNI__");
   if (HasSERIALIZE)
     Builder.defineMacro("__SERIALIZE__");
   if (HasTSXLDTRK)
     Builder.defineMacro("__TSXLDTRK__");
   if (HasUINTR)
     Builder.defineMacro("__UINTR__");
 
   // Each case falls through to the previous one here.
return llvm::StringSwitch<bool>(Name)
       .Case("avx512dq", true)
       .Case("avx512bitalg", true)
       .Case("avx512bw", true)
       .Case("avx512vl", true)
       .Case("avx512vbmi", true)
       .Case("avx512vbmi2", true)
       .Case("avx512ifma", true)
       .Case("avx512vp2intersect", true)
+      .Case("avxvnni", true)
       .Case("bmi", true)
       .Case("bmi2", true)
       .Case("cldemote", true)
       .Case("clflushopt", true)
       .Case("clwb", true)
       .Case("clzero", true)
       .Case("cx16", true)
       .Case("enqcmd", true)

 
 bool X86TargetInfo::hasFeature(StringRef Feature) const {
   return llvm::StringSwitch<bool>(Feature)
       .Case("adx", HasADX)
       .Case("aes", HasAES)
       .Case("amx-bf16", HasAMXBF16)
       .Case("amx-int8", HasAMXINT8)
       .Case("amx-tile", HasAMXTILE)
+      .Case("avxvnni", HasAVXVNNI)
       .Case("avx", SSELevel >= AVX)
       .Case("avx2", SSELevel >= AVX2)
       .Case("avx512f", SSELevel >= AVX512F)
       .Case("avx512cd", HasAVX512CD)
       .Case("avx512vpopcntdq", HasAVX512VPOPCNTDQ)
       .Case("avx512vnni", HasAVX512VNNI)
       .Case("avx512bf16", HasAVX512BF16)
       .Case("avx512er", HasAVX512ER)

clang/lib/Headers/CMakeLists.txt

set(files
   avx512vldqintrin.h
   avx512vlintrin.h
   avx512vp2intersectintrin.h
   avx512vlvp2intersectintrin.h
   avx512vpopcntdqvlintrin.h
   avx512vnniintrin.h
   avx512vlvnniintrin.h
   avxintrin.h
+  avxvnniintrin.h
   bmi2intrin.h
   bmiintrin.h
   __clang_cuda_builtin_vars.h
   __clang_cuda_math.h
   __clang_cuda_cmath.h
   __clang_cuda_complex_builtins.h
   __clang_cuda_device_functions.h
   __clang_cuda_intrinsics.h

clang/lib/Headers/avx512vlvnniintrin.h

 
 #ifndef __AVX512VLVNNIINTRIN_H
 #define __AVX512VLVNNIINTRIN_H
 
 /* Define the default attributes for the functions in this file. */
 #define __DEFAULT_FN_ATTRS128 __attribute__((__always_inline__, __nodebug__, __target__("avx512vl,avx512vnni"), __min_vector_width__(128)))
 #define __DEFAULT_FN_ATTRS256 __attribute__((__always_inline__, __nodebug__, __target__("avx512vl,avx512vnni"), __min_vector_width__(256)))
 
-
-static __inline__ __m256i __DEFAULT_FN_ATTRS256
-_mm256_dpbusd_epi32(__m256i __S, __m256i __A, __m256i __B)
-{
-  return (__m256i)__builtin_ia32_vpdpbusd256((__v8si)__S, (__v8si)__A,
-                                             (__v8si)__B);
-}
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a A with
+/// corresponding signed 8-bit integers in \a B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a S, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.word := Signed(ZeroExtend16(A.byte[4*j]) * SignExtend16(B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(A.byte[4*j+1]) * SignExtend16(B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(A.byte[4*j+2]) * SignExtend16(B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(A.byte[4*j+3]) * SignExtend16(B.byte[4*j+3]))
+///      DST.dword[j] := S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+#define _mm256_dpbusd_epi32(S, A, B) \
+  (__m256i)__builtin_ia32_vpdpbusd256((__v8si)(S), (__v8si)(A), (__v8si)(B))
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a A with
+/// corresponding signed 8-bit integers in \a B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a S using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.word := Signed(ZeroExtend16(A.byte[4*j]) * SignExtend16(B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(A.byte[4*j+1]) * SignExtend16(B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(A.byte[4*j+2]) * SignExtend16(B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(A.byte[4*j+3]) * SignExtend16(B.byte[4*j+3]))
+///      DST.dword[j] := Saturate32(S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4)
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+#define _mm256_dpbusds_epi32(S, A, B) \
+  (__m256i)__builtin_ia32_vpdpbusds256((__v8si)(S), (__v8si)(A), (__v8si)(B))
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a A with
+/// corresponding 16-bit integers in \a B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a S,
+///  and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.dword := SignExtend32(A.word[2*j]) * SignExtend32(B.word[2*j])
+///      tmp2.dword := SignExtend32(A.word[2*j+1]) * SignExtend32(B.word[2*j+1])
+///      DST.dword[j] := S.dword[j] + tmp1 + tmp2
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+#define _mm256_dpwssd_epi32(S, A, B) \
+  (__m256i)__builtin_ia32_vpdpwssd256((__v8si)(S), (__v8si)(A), (__v8si)(B))
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a A with
+/// corresponding 16-bit integers in \a B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a S
+/// using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.dword := SignExtend32(A.word[2*j]) * SignExtend32(B.word[2*j])
+///      tmp2.dword := SignExtend32(A.word[2*j+1]) * SignExtend32(B.word[2*j+1])
+///      DST.dword[j] := Saturate32(S.dword[j] + tmp1 + tmp2)
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+#define _mm256_dpwssds_epi32(S, A, B) \
+  (__m256i)__builtin_ia32_vpdpwssds256((__v8si)(S), (__v8si)(A), (__v8si)(B))
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a A with
+/// corresponding signed 8-bit integers in \a B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a S, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.word := Signed(ZeroExtend16(A.byte[4*j]) * SignExtend16(B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(A.byte[4*j+1]) * SignExtend16(B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(A.byte[4*j+2]) * SignExtend16(B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(A.byte[4*j+3]) * SignExtend16(B.byte[4*j+3]))
+///      DST.dword[j] := S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+#define _mm_dpbusd_epi32(S, A, B) \
+  (__m128i)__builtin_ia32_vpdpbusd128((__v4si)(S), (__v4si)(A), (__v4si)(B))
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a A with
+/// corresponding signed 8-bit integers in \a B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a S using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.word := Signed(ZeroExtend16(A.byte[4*j]) * SignExtend16(B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(A.byte[4*j+1]) * SignExtend16(B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(A.byte[4*j+2]) * SignExtend16(B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(A.byte[4*j+3]) * SignExtend16(B.byte[4*j+3]))
+///      DST.dword[j] := Saturate32(S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4)
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+#define _mm_dpbusds_epi32(S, A, B) \
+  (__m128i)__builtin_ia32_vpdpbusds128((__v4si)(S), (__v4si)(A), (__v4si)(B))
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a A with
+/// corresponding 16-bit integers in \a B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a S,
+/// and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.dword := SignExtend32(A.word[2*j]) * SignExtend32(B.word[2*j])
+///      tmp2.dword := SignExtend32(A.word[2*j+1]) * SignExtend32(B.word[2*j+1])
+///      DST.dword[j] := S.dword[j] + tmp1 + tmp2
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+#define _mm_dpwssd_epi32(S, A, B) \
+  (__m128i)__builtin_ia32_vpdpwssd128((__v4si)(S), (__v4si)(A), (__v4si)(B))
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a A with
+/// corresponding 16-bit integers in \a B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a S
+/// using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.dword := SignExtend32(A.word[2*j]) * SignExtend32(B.word[2*j])
+///      tmp2.dword := SignExtend32(A.word[2*j+1]) * SignExtend32(B.word[2*j+1])
+///      DST.dword[j] := Saturate32(S.dword[j] + tmp1 + tmp2)
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+#define _mm_dpwssds_epi32(S, A, B) \
+  (__m128i)__builtin_ia32_vpdpwssds128((__v4si)(S), (__v4si)(A), (__v4si)(B))
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_mask_dpbusd_epi32(__m256i __S, __mmask8 __U, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                      (__v8si)_mm256_dpbusd_epi32(__S, __A, __B),
                                      (__v8si)__S);
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_maskz_dpbusd_epi32(__mmask8 __U, __m256i __S, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                      (__v8si)_mm256_dpbusd_epi32(__S, __A, __B),
                                      (__v8si)_mm256_setzero_si256());
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
-_mm256_dpbusds_epi32(__m256i __S, __m256i __A, __m256i __B)
-{
-  return (__m256i)__builtin_ia32_vpdpbusds256((__v8si)__S, (__v8si)__A,
-                                              (__v8si)__B);
-}
-
-static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_mask_dpbusds_epi32(__m256i __S, __mmask8 __U, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                     (__v8si)_mm256_dpbusds_epi32(__S, __A, __B),
                                     (__v8si)__S);
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_maskz_dpbusds_epi32(__mmask8 __U, __m256i __S, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                      (__v8si)_mm256_dpbusds_epi32(__S, __A, __B),
                                      (__v8si)_mm256_setzero_si256());
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
-_mm256_dpwssd_epi32(__m256i __S, __m256i __A, __m256i __B)
-{
-  return (__m256i)__builtin_ia32_vpdpwssd256((__v8si)__S, (__v8si)__A,
-                                             (__v8si)__B);
-}
-
-static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_mask_dpwssd_epi32(__m256i __S, __mmask8 __U, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                      (__v8si)_mm256_dpwssd_epi32(__S, __A, __B),
                                      (__v8si)__S);
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_maskz_dpwssd_epi32(__mmask8 __U, __m256i __S, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                      (__v8si)_mm256_dpwssd_epi32(__S, __A, __B),
                                      (__v8si)_mm256_setzero_si256());
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
-_mm256_dpwssds_epi32(__m256i __S, __m256i __A, __m256i __B)
-{
-  return (__m256i)__builtin_ia32_vpdpwssds256((__v8si)__S, (__v8si)__A,
-                                              (__v8si)__B);
-}
-
-static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_mask_dpwssds_epi32(__m256i __S, __mmask8 __U, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                     (__v8si)_mm256_dpwssds_epi32(__S, __A, __B),
                                     (__v8si)__S);
 }
 
 static __inline__ __m256i __DEFAULT_FN_ATTRS256
 _mm256_maskz_dpwssds_epi32(__mmask8 __U, __m256i __S, __m256i __A, __m256i __B)
 {
   return (__m256i)__builtin_ia32_selectd_256(__U,
                                     (__v8si)_mm256_dpwssds_epi32(__S, __A, __B),
                                     (__v8si)_mm256_setzero_si256());
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
-_mm_dpbusd_epi32(__m128i __S, __m128i __A, __m128i __B)
-{
-  return (__m128i)__builtin_ia32_vpdpbusd128((__v4si)__S, (__v4si)__A,
-                                             (__v4si)__B);
-}
-
-static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_mask_dpbusd_epi32(__m128i __S, __mmask8 __U, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                         (__v4si)_mm_dpbusd_epi32(__S, __A, __B),
                                         (__v4si)__S);
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_maskz_dpbusd_epi32(__mmask8 __U, __m128i __S, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                         (__v4si)_mm_dpbusd_epi32(__S, __A, __B),
                                         (__v4si)_mm_setzero_si128());
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
-_mm_dpbusds_epi32(__m128i __S, __m128i __A, __m128i __B)
-{
-  return (__m128i)__builtin_ia32_vpdpbusds128((__v4si)__S, (__v4si)__A,
-                                              (__v4si)__B);
-}
-
-static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_mask_dpbusds_epi32(__m128i __S, __mmask8 __U, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                        (__v4si)_mm_dpbusds_epi32(__S, __A, __B),
                                        (__v4si)__S);
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_maskz_dpbusds_epi32(__mmask8 __U, __m128i __S, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                        (__v4si)_mm_dpbusds_epi32(__S, __A, __B),
                                        (__v4si)_mm_setzero_si128());
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
-_mm_dpwssd_epi32(__m128i __S, __m128i __A, __m128i __B)
-{
-  return (__m128i)__builtin_ia32_vpdpwssd128((__v4si)__S, (__v4si)__A,
-                                             (__v4si)__B);
-}
-
-static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_mask_dpwssd_epi32(__m128i __S, __mmask8 __U, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                         (__v4si)_mm_dpwssd_epi32(__S, __A, __B),
                                         (__v4si)__S);
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_maskz_dpwssd_epi32(__mmask8 __U, __m128i __S, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                         (__v4si)_mm_dpwssd_epi32(__S, __A, __B),
                                         (__v4si)_mm_setzero_si128());
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128
-_mm_dpwssds_epi32(__m128i __S, __m128i __A, __m128i __B)
-{
-  return (__m128i)__builtin_ia32_vpdpwssds128((__v4si)__S, (__v4si)__A,
-                                              (__v4si)__B);
-}
-
-static __inline__ __m128i __DEFAULT_FN_ATTRS128
 _mm_mask_dpwssds_epi32(__m128i __S, __mmask8 __U, __m128i __A, __m128i __B)
 {
   return (__m128i)__builtin_ia32_selectd_128(__U,
                                        (__v4si)_mm_dpwssds_epi32(__S, __A, __B),
                                        (__v4si)__S);
 }
 
 static __inline__ __m128i __DEFAULT_FN_ATTRS128

clang/lib/Headers/avxvnniintrin.h

+/*===--------------- avxvnniintrin.h - VNNI intrinsics --------------------===
+ *
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ *
+ *===-----------------------------------------------------------------------===
+ */
+#ifndef __IMMINTRIN_H
+#error "Never use <avxvnniintrin.h> directly; include <immintrin.h> instead."
+#endif
+
+#ifndef __AVXVNNIINTRIN_H
+#define __AVXVNNIINTRIN_H
+
+/* Below intrinsics defined in avx512vlvnniintrin.h can be used for */
+/// \fn __m256i _mm256_dpbusd_epi32(__m256i __S, __m256i __A, __m256i __B)
+/// \fn __m256i _mm256_dpbusds_epi32(__m256i __S, __m256i __A, __m256i __B)
+/// \fn __m256i _mm256_dpwssd_epi32(__m256i __S, __m256i __A, __m256i __B)
+/// \fn __m256i _mm256_dpwssds_epi32(__m256i __S, __m256i __A, __m256i __B)
+/// \fn __m128i _mm_dpbusd_epi32(__m128i __S, __m128i __A, __m128i __B)
+/// \fn __m128i _mm_dpbusds_epi32(__m128i __S, __m128i __A, __m128i __B)
+/// \fn __m128i _mm_dpwssd_epi32(__m128i __S, __m128i __A, __m128i __B)
+/// \fn __m128i _mm_dpwssds_epi32(__m128i __S, __m128i __A, __m128i __B)
+
+/* Intrinsics with _avx_ prefix are for compatibility with msvc. */
+/* Define the default attributes for the functions in this file. */
+#define __DEFAULT_FN_ATTRS256 __attribute__((__always_inline__, __nodebug__, __target__("avxvnni"), __min_vector_width__(256)))
+#define __DEFAULT_FN_ATTRS128 __attribute__((__always_inline__, __nodebug__, __target__("avxvnni"), __min_vector_width__(128)))
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a __A with
+/// corresponding signed 8-bit integers in \a __B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a __S, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.word := Signed(ZeroExtend16(__A.byte[4*j]) * SignExtend16(__B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(__A.byte[4*j+1]) * SignExtend16(__B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(__A.byte[4*j+2]) * SignExtend16(__B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(__A.byte[4*j+3]) * SignExtend16(__B.byte[4*j+3]))
+///      DST.dword[j] := __S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+static __inline__ __m256i __DEFAULT_FN_ATTRS256
+_mm256_dpbusd_avx_epi32(__m256i __S, __m256i __A, __m256i __B)
+{
+  return (__m256i)__builtin_ia32_vpdpbusd256((__v8si)__S, (__v8si)__A, (__v8si)__B);
+}
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a __A with
+/// corresponding signed 8-bit integers in \a __B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a __S using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.word := Signed(ZeroExtend16(__A.byte[4*j]) * SignExtend16(__B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(__A.byte[4*j+1]) * SignExtend16(__B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(__A.byte[4*j+2]) * SignExtend16(__B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(__A.byte[4*j+3]) * SignExtend16(__B.byte[4*j+3]))
+///      DST.dword[j] := Saturate32(__S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4)
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+static __inline__ __m256i __DEFAULT_FN_ATTRS256
+_mm256_dpbusds_avx_epi32(__m256i __S, __m256i __A, __m256i __B)
+{
+  return (__m256i)__builtin_ia32_vpdpbusds256((__v8si)__S, (__v8si)__A, (__v8si)__B);
+}
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a __A with
+/// corresponding 16-bit integers in \a __B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a __S,
+///  and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.dword := SignExtend32(__A.word[2*j]) * SignExtend32(__B.word[2*j])
+///      tmp2.dword := SignExtend32(__A.word[2*j+1]) * SignExtend32(__B.word[2*j+1])
+///      DST.dword[j] := __S.dword[j] + tmp1 + tmp2
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+static __inline__ __m256i __DEFAULT_FN_ATTRS256
+_mm256_dpwssd_avx_epi32(__m256i __S, __m256i __A, __m256i __B)
+{
+  return (__m256i)__builtin_ia32_vpdpwssd256((__v8si)__S, (__v8si)__A, (__v8si)__B);
+}
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a __A with
+/// corresponding 16-bit integers in \a __B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a __S
+/// using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 7
+///      tmp1.dword := SignExtend32(__A.word[2*j]) * SignExtend32(__B.word[2*j])
+///      tmp2.dword := SignExtend32(__A.word[2*j+1]) * SignExtend32(__B.word[2*j+1])
+///      DST.dword[j] := Saturate32(__S.dword[j] + tmp1 + tmp2)
+///    ENDFOR
+///    DST[MAX:256] := 0
+/// \endoperation
+static __inline__ __m256i __DEFAULT_FN_ATTRS256
+_mm256_dpwssds_avx_epi32(__m256i __S, __m256i __A, __m256i __B)
+{
+  return (__m256i)__builtin_ia32_vpdpwssds256((__v8si)__S, (__v8si)__A, (__v8si)__B);
+}
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a __A with
+/// corresponding signed 8-bit integers in \a __B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a __S, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.word := Signed(ZeroExtend16(__A.byte[4*j]) * SignExtend16(__B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(__A.byte[4*j+1]) * SignExtend16(__B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(__A.byte[4*j+2]) * SignExtend16(__B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(__A.byte[4*j+3]) * SignExtend16(__B.byte[4*j+3]))
+///      DST.dword[j] := __S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+static __inline__ __m128i __DEFAULT_FN_ATTRS128
+_mm_dpbusd_avx_epi32(__m128i __S, __m128i __A, __m128i __B)
+{
+  return (__m128i)__builtin_ia32_vpdpbusd128((__v4si)__S, (__v4si)__A, (__v4si)__B);
+}
+
+/// Multiply groups of 4 adjacent pairs of unsigned 8-bit integers in \a __A with
+/// corresponding signed 8-bit integers in \a __B, producing 4 intermediate signed
+/// 16-bit results. Sum these 4 results with the corresponding 32-bit integer
+/// in \a __S using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPBUSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.word := Signed(ZeroExtend16(__A.byte[4*j]) * SignExtend16(__B.byte[4*j]))
+///      tmp2.word := Signed(ZeroExtend16(__A.byte[4*j+1]) * SignExtend16(__B.byte[4*j+1]))
+///      tmp3.word := Signed(ZeroExtend16(__A.byte[4*j+2]) * SignExtend16(__B.byte[4*j+2]))
+///      tmp4.word := Signed(ZeroExtend16(__A.byte[4*j+3]) * SignExtend16(__B.byte[4*j+3]))
+///      DST.dword[j] := Saturate32(__S.dword[j] + tmp1 + tmp2 + tmp3 + tmp4)
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+static __inline__ __m128i __DEFAULT_FN_ATTRS128
+_mm_dpbusds_avx_epi32(__m128i __S, __m128i __A, __m128i __B)
+{
+  return (__m128i)__builtin_ia32_vpdpbusds128((__v4si)__S, (__v4si)__A, (__v4si)__B);
+}
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a __A with
+/// corresponding 16-bit integers in \a __B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a __S,
+/// and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSD </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.dword := SignExtend32(__A.word[2*j]) * SignExtend32(__B.word[2*j])
+///      tmp2.dword := SignExtend32(__A.word[2*j+1]) * SignExtend32(__B.word[2*j+1])
+///      DST.dword[j] := __S.dword[j] + tmp1 + tmp2
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+static __inline__ __m128i __DEFAULT_FN_ATTRS128
+_mm_dpwssd_avx_epi32(__m128i __S, __m128i __A, __m128i __B)
+{
+  return (__m128i)__builtin_ia32_vpdpwssd128((__v4si)__S, (__v4si)__A, (__v4si)__B);
+}
+
+/// Multiply groups of 2 adjacent pairs of signed 16-bit integers in \a __A with
+/// corresponding 16-bit integers in \a __B, producing 2 intermediate signed 32-bit
+/// results. Sum these 2 results with the corresponding 32-bit integer in \a __S
+/// using signed saturation, and store the packed 32-bit results in DST.
+///
+/// This intrinsic corresponds to the <c> VPDPWSSDS </c> instructions.
+///
+/// \operation
+///    FOR j := 0 to 3
+///      tmp1.dword := SignExtend32(__A.word[2*j]) * SignExtend32(__B.word[2*j])
+///      tmp2.dword := SignExtend32(__A.word[2*j+1]) * SignExtend32(__B.word[2*j+1])
+///      DST.dword[j] := Saturate32(__S.dword[j] + tmp1 + tmp2)
+///    ENDFOR
+///    DST[MAX:128] := 0
+/// \endoperation
+static __inline__ __m128i __DEFAULT_FN_ATTRS128
+_mm_dpwssds_avx_epi32(__m128i __S, __m128i __A, __m128i __B)
+{
+  return (__m128i)__builtin_ia32_vpdpwssds128((__v4si)__S, (__v4si)__A, (__v4si)__B);
+}
+
+#undef __DEFAULT_FN_ATTRS128
+#undef __DEFAULT_FN_ATTRS256
+
+#endif // __AVXVNNIINTRIN_H

clang/lib/Headers/cpuid.h

 #define bit_TSXLDTRK      0x00010000
 #define bit_PCONFIG       0x00040000
 #define bit_IBT           0x00100000
 #define bit_AMXBF16       0x00400000
 #define bit_AMXTILE       0x01000000
 #define bit_AMXINT8       0x02000000
 
 /* Features in %eax for leaf 7 sub-leaf 1 */
+#define bit_AVXVNNI       0x00000008
 #define bit_AVX512BF16    0x00000020
 #define bit_HRESET        0x00400000
 
 /* Features in %eax for leaf 13 sub-leaf 1 */
 #define bit_XSAVEOPT    0x00000001
 #define bit_XSAVEC      0x00000002
 #define bit_XSAVES      0x00000008
 

clang/lib/Headers/immintrin.h

 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__AVX512VNNI__)
 #include <avx512vnniintrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
-    (defined(__AVX512VL__) && defined(__AVX512VNNI__))
 #include <avx512vlvnniintrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
+    defined(__AVXVNNI__)
+#include <avxvnniintrin.h>
+#endif
+
+#if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__AVX512DQ__)
 #include <avx512dqintrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     (defined(__AVX512VL__) && defined(__AVX512BITALG__))
 #include <avx512vlbitalgintrin.h>
 #endif

clang/test/CodeGen/X86/avxvnni-builtins.c

+// RUN: %clang_cc1 -ffreestanding %s -triple=x86_64-apple-darwin -target-feature +avxvnni -emit-llvm -o - -Wall -Werror | FileCheck %s
+
+#include <immintrin.h>
+
+__m256i test_mm256_dpbusd_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpbusd_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusd.256
+  return _mm256_dpbusd_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpbusds_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpbusds_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusds.256
+  return _mm256_dpbusds_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpwssd_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpwssd_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssd.256
+  return _mm256_dpwssd_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpwssds_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpwssds_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssds.256
+  return _mm256_dpwssds_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpbusd_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpbusd_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusd.128
+  return _mm_dpbusd_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpbusds_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpbusds_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusds.128
+  return _mm_dpbusds_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpwssd_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpwssd_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssd.128
+  return _mm_dpwssd_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpwssds_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpwssds_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssds.128
+  return _mm_dpwssds_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpbusd_avx_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpbusd_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusd.256
+  return _mm256_dpbusd_avx_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpbusds_avx_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpbusds_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusds.256
+  return _mm256_dpbusds_avx_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpwssd_avx_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpwssd_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssd.256
+  return _mm256_dpwssd_avx_epi32(__S, __A, __B);
+}
+
+__m256i test_mm256_dpwssds_avx_epi32(__m256i __S, __m256i __A, __m256i __B) {
+  // CHECK-LABEL: @test_mm256_dpwssds_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssds.256
+  return _mm256_dpwssds_avx_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpbusd_avx_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpbusd_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusd.128
+  return _mm_dpbusd_avx_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpbusds_avx_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpbusds_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpbusds.128
+  return _mm_dpbusds_avx_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpwssd_avx_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpwssd_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssd.128
+  return _mm_dpwssd_avx_epi32(__S, __A, __B);
+}
+
+__m128i test_mm_dpwssds_avx_epi32(__m128i __S, __m128i __A, __m128i __B) {
+  // CHECK-LABEL: @test_mm_dpwssds_avx_epi32
+  // CHECK: @llvm.x86.avx512.vpdpwssds.128
+  return _mm_dpwssds_avx_epi32(__S, __A, __B);
+}

clang/test/CodeGen/attr-target-x86.c

 // CHECK: qax{{.*}} #5
 // CHECK: qq{{.*}} #6
 // CHECK: lake{{.*}} #7
 // CHECK: use_before_def{{.*}} #7
 // CHECK: walrus{{.*}} #8
 // CHECK: #0 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87" "tune-cpu"="i686"
 // CHECK: #1 = {{.*}}"target-cpu"="ivybridge" "target-features"="+avx,+cx16,+cx8,+f16c,+fsgsbase,+fxsr,+mmx,+pclmul,+popcnt,+rdrnd,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave,+xsaveopt"
 // CHECK-NOT: tune-cpu
-// CHECK: #2 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-aes,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-f16c,-fma,-fma4,-gfni,-kl,-pclmul,-sha,-sse2,-sse3,-sse4.1,-sse4.2,-sse4a,-ssse3,-vaes,-vpclmulqdq,-widekl,-xop" "tune-cpu"="i686"
+// CHECK: #2 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-aes,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,-f16c,-fma,-fma4,-gfni,-kl,-pclmul,-sha,-sse2,-sse3,-sse4.1,-sse4.2,-sse4a,-ssse3,-vaes,-vpclmulqdq,-widekl,-xop" "tune-cpu"="i686"
 // CHECK: #3 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+mmx,+popcnt,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87" "tune-cpu"="i686"
-// CHECK: #4 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-f16c,-fma,-fma4,-sse4.1,-sse4.2,-vaes,-vpclmulqdq,-xop" "tune-cpu"="i686"
+// CHECK: #4 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-avx,-avx2,-avx512bf16,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vp2intersect,-avx512vpopcntdq,-avxvnni,-f16c,-fma,-fma4,-sse4.1,-sse4.2,-vaes,-vpclmulqdq,-xop" "tune-cpu"="i686"
 // CHECK: #5 = {{.*}}"target-cpu"="ivybridge" "target-features"="+avx,+cx16,+cx8,+f16c,+fsgsbase,+fxsr,+mmx,+pclmul,+popcnt,+rdrnd,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave,+xsaveopt,-aes,-vaes"
 // CHECK-NOT: tune-cpu
 // CHECK: #6 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87,-3dnow,-3dnowa,-mmx"
 // CHECK: #7 = {{.*}}"target-cpu"="lakemont" "target-features"="+cx8,+mmx"
 // CHECK-NOT: tune-cpu
 // CHECK: #8 = {{.*}}"target-cpu"="i686" "target-features"="+cx8,+x87" "tune-cpu"="sandybridge"
 
 // CHECK: "target-cpu"="x86-64-v2"
 // CHECK-SAME: "target-features"="+cx16,+cx8,+fxsr,+mmx,+popcnt,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87"
 // CHECK: "target-cpu"="x86-64-v3"
 // CHECK-SAME: "target-features"="+avx,+avx2,+bmi,+bmi2,+cx16,+cx8,+f16c,+fma,+fxsr,+lzcnt,+mmx,+movbe,+popcnt,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave"
 // CHECK: "target-cpu"="x86-64-v4"
 // CHECK-SAME: "target-features"="+avx,+avx2,+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl,+bmi,+bmi2,+cx16,+cx8,+f16c,+fma,+fxsr,+lzcnt,+mmx,+movbe,+popcnt,+sahf,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave"

clang/test/Driver/x86-target-features.c

 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-hreset %s -### -o %t.o 2>&1 | FileCheck -check-prefix=NO-HRESET %s
 // HRESET: "-target-feature" "+hreset"
 // NO-HRESET: "-target-feature" "-hreset"
 
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -muintr %s -### -o %t.o 2>&1 | FileCheck -check-prefix=UINTR %s
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-uintr %s -### -o %t.o 2>&1 | FileCheck -check-prefix=NO-UINTR %s
 // UINTR: "-target-feature" "+uintr"
 // NO-UINTR: "-target-feature" "-uintr"
+
+// RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mavxvnni %s -### -o %t.o 2>&1 | FileCheck --check-prefix=AVX-VNNI %s
+// RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-avxvnni %s -### -o %t.o 2>&1 | FileCheck --check-prefix=NO-AVX-VNNI %s
+// AVX-VNNI: "-target-feature" "+avxvnni"
+// NO-AVX-VNNI: "-target-feature" "-avxvnni"

clang/test/Preprocessor/predefined-arch-macros.c

 // CHECK_TGL_M64: #define __amd64 1
 // CHECK_TGL_M64: #define __amd64__ 1
 // CHECK_TGL_M64: #define __corei7 1
 // CHECK_TGL_M64: #define __corei7__ 1
 // CHECK_TGL_M64: #define __tune_corei7__ 1
 // CHECK_TGL_M64: #define __x86_64 1
 // CHECK_TGL_M64: #define __x86_64__ 1
 
 // RUN: %clang -march=sapphirerapids -m32 -E -dM %s -o - 2>&1 \
 // RUN:     -target i386-unknown-linux \
 // RUN:   | FileCheck -match-full-lines %s -check-prefix=CHECK_SPR_M32
 // CHECK_SPR_M32: #define __AES__ 1
 // CHECK_SPR_M32: #define __AMXBF16__ 1
 // CHECK_SPR_M32: #define __AMXINT8__ 1
 // CHECK_SPR_M32: #define __AMXTILE__ 1
 // CHECK_SPR_M32: #define __AVX2__ 1
 // CHECK_SPR_M32: #define __AVX512BF16__ 1
 // CHECK_SPR_M32: #define __AVX512BITALG__ 1
 // CHECK_SPR_M32: #define __AVX512BW__ 1
 // CHECK_SPR_M32: #define __AVX512CD__ 1
 // CHECK_SPR_M32: #define __AVX512DQ__ 1
 // CHECK_SPR_M32: #define __AVX512F__ 1
 // CHECK_SPR_M32: #define __AVX512IFMA__ 1
 // CHECK_SPR_M32: #define __AVX512VBMI2__ 1
 // CHECK_SPR_M32: #define __AVX512VBMI__ 1
 // CHECK_SPR_M32: #define __AVX512VL__ 1
 // CHECK_SPR_M32: #define __AVX512VNNI__ 1
 // CHECK_SPR_M32: #define __AVX512VPOPCNTDQ__ 1
+// CHECK_SPR_M32: #define __AVXVNNI__ 1
 // CHECK_SPR_M32: #define __AVX__ 1
 // CHECK_SPR_M32: #define __BMI2__ 1
 // CHECK_SPR_M32: #define __BMI__ 1
 // CHECK_SPR_M32: #define __CLDEMOTE__ 1
 // CHECK_SPR_M32: #define __CLFLUSHOPT__ 1
 // CHECK_SPR_M32: #define __CLWB__ 1
 // CHECK_SPR_M32: #define __ENQCMD__ 1
 // CHECK_SPR_M32: #define __F16C__ 1

 // CHECK_SPR_M32: #define __XSAVE__ 1
 // CHECK_SPR_M32: #define __corei7 1
 // CHECK_SPR_M32: #define __corei7__ 1
 // CHECK_SPR_M32: #define __i386 1
 // CHECK_SPR_M32: #define __i386__ 1
 // CHECK_SPR_M32: #define __tune_corei7__ 1
 // CHECK_SPR_M32: #define i386 1
 
 // RUN: %clang -march=sapphirerapids -m64 -E -dM %s -o - 2>&1 \
 // RUN:     -target i386-unknown-linux \
 // RUN:   | FileCheck -match-full-lines %s -check-prefix=CHECK_SPR_M64
 // CHECK_SPR_M64: #define __AES__ 1
 // CHECK_SPR_M64: #define __AMXBF16__ 1
 // CHECK_SPR_M64: #define __AMXINT8__ 1
 // CHECK_SPR_M64: #define __AMXTILE__ 1
 // CHECK_SPR_M64: #define __AVX2__ 1
 // CHECK_SPR_M64: #define __AVX512BF16__ 1
 // CHECK_SPR_M64: #define __AVX512BITALG__ 1
 // CHECK_SPR_M64: #define __AVX512BW__ 1
 // CHECK_SPR_M64: #define __AVX512CD__ 1
 // CHECK_SPR_M64: #define __AVX512DQ__ 1
 // CHECK_SPR_M64: #define __AVX512F__ 1
 // CHECK_SPR_M64: #define __AVX512IFMA__ 1
 // CHECK_SPR_M64: #define __AVX512VBMI2__ 1
 // CHECK_SPR_M64: #define __AVX512VBMI__ 1
 // CHECK_SPR_M64: #define __AVX512VL__ 1
 // CHECK_SPR_M64: #define __AVX512VNNI__ 1
 // CHECK_SPR_M64: #define __AVX512VPOPCNTDQ__ 1
+// CHECK_SPR_M64: #define __AVXVNNI__ 1
 // CHECK_SPR_M64: #define __AVX__ 1
 // CHECK_SPR_M64: #define __BMI2__ 1
 // CHECK_SPR_M64: #define __BMI__ 1
 // CHECK_SPR_M64: #define __CLDEMOTE__ 1
 // CHECK_SPR_M64: #define __CLFLUSHOPT__ 1
 // CHECK_SPR_M64: #define __CLWB__ 1
 // CHECK_SPR_M64: #define __ENQCMD__ 1
 // CHECK_SPR_M64: #define __F16C__ 1

 
 // RUN: %clang -march=alderlake -m32 -E -dM %s -o - 2>&1 \
 // RUN:     -target i386-unknown-linux \
 // RUN:   | FileCheck -match-full-lines %s -check-prefix=CHECK_ADL_M32
 // CHECK_ADL_M32: #define __ADX__ 1
 // CHECK_ADL_M32: #define __AES__ 1
 // CHECK_ADL_M32: #define __AVX2__ 1
 // CHECK_ADL_M32-NOT: AVX512
+// CHECK_ADL_M32: #define __AVXVNNI__ 1
 // CHECK_ADL_M32: #define __AVX__ 1
 // CHECK_ADL_M32: #define __BMI2__ 1
 // CHECK_ADL_M32: #define __BMI__ 1
 // CHECK_ADL_M32: #define __CLDEMOTE__ 1
 // CHECK_ADL_M32: #define __CLFLUSHOPT__ 1
 // CHECK_ADL_M32: #define __F16C__ 1
 // CHECK_ADL_M32: #define __FMA__ 1
 // CHECK_ADL_M32: #define __HRESET__ 1

 // CHECK_ADL_M32: #define i386 1
 
 // RUN: %clang -march=alderlake -m64 -E -dM %s -o - 2>&1 \
 // RUN:     -target i386-unknown-linux \
 // RUN:   | FileCheck -match-full-lines %s -check-prefix=CHECK_ADL_M64
 // CHECK_ADL_M64: #define __ADX__ 1
 // CHECK_ADL_M64: #define __AES__ 1
 // CHECK_ADL_M64: #define __AVX2__ 1
+// CHECK_ADL_M64: #define __AVXVNNI__ 1
 // CHECK_ADL_M64-NOT: AVX512
 // CHECK_ADL_M64: #define __AVX__ 1
 // CHECK_ADL_M64: #define __BMI2__ 1
 // CHECK_ADL_M64: #define __BMI__ 1
 // CHECK_ADL_M64: #define __CLDEMOTE__ 1
 // CHECK_ADL_M64: #define __CLFLUSHOPT__ 1
 // CHECK_ADL_M64: #define __F16C__ 1
 // CHECK_ADL_M64: #define __FMA__ 1
 // CHECK_ADL_M64: #define __HRESET__ 1

clang/test/Preprocessor/x86_target_features.c

 
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -muintr -x c -E -dM -o - %s | FileCheck -check-prefix=UINTR %s
 
 // UINTR: #define __UINTR__ 1
 
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-uintr -x c -E -dM -o - %s | FileCheck -check-prefix=NOUINTR %s
 
 // NOUINTR-NOT: #define __UINTR__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mavxvnni -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVXVNNI %s
+
+// AVXVNNI: #define __AVX2__ 1
+// AVXVNNI: #define __AVXVNNI__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mno-avxvnni -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=NOAVXVNNI %s
+
+// NOAVXVNNI-NOT: #define __AVXVNNI__ 1
+
+// RUN: %clang -target i386-unknown-unknown -march=atom -mavxvnni -mno-avx2 -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVXVNNINOAVX2 %s
+
+// AVXVNNINOAVX2-NOT: #define __AVX2__ 1
+// AVXVNNINOAVX2-NOT: #define __AVXVNNI__ 1

llvm/docs/ReleaseNotes.rst

* The assembler now has support for {disp32} and {disp8} pseudo prefixes for
   assembler also supports the .d32 and .d8 mnemonic suffixes to do the same.
 * A new function attribute "tune-cpu" has been added to support -mtune like gcc.
   This allows microarchitectural optimizations to be applied independent from
   the "target-cpu" attribute or TargetMachine CPU which will be used to select
   Instruction Set. If the attribute is not present, the tune CPU will follow
   the target CPU.
 * Support for ``HRESET`` instructions has been added.
 * Support for ``UINTR`` instructions has been added.
+* Support for ``AVXVNNI`` instructions has been added.
 
 Changes to the AMDGPU Target
 -----------------------------
 
 During this release ...
 
 * The new ``byref`` attribute is now the preferred method for
   representing aggregate kernel arguments.

llvm/include/llvm/Support/X86TargetParser.def

 X86_FEATURE       (WAITPKG,         "waitpkg")
 X86_FEATURE       (WBNOINVD,        "wbnoinvd")
 X86_FEATURE       (X87,             "x87")
 X86_FEATURE       (XSAVE,           "xsave")
 X86_FEATURE       (XSAVEC,          "xsavec")
 X86_FEATURE       (XSAVEOPT,        "xsaveopt")
 X86_FEATURE       (XSAVES,          "xsaves")
 X86_FEATURE       (HRESET,          "hreset")
+X86_FEATURE       (AVXVNNI,         "avxvnni")
 // These features aren't really CPU features, but the frontend can set them.
 X86_FEATURE       (RETPOLINE_EXTERNAL_THUNK,    "retpoline-external-thunk")
 X86_FEATURE       (RETPOLINE_INDIRECT_BRANCHES, "retpoline-indirect-branches")
 X86_FEATURE       (RETPOLINE_INDIRECT_CALLS,    "retpoline-indirect-calls")
 X86_FEATURE       (LVI_CFI,                     "lvi-cfi")
 X86_FEATURE       (LVI_LOAD_HARDENING,          "lvi-load-hardening")
 #undef X86_FEATURE_COMPAT
 #undef X86_FEATURE

llvm/lib/Support/Host.cpp

#endif
   // detecting features using the "-march=native" flag.
   // For more info, see X86 ISA docs.
   Features["pconfig"] = HasLeaf7 && ((EDX >> 18) & 1);
   Features["amx-bf16"]   = HasLeaf7 && ((EDX >> 22) & 1) && HasAMXSave;
   Features["amx-tile"]   = HasLeaf7 && ((EDX >> 24) & 1) && HasAMXSave;
   Features["amx-int8"]   = HasLeaf7 && ((EDX >> 25) & 1) && HasAMXSave;
   bool HasLeaf7Subleaf1 =
       MaxLevel >= 7 && !getX86CpuIDAndInfoEx(0x7, 0x1, &EAX, &EBX, &ECX, &EDX);
+  Features["avxvnni"]    = HasLeaf7Subleaf1 && ((EAX >> 4) & 1) && HasAVXSave;
   Features["avx512bf16"] = HasLeaf7Subleaf1 && ((EAX >> 5) & 1) && HasAVX512Save;
   Features["hreset"]     = HasLeaf7Subleaf1 && ((EAX >> 22) & 1);
 
   bool HasLeafD = MaxLevel >= 0xd &&
                   !getX86CpuIDAndInfoEx(0xd, 0x1, &EAX, &EBX, &ECX, &EDX);
 
   // Only enable XSAVE if OS has enabled support for saving YMM state.
   Features["xsaveopt"] = HasLeafD && ((EAX >> 0) & 1) && HasAVXSave;

llvm/lib/Support/X86TargetParser.cpp

 constexpr FeatureBitset FeaturesTigerlake =
     FeaturesICLClient | FeatureAVX512VP2INTERSECT | FeatureMOVDIR64B |
     FeatureMOVDIRI | FeatureSHSTK | FeatureKL | FeatureWIDEKL;
 constexpr FeatureBitset FeaturesSapphireRapids =
     FeaturesICLServer | FeatureAMX_TILE | FeatureAMX_INT8 | FeatureAMX_BF16 |
     FeatureAVX512BF16 | FeatureAVX512VP2INTERSECT | FeatureCLDEMOTE |
     FeatureENQCMD | FeatureMOVDIR64B | FeatureMOVDIRI | FeaturePTWRITE |
     FeatureSERIALIZE | FeatureSHSTK | FeatureTSXLDTRK | FeatureUINTR |
-    FeatureWAITPKG;
+    FeatureWAITPKG | FeatureAVXVNNI;
 constexpr FeatureBitset FeaturesAlderlake =
     FeaturesSkylakeClient | FeatureCLDEMOTE | FeatureHRESET | FeaturePTWRITE |
-    FeatureSERIALIZE | FeatureWAITPKG;
+    FeatureSERIALIZE | FeatureWAITPKG | FeatureAVXVNNI;
 
 // Intel Atom processors.
 // Bonnell has feature parity with Core2 and adds MOVBE.
 constexpr FeatureBitset FeaturesBonnell = FeaturesCore2 | FeatureMOVBE;
 // Silvermont has parity with Westmere and Bonnell plus PRFCHW and RDRND.
 constexpr FeatureBitset FeaturesSilvermont =
     FeaturesBonnell | FeaturesWestmere | FeaturePRFCHW | FeatureRDRND;
 constexpr FeatureBitset FeaturesGoldmont =

 constexpr FeatureBitset ImpliedFeaturesAMX_BF16 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesAMX_INT8 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesHRESET = {};
 
 // Key Locker Features
 constexpr FeatureBitset ImpliedFeaturesKL = FeatureSSE2;
 constexpr FeatureBitset ImpliedFeaturesWIDEKL = FeatureKL;
 
+// AVXVNNI Features
+constexpr FeatureBitset ImpliedFeaturesAVXVNNI = FeatureAVX2;
+
 constexpr FeatureInfo FeatureInfos[X86::CPU_FEATURE_MAX] = {
 #define X86_FEATURE(ENUM, STR) {{STR}, ImpliedFeatures##ENUM},
 #include "llvm/Support/X86TargetParser.def"
 };
 
 void llvm::X86::getFeaturesForCPU(StringRef CPU,
                                   SmallVectorImpl<StringRef> &EnabledFeatures) {
   auto I = llvm::find_if(Processors,

llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp

if (ForcedVEXEncoding == VEXEncoding_EVEX &&
     return Match_Unsupported;
 
   if ((ForcedVEXEncoding == VEXEncoding_VEX ||
        ForcedVEXEncoding == VEXEncoding_VEX2 ||
        ForcedVEXEncoding == VEXEncoding_VEX3) &&
       (MCID.TSFlags & X86II::EncodingMask) != X86II::VEX)
     return Match_Unsupported;
 
+  // These instructions are only available with {vex}, {vex2} or {vex3} prefix
+  if (MCID.TSFlags & X86II::ExplicitVEXPrefix &&
+      (ForcedVEXEncoding != VEXEncoding_VEX &&
+       ForcedVEXEncoding != VEXEncoding_VEX3))
+    return Match_Unsupported;
+
   // These instructions match ambiguously with their VEX encoded counterparts
   // and appear first in the matching table. Reject them unless we're forcing
   // EVEX encoding.
   // FIXME: We really need a way to break the ambiguity.
   switch (Opc) {
   case X86::VCVTSD2SIZrm_Int:
   case X86::VCVTSD2SI64Zrm_Int:
   case X86::VCVTSS2SIZrm_Int:

llvm/lib/Target/X86/MCTargetDesc/X86BaseInfo.h

enum : uint64_t {
     CD8_Scale_Mask = 127ULL << CD8_Scale_Shift,
 
     /// Explicitly specified rounding control
     EVEX_RCShift = CD8_Scale_Shift + 7,
     EVEX_RC = 1ULL << EVEX_RCShift,
 
     // NOTRACK prefix
     NoTrackShift = EVEX_RCShift + 1,
-    NOTRACK = 1ULL << NoTrackShift
+    NOTRACK = 1ULL << NoTrackShift,
+
+    // Force VEX encoding
+    ExplicitVEXShift = NoTrackShift + 1,
+    ExplicitVEXPrefix = 1ULL << ExplicitVEXShift
   };
 
   /// \returns true if the instruction with given opcode is a prefix.
   inline bool isPrefix(uint64_t TSFlags) {
     return (TSFlags & X86II::FormMask) == PrefixByte;
   }
 
   /// \returns true if the instruction with given opcode is a pseudo.

llvm/lib/Target/X86/MCTargetDesc/X86InstPrinterCommon.cpp

if ((TSFlags & X86II::NOTRACK) || (Flags & X86::IP_HAS_NOTRACK))
     O << "\tnotrack\t";
 
   if (Flags & X86::IP_HAS_REPEAT_NE)
     O << "\trepne\t";
   else if (Flags & X86::IP_HAS_REPEAT)
     O << "\trep\t";
 
   // These all require a pseudo prefix
-  if (Flags & X86::IP_USE_VEX)
+  if (Flags & X86::IP_USE_VEX || TSFlags & X86II::ExplicitVEXPrefix)
     O << "\t{vex}";
   else if (Flags & X86::IP_USE_VEX2)
     O << "\t{vex2}";
   else if (Flags & X86::IP_USE_VEX3)
     O << "\t{vex3}";
   else if (Flags & X86::IP_USE_EVEX)
     O << "\t{evex}";
 

llvm/lib/Target/X86/X86.td

 def FeatureIFMA     : SubtargetFeature<"avx512ifma", "HasIFMA", "true",
                       "Enable AVX-512 Integer Fused Multiple-Add",
                                       [FeatureAVX512]>;
 def FeaturePKU   : SubtargetFeature<"pku", "HasPKU", "true",
                       "Enable protection keys">;
 def FeatureVNNI    : SubtargetFeature<"avx512vnni", "HasVNNI", "true",
                           "Enable AVX-512 Vector Neural Network Instructions",
                                       [FeatureAVX512]>;
+def FeatureAVXVNNI    : SubtargetFeature<"avxvnni", "HasAVXVNNI", "true",
+                           "Support AVX_VNNI encoding",
+                                      [FeatureAVX2]>;
 def FeatureBF16    : SubtargetFeature<"avx512bf16", "HasBF16", "true",
                            "Support bfloat16 floating point",
                                       [FeatureBWI]>;
 def FeatureBITALG  : SubtargetFeature<"avx512bitalg", "HasBITALG", "true",
                        "Enable AVX-512 Bit Algorithms",
                         [FeatureBWI]>;
 def FeatureVP2INTERSECT  : SubtargetFeature<"avx512vp2intersect",
                                             "HasVP2INTERSECT", "true",
def ProcessorFeatures {
   list<SubtargetFeature> SPRAdditionalFeatures = [FeatureAMXTILE,
                                                   FeatureAMXINT8,
                                                   FeatureAMXBF16,
                                                   FeatureBF16,
                                                   FeatureSERIALIZE,
                                                   FeatureCLDEMOTE,
                                                   FeatureWAITPKG,
                                                   FeaturePTWRITE,
+                                                  FeatureAVXVNNI,
                                                   FeatureTSXLDTRK,
                                                   FeatureENQCMD,
                                                   FeatureSHSTK,
                                                   FeatureVP2INTERSECT,
                                                   FeatureMOVDIRI,
                                                   FeatureMOVDIR64B,
                                                   FeatureUINTR];
   list<SubtargetFeature> SPRTuning = ICXTuning;
   list<SubtargetFeature> SPRFeatures =
     !listconcat(ICXFeatures, SPRAdditionalFeatures);
 
   // Alderlake
-  list<SubtargetFeature> ADLAdditionalFeatures = [FeatureCLDEMOTE,
+  list<SubtargetFeature> ADLAdditionalFeatures = [FeatureAVXVNNI,
+                                                  FeatureCLDEMOTE,
                                                   FeatureHRESET,
                                                   FeaturePTWRITE,
                                                   FeatureSERIALIZE,
                                                   FeatureWAITPKG];
   list<SubtargetFeature> ADLTuning = SKLTuning;
   list<SubtargetFeature> ADLFeatures =
     !listconcat(SKLFeatures, ADLAdditionalFeatures);
 

llvm/lib/Target/X86/X86EvexToVex.cpp

public:
   MachineFunctionProperties getRequiredProperties() const override {
     return MachineFunctionProperties().set(
         MachineFunctionProperties::Property::NoVRegs);
   }
 
 private:
   /// Machine instruction info used throughout the class.
   const X86InstrInfo *TII = nullptr;
+
+  const X86Subtarget *ST = nullptr;
 };
 
 } // end anonymous namespace
 
 char EvexToVexInstPass::ID = 0;
 
 bool EvexToVexInstPass::runOnMachineFunction(MachineFunction &MF) {
   TII = MF.getSubtarget<X86Subtarget>().getInstrInfo();
 
-  const X86Subtarget &ST = MF.getSubtarget<X86Subtarget>();
-  if (!ST.hasAVX512())
+  ST = &MF.getSubtarget<X86Subtarget>();
+  if (!ST->hasAVX512())
     return false;
 
   bool Changed = false;
 
   /// Go over all basic blocks in function and replace
   /// EVEX encoded instrs by VEX encoding when possible.
   for (MachineBasicBlock &MBB : MF) {
 
for (const MachineOperand &MO : MI.explicit_operands()) {
     if (isHiRegIdx(Reg))
       return true;
   }
 
   return false;
 }
 
 // Do any custom cleanup needed to finalize the conversion.
-static bool performCustomAdjustments(MachineInstr &MI, unsigned NewOpc) {
+static bool performCustomAdjustments(MachineInstr &MI, unsigned NewOpc,
+                                     const X86Subtarget *ST) {
   (void)NewOpc;
   unsigned Opc = MI.getOpcode();
   switch (Opc) {
+  case X86::VPDPBUSDSZ256m:
+  case X86::VPDPBUSDSZ256r:
+  case X86::VPDPBUSDSZ128m:
+  case X86::VPDPBUSDSZ128r:
+  case X86::VPDPBUSDZ256m:
+  case X86::VPDPBUSDZ256r:
+  case X86::VPDPBUSDZ128m:
+  case X86::VPDPBUSDZ128r:
+  case X86::VPDPWSSDSZ256m:
+  case X86::VPDPWSSDSZ256r:
+  case X86::VPDPWSSDSZ128m:
+  case X86::VPDPWSSDSZ128r:
+  case X86::VPDPWSSDZ256m:
+  case X86::VPDPWSSDZ256r:
+  case X86::VPDPWSSDZ128m:
+  case X86::VPDPWSSDZ128r:
+    // These can only VEX convert if AVXVNNI is enabled.
+    return ST->hasAVXVNNI();
   case X86::VALIGNDZ128rri:
   case X86::VALIGNDZ128rmi:
   case X86::VALIGNQZ128rri:
   case X86::VALIGNQZ128rmi: {
     assert((NewOpc == X86::VPALIGNRrri || NewOpc == X86::VPALIGNRrmi) &&
            "Unexpected new opcode!");
     unsigned Scale = (Opc == X86::VALIGNQZ128rri ||
                       Opc == X86::VALIGNQZ128rmi) ? 8 : 4;
#endif
   if (I == Table.end() || I->EvexOpcode != MI.getOpcode())
     return false;
 
   unsigned NewOpc = I->VexOpcode;
 
   if (usesExtendedRegister(MI))
     return false;
 
-  if (!performCustomAdjustments(MI, NewOpc))
+  if (!performCustomAdjustments(MI, NewOpc, ST))
     return false;
 
   MI.setDesc(TII->get(NewOpc));
   MI.setAsmPrinterFlag(X86::AC_EVEX_2_VEX);
   return true;
 }
 
 INITIALIZE_PASS(EvexToVexInstPass, EVEX2VEX_NAME, EVEX2VEX_DESC, false, false)
 
 FunctionPass *llvm::createX86EvexToVexInsts() {
   return new EvexToVexInstPass();
 }

llvm/lib/Target/X86/X86InstrFoldTables.cpp

static const X86MemoryFoldTableEntry MemoryFoldTable3[] = {
   { X86::VPCMPWZ256rrik,             X86::VPCMPWZ256rmik,             0 },
   { X86::VPCMPWZrrik,                X86::VPCMPWZrmik,                0 },
   { X86::VPCONFLICTDZ128rrk,         X86::VPCONFLICTDZ128rmk,         0 },
   { X86::VPCONFLICTDZ256rrk,         X86::VPCONFLICTDZ256rmk,         0 },
   { X86::VPCONFLICTDZrrk,            X86::VPCONFLICTDZrmk,            0 },
   { X86::VPCONFLICTQZ128rrk,         X86::VPCONFLICTQZ128rmk,         0 },
   { X86::VPCONFLICTQZ256rrk,         X86::VPCONFLICTQZ256rmk,         0 },
   { X86::VPCONFLICTQZrrk,            X86::VPCONFLICTQZrmk,            0 },
+  { X86::VPDPBUSDSYrr,               X86::VPDPBUSDSYrm,               0 },
   { X86::VPDPBUSDSZ128r,             X86::VPDPBUSDSZ128m,             0 },
   { X86::VPDPBUSDSZ256r,             X86::VPDPBUSDSZ256m,             0 },
   { X86::VPDPBUSDSZr,                X86::VPDPBUSDSZm,                0 },
+  { X86::VPDPBUSDSrr,                X86::VPDPBUSDSrm,                0 },
+  { X86::VPDPBUSDYrr,                X86::VPDPBUSDYrm,                0 },
   { X86::VPDPBUSDZ128r,              X86::VPDPBUSDZ128m,              0 },
   { X86::VPDPBUSDZ256r,              X86::VPDPBUSDZ256m,              0 },
   { X86::VPDPBUSDZr,                 X86::VPDPBUSDZm,                 0 },
+  { X86::VPDPBUSDrr,                 X86::VPDPBUSDrm,                 0 },
+  { X86::VPDPWSSDSYrr,               X86::VPDPWSSDSYrm,               0 },
   { X86::VPDPWSSDSZ128r,             X86::VPDPWSSDSZ128m,             0 },
   { X86::VPDPWSSDSZ256r,             X86::VPDPWSSDSZ256m,             0 },
   { X86::VPDPWSSDSZr,                X86::VPDPWSSDSZm,                0 },
+  { X86::VPDPWSSDSrr,                X86::VPDPWSSDSrm,                0 },
+  { X86::VPDPWSSDYrr,                X86::VPDPWSSDYrm,                0 },
   { X86::VPDPWSSDZ128r,              X86::VPDPWSSDZ128m,              0 },
   { X86::VPDPWSSDZ256r,              X86::VPDPWSSDZ256m,              0 },
   { X86::VPDPWSSDZr,                 X86::VPDPWSSDZm,                 0 },
+  { X86::VPDPWSSDrr,                 X86::VPDPWSSDrm,                 0 },
   { X86::VPERMBZ128rrkz,             X86::VPERMBZ128rmkz,             0 },
   { X86::VPERMBZ256rrkz,             X86::VPERMBZ256rmkz,             0 },
   { X86::VPERMBZrrkz,                X86::VPERMBZrmkz,                0 },
   { X86::VPERMDZ256rrkz,             X86::VPERMDZ256rmkz,             0 },
   { X86::VPERMDZrrkz,                X86::VPERMDZrmkz,                0 },
   { X86::VPERMI2B128rr,              X86::VPERMI2B128rm,              0 },
   { X86::VPERMI2B256rr,              X86::VPERMI2B256rm,              0 },
   { X86::VPERMI2Brr,                 X86::VPERMI2Brm,                 0 },

llvm/lib/Target/X86/X86InstrFormats.td

 }
 
 // Mark the instruction as "illegal to memory fold/unfold"
 class NotMemoryFoldable { bit isMemoryFoldable = 0; }
 
 // Prevent EVEX->VEX conversion from considering this instruction.
 class NotEVEX2VEXConvertible { bit notEVEX2VEXConvertible = 1; }
 
+// Force the instruction to use VEX encoding.
+class ExplicitVEXPrefix { bit ExplicitVEXPrefix = 1; }
+
 class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
               string AsmStr, Domain d = GenericDomain>
   : Instruction {
   let Namespace = "X86";
 
   bits<8> Opcode = opcod;
   Format Form = f;
   bits<7> FormBits = Form.Value;
class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
   // instruction to replace the current one in case it got picked during generation.
   string FoldGenRegForm = ?;
 
   // Used to prevent an explicit EVEX2VEX override for this instruction.
   string EVEX2VEXOverride = ?;
 
   bit isMemoryFoldable = 1;     // Is it allowed to memory fold/unfold this instruction?
   bit notEVEX2VEXConvertible = 0; // Prevent EVEX->VEX conversion.
+  bit ExplicitVEXPrefix = 0; // Force the instruction to use VEX encoding.
 
   // TSFlags layout should be kept in sync with X86BaseInfo.h.
   let TSFlags{6-0}   = FormBits;
   let TSFlags{8-7}   = OpSizeBits;
   let TSFlags{10-9}  = AdSizeBits;
   // No need for 3rd bit, we don't need to distinguish NoPrfx from PS.
   let TSFlags{12-11} = OpPrefixBits{1-0};
   let TSFlags{15-13} = OpMapBits;
class X86Inst<bits<8> opcod, Format f, ImmType i, dag outs, dag ins,
   let TSFlags{41}    = hasEVEX_K;
   let TSFlags{42}    = hasEVEX_Z;
   let TSFlags{43}    = hasEVEX_L2;
   let TSFlags{44}    = hasEVEX_B;
   // If we run out of TSFlags bits, it's possible to encode this in 3 bits.
   let TSFlags{51-45} = CD8_Scale;
   let TSFlags{52}    = hasEVEX_RC;
   let TSFlags{53}    = hasNoTrackPrefix;
+  let TSFlags{54}    = ExplicitVEXPrefix;
 }
 
 class PseudoI<dag oops, dag iops, list<dag> pattern>
   : X86Inst<0, Pseudo, NoImm, oops, iops, ""> {
   let Pattern = pattern;
 }
 
 class I<bits<8> o, Format f, dag outs, dag ins, string asm,

llvm/lib/Target/X86/X86InstrInfo.cpp

bool X86InstrInfo::findCommutedOpIndices(const MachineInstr &MI,
   case X86::VPTERNLOGQZrmbi:
   case X86::VPTERNLOGDZ128rmbikz:
   case X86::VPTERNLOGDZ256rmbikz:
   case X86::VPTERNLOGDZrmbikz:
   case X86::VPTERNLOGQZ128rmbikz:
   case X86::VPTERNLOGQZ256rmbikz:
   case X86::VPTERNLOGQZrmbikz:
     return findThreeSrcCommutedOpIndices(MI, SrcOpIdx1, SrcOpIdx2);
+  case X86::VPDPWSSDYrr:
+  case X86::VPDPWSSDrr:
+  case X86::VPDPWSSDSYrr:
+  case X86::VPDPWSSDSrr:
   case X86::VPDPWSSDZ128r:
   case X86::VPDPWSSDZ128rk:
   case X86::VPDPWSSDZ128rkz:
   case X86::VPDPWSSDZ256r:
   case X86::VPDPWSSDZ256rk:
   case X86::VPDPWSSDZ256rkz:
   case X86::VPDPWSSDZr:
   case X86::VPDPWSSDZrk:

llvm/lib/Target/X86/X86InstrInfo.td

 def HasVLX       : Predicate<"Subtarget->hasVLX()">;
 def NoVLX        : Predicate<"!Subtarget->hasVLX()">;
 def NoVLX_Or_NoBWI : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasBWI()">;
 def NoVLX_Or_NoDQI : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasDQI()">;
 def PKU        : Predicate<"Subtarget->hasPKU()">;
 def HasVNNI    : Predicate<"Subtarget->hasVNNI()">;
 def HasVP2INTERSECT : Predicate<"Subtarget->hasVP2INTERSECT()">;
 def HasBF16      : Predicate<"Subtarget->hasBF16()">;
+def HasAVXVNNI : Predicate <"Subtarget->hasAVXVNNI()">;
+def NoVLX_Or_NoVNNI : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasVNNI()">;
 
 def HasBITALG    : Predicate<"Subtarget->hasBITALG()">;
 def HasPOPCNT    : Predicate<"Subtarget->hasPOPCNT()">;
 def HasAES       : Predicate<"Subtarget->hasAES()">;
 def HasVAES      : Predicate<"Subtarget->hasVAES()">;
 def NoVLX_Or_NoVAES : Predicate<"!Subtarget->hasVLX() || !Subtarget->hasVAES()">;
 def HasFXSR      : Predicate<"Subtarget->hasFXSR()">;
 def HasXSAVE     : Predicate<"Subtarget->hasXSAVE()">;

llvm/lib/Target/X86/X86InstrSSE.td

 defm VMASKMOVPD : avx_movmask_rm<0x2D, 0x2F, "vmaskmovpd",
                                  int_x86_avx_maskload_pd,
                                  int_x86_avx_maskload_pd_256,
                                  int_x86_avx_maskstore_pd,
                                  int_x86_avx_maskstore_pd_256,
                                  WriteFMaskMove64, WriteFMaskMove64Y>;
 
 //===----------------------------------------------------------------------===//
+// AVX_VNNI
+//===----------------------------------------------------------------------===//
+let Predicates = [HasAVXVNNI, NoVLX_Or_NoVNNI], Constraints = "$src1 = $dst" in
+multiclass avx_vnni_rm<bits<8> opc, string OpcodeStr, SDNode OpNode,
+                       bit IsCommutable> {
+  let isCommutable = IsCommutable in
+  def rr  : AVX8I<opc, MRMSrcReg, (outs VR128:$dst),
+             (ins VR128:$src1, VR128:$src2, VR128:$src3),
+             !strconcat(OpcodeStr, "\t{$src3, $src2, $dst|$dst, $src2, $src3}"),
+             [(set VR128:$dst, (v4i32 (OpNode VR128:$src1,
+                                       VR128:$src2, VR128:$src3)))]>,
+             VEX_4V, Sched<[SchedWriteVecIMul.XMM]>;
+
+  def rm  : AVX8I<opc, MRMSrcMem, (outs VR128:$dst),
+             (ins VR128:$src1, VR128:$src2, i128mem:$src3),
+             !strconcat(OpcodeStr, "\t{$src3, $src2, $dst|$dst, $src2, $src3}"),
+             [(set VR128:$dst, (v4i32 (OpNode VR128:$src1, VR128:$src2,
+                                      (loadv4i32 addr:$src3))))]>,
+             VEX_4V, Sched<[SchedWriteVecIMul.XMM]>;
+
+  let isCommutable = IsCommutable in
+  def Yrr  : AVX8I<opc, MRMSrcReg, (outs VR256:$dst),
+             (ins VR256:$src1, VR256:$src2, VR256:$src3),
+             !strconcat(OpcodeStr, "\t{$src3, $src2, $dst|$dst, $src2, $src3}"),
+             [(set VR256:$dst, (v8i32 (OpNode VR256:$src1,
+                                       VR256:$src2, VR256:$src3)))]>,
+             VEX_4V, VEX_L, Sched<[SchedWriteVecIMul.XMM]>;
+
+  def Yrm  : AVX8I<opc, MRMSrcMem, (outs VR256:$dst),
+             (ins VR256:$src1, VR256:$src2, i256mem:$src3),
+             !strconcat(OpcodeStr, "\t{$src3, $src2, $dst|$dst, $src2, $src3}"),
+             [(set VR256:$dst, (v8i32 (OpNode VR256:$src1, VR256:$src2,
+                                      (loadv8i32 addr:$src3))))]>,
+             VEX_4V, VEX_L, Sched<[SchedWriteVecIMul.XMM]>;
+}
+
+defm VPDPBUSD   : avx_vnni_rm<0x50, "vpdpbusd", X86Vpdpbusd, 0>, ExplicitVEXPrefix;
+defm VPDPBUSDS  : avx_vnni_rm<0x51, "vpdpbusds", X86Vpdpbusds, 0>, ExplicitVEXPrefix;
+defm VPDPWSSD   : avx_vnni_rm<0x52, "vpdpwssd",  X86Vpdpwssd, 1>, ExplicitVEXPrefix;
+defm VPDPWSSDS  : avx_vnni_rm<0x53, "vpdpwssds", X86Vpdpwssds, 1>, ExplicitVEXPrefix;
+
+//===----------------------------------------------------------------------===//
 // VPERMIL - Permute Single and Double Floating-Point Values
 //
 
 multiclass avx_permil<bits<8> opc_rm, bits<8> opc_rmi, string OpcodeStr,
                       RegisterClass RC, X86MemOperand x86memop_f,
                       X86MemOperand x86memop_i,
                       ValueType f_vt, ValueType i_vt,
                       X86FoldableSchedWrite sched,

llvm/lib/Target/X86/X86Subtarget.h

class X86Subtarget final : public X86GenSubtargetInfo {
   bool HasVLX = false;
 
   /// Processor has PKU extenstions
   bool HasPKU = false;
 
   /// Processor has AVX-512 Vector Neural Network Instructions
   bool HasVNNI = false;
 
+  /// Processor has AVX Vector Neural Network Instructions
+  bool HasAVXVNNI = false;
+
   /// Processor has AVX-512 bfloat16 floating-point extensions
   bool HasBF16 = false;
 
   /// Processor supports ENQCMD instructions
   bool HasENQCMD = false;
 
   /// Processor has AVX-512 Bit Algorithms instructions
   bool HasBITALG = false;
public:
   bool hasHRESET() const { return HasHRESET; }
   bool hasSERIALIZE() const { return HasSERIALIZE; }
   bool hasTSXLDTRK() const { return HasTSXLDTRK; }
   bool hasUINTR() const { return HasUINTR; }
   bool useRetpolineIndirectCalls() const { return UseRetpolineIndirectCalls; }
   bool useRetpolineIndirectBranches() const {
     return UseRetpolineIndirectBranches;
   }
+  bool hasAVXVNNI() const { return HasAVXVNNI; }
   bool hasAMXTILE() const { return HasAMXTILE; }
   bool hasAMXBF16() const { return HasAMXBF16; }
   bool hasAMXINT8() const { return HasAMXINT8; }
   bool useRetpolineExternalThunk() const { return UseRetpolineExternalThunk; }
 
   // These are generic getters that OR together all of the thunk types
   // supported by the subtarget. Therefore useIndirectThunk*() will return true
   // if any respective thunk feature is enabled.

llvm/test/CodeGen/X86/avx-vnni/avx_vnni-intrinsics.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+avxvnni --show-mc-encoding | FileCheck %s --check-prefixes=CHECK,AVX-X86
+; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+avx512vnni,+avx512vl,+avxvnni --show-mc-encoding | FileCheck %s --check-prefixes=CHECK,AVX512-X86
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avxvnni --show-mc-encoding | FileCheck %s --check-prefixes=CHECK,AVX-X64
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512vnni,+avx512vl,+avxvnni --show-mc-encoding | FileCheck %s --check-prefixes=CHECK,AVX512-X64
+
+declare <8 x i32> @llvm.x86.avx512.vpdpbusd.256(<8 x i32>, <8 x i32>, <8 x i32>)
+
+define <8 x i32>@test_int_x86_avx_vpdpbusd_256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpbusd_256:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpbusd %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x50,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpbusd_256:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpbusd %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x50,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpbusd_256:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpbusd %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x50,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpbusd_256:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpbusd %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x50,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <8 x i32> @llvm.x86.avx512.vpdpbusd.256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2)
+  ret <8 x i32> %res
+}
+
+declare <4 x i32> @llvm.x86.avx512.vpdpbusd.128(<4 x i32>, <4 x i32>, <4 x i32>)
+
+define <4 x i32>@test_int_x86_avx_vpdpbusd_128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpbusd_128:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpbusd %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x50,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpbusd_128:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpbusd %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x50,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpbusd_128:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpbusd %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x50,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpbusd_128:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpbusd %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x50,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <4 x i32> @llvm.x86.avx512.vpdpbusd.128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2)
+  ret <4 x i32> %res
+}
+
+declare <8 x i32> @llvm.x86.avx512.vpdpbusds.256(<8 x i32>, <8 x i32>, <8 x i32>)
+
+define <8 x i32>@test_int_x86_avx_vpdpbusds_256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpbusds_256:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpbusds %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x51,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpbusds_256:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpbusds %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x51,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpbusds_256:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpbusds %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x51,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpbusds_256:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpbusds %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x51,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <8 x i32> @llvm.x86.avx512.vpdpbusds.256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2)
+  ret <8 x i32> %res
+}
+
+declare <4 x i32> @llvm.x86.avx512.vpdpbusds.128(<4 x i32>, <4 x i32>, <4 x i32>)
+
+define <4 x i32>@test_int_x86_avx_vpdpbusds_128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpbusds_128:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpbusds %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x51,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpbusds_128:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpbusds %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x51,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpbusds_128:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpbusds %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x51,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpbusds_128:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpbusds %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x51,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <4 x i32> @llvm.x86.avx512.vpdpbusds.128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2)
+  ret <4 x i32> %res
+}
+
+declare <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32>, <8 x i32>, <8 x i32>)
+
+define <8 x i32>@test_int_x86_avx_vpdpwssd_256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpwssd_256:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpwssd %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x52,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpwssd_256:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpwssd %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x52,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpwssd_256:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpwssd %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x52,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpwssd_256:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpwssd %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x52,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2)
+  ret <8 x i32> %res
+}
+
+declare <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32>, <4 x i32>, <4 x i32>)
+
+define <4 x i32>@test_int_x86_avx_vpdpwssd_128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpwssd_128:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpwssd %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x52,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpwssd_128:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpwssd %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x52,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpwssd_128:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpwssd %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x52,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpwssd_128:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpwssd %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x52,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2)
+  ret <4 x i32> %res
+}
+
+declare <8 x i32> @llvm.x86.avx512.vpdpwssds.256(<8 x i32>, <8 x i32>, <8 x i32>)
+
+define <8 x i32>@test_int_x86_avx_vpdpwssds_256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpwssds_256:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpwssds %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x53,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpwssds_256:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpwssds %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x53,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpwssds_256:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpwssds %ymm2, %ymm1, %ymm0 # encoding: [0xc4,0xe2,0x75,0x53,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpwssds_256:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpwssds %ymm2, %ymm1, %ymm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x75,0x53,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <8 x i32> @llvm.x86.avx512.vpdpwssds.256(<8 x i32> %x0, <8 x i32> %x1, <8 x i32> %x2)
+  ret <8 x i32> %res
+}
+
+declare <4 x i32> @llvm.x86.avx512.vpdpwssds.128(<4 x i32>, <4 x i32>, <4 x i32>)
+
+define <4 x i32>@test_int_x86_avx_vpdpwssds_128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2) {
+; AVX-X86-LABEL: test_int_x86_avx_vpdpwssds_128:
+; AVX-X86:       # %bb.0:
+; AVX-X86-NEXT:    {vex} vpdpwssds %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x53,0xc2]
+; AVX-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX512-X86-LABEL: test_int_x86_avx_vpdpwssds_128:
+; AVX512-X86:       # %bb.0:
+; AVX512-X86-NEXT:    {vex} vpdpwssds %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x53,0xc2]
+; AVX512-X86-NEXT:    retl # encoding: [0xc3]
+;
+; AVX-X64-LABEL: test_int_x86_avx_vpdpwssds_128:
+; AVX-X64:       # %bb.0:
+; AVX-X64-NEXT:    {vex} vpdpwssds %xmm2, %xmm1, %xmm0 # encoding: [0xc4,0xe2,0x71,0x53,0xc2]
+; AVX-X64-NEXT:    retq # encoding: [0xc3]
+;
+; AVX512-X64-LABEL: test_int_x86_avx_vpdpwssds_128:
+; AVX512-X64:       # %bb.0:
+; AVX512-X64-NEXT:    {vex} vpdpwssds %xmm2, %xmm1, %xmm0 # EVEX TO VEX Compression encoding: [0xc4,0xe2,0x71,0x53,0xc2]
+; AVX512-X64-NEXT:    retq # encoding: [0xc3]
+  %res = call <4 x i32> @llvm.x86.avx512.vpdpwssds.128(<4 x i32> %x0, <4 x i32> %x1, <4 x i32> %x2)
+  ret <4 x i32> %res
+}

llvm/test/CodeGen/X86/avx-vnni/stack-folding-int-avxvnni.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O3 -disable-peephole -mtriple=x86_64-unknown-unknown -mattr=+avxvnni < %s | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-unknown"
+
+declare <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32>, <4 x i32>, <4 x i32>)
+declare <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32>, <8 x i32>, <8 x i32>)
+declare <4 x i32> @llvm.x86.avx512.vpdpwssds.128(<4 x i32>, <4 x i32>, <4 x i32>)
+declare <8 x i32> @llvm.x86.avx512.vpdpwssds.256(<8 x i32>, <8 x i32>, <8 x i32>)
+declare <4 x i32> @llvm.x86.avx512.vpdpbusd.128(<4 x i32>, <4 x i32>, <4 x i32>)
+declare <8 x i32> @llvm.x86.avx512.vpdpbusd.256(<8 x i32>, <8 x i32>, <8 x i32>)
+declare <4 x i32> @llvm.x86.avx512.vpdpbusds.128(<4 x i32>, <4 x i32>, <4 x i32>)
+declare <8 x i32> @llvm.x86.avx512.vpdpbusds.256(<8 x i32>, <8 x i32>, <8 x i32>)
+
+define <4 x i32> @stack_fold_vpdpwssd(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssd:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssd {{[-0-9]+}}(%r{{[sb]}}p), %xmm1, %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpwssd_commuted(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssd_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssd {{[-0-9]+}}(%r{{[sb]}}p), %xmm1, %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpwssd.128(<4 x i32> %a0, <4 x i32> %a2, <4 x i32> %a1)
+  ret <4 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpwssd_256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssd_256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssd {{[-0-9]+}}(%r{{[sb]}}p), %ymm1, %ymm0 # 32-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2)
+  ret <8 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpwssd_256_commuted(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssd_256_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssd {{[-0-9]+}}(%r{{[sb]}}p), %ymm1, %ymm0 # 32-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpwssd.256(<8 x i32> %a0, <8 x i32> %a2, <8 x i32> %a1)
+  ret <8 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpwssds(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssds:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssds {{[-0-9]+}}(%r{{[sb]}}p), %xmm1, %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpwssds.128(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpwssds_commuted(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssds_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssds {{[-0-9]+}}(%r{{[sb]}}p), %xmm1, %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpwssds.128(<4 x i32> %a0, <4 x i32> %a2, <4 x i32> %a1)
+  ret <4 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpwssds_256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssds_256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssds {{[-0-9]+}}(%r{{[sb]}}p), %ymm1, %ymm0 # 32-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpwssds.256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2)
+  ret <8 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpwssds_256_commuted(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpwssds_256_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpwssds {{[-0-9]+}}(%r{{[sb]}}p), %ymm1, %ymm0 # 32-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpwssds.256(<8 x i32> %a0, <8 x i32> %a2, <8 x i32> %a1)
+  ret <8 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpbusd(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusd:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpbusd {{[-0-9]+}}(%r{{[sb]}}p), %xmm1, %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpbusd.128(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpbusd_commuted(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusd_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm2 # 16-byte Reload
+; CHECK-NEXT:    {vex} vpdpbusd %xmm1, %xmm2, %xmm0
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpbusd.128(<4 x i32> %a0, <4 x i32> %a2, <4 x i32> %a1)
+  ret <4 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpbusd_256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusd_256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpbusd {{[-0-9]+}}(%r{{[sb]}}p), %ymm1, %ymm0 # 32-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpbusd.256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2)
+  ret <8 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpbusd_256_commuted(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusd_256_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    vmovdqu {{[-0-9]+}}(%r{{[sb]}}p), %ymm2 # 32-byte Reload
+; CHECK-NEXT:    {vex} vpdpbusd %ymm1, %ymm2, %ymm0
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpbusd.256(<8 x i32> %a0, <8 x i32> %a2, <8 x i32> %a1)
+  ret <8 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpbusds(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusds:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpbusds {{[-0-9]+}}(%r{{[sb]}}p), %xmm1, %xmm0 # 16-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpbusds.128(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %2
+}
+
+define <4 x i32> @stack_fold_vpdpbusds_commuted(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusds_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    vmovdqa {{[-0-9]+}}(%r{{[sb]}}p), %xmm2 # 16-byte Reload
+; CHECK-NEXT:    {vex} vpdpbusds %xmm1, %xmm2, %xmm0
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <4 x i32> @llvm.x86.avx512.vpdpbusds.128(<4 x i32> %a0, <4 x i32> %a2, <4 x i32> %a1)
+  ret <4 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpbusds_256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusds_256:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    {vex} vpdpbusds {{[-0-9]+}}(%r{{[sb]}}p), %ymm1, %ymm0 # 32-byte Folded Reload
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpbusds.256(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2)
+  ret <8 x i32> %2
+}
+
+define <8 x i32> @stack_fold_vpdpbusds_256_commuted(<8 x i32> %a0, <8 x i32> %a1, <8 x i32> %a2) {
+; CHECK-LABEL: stack_fold_vpdpbusds_256_commuted:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    vmovups %ymm2, {{[-0-9]+}}(%r{{[sb]}}p) # 32-byte Spill
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    nop
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    vmovdqu {{[-0-9]+}}(%r{{[sb]}}p), %ymm2 # 32-byte Reload
+; CHECK-NEXT:    {vex} vpdpbusds %ymm1, %ymm2, %ymm0
+; CHECK-NEXT:    retq
+  %1 = tail call <2 x i64> asm sideeffect "nop", "=x,~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15},~{flags}"()
+  %2 = call <8 x i32> @llvm.x86.avx512.vpdpbusds.256(<8 x i32> %a0, <8 x i32> %a2, <8 x i32> %a1)
+  ret <8 x i32> %2
+}

llvm/test/MC/Disassembler/X86/avx_vnni.txt

+# RUN: llvm-mc --disassemble %s -triple=i686-apple-darwin9 | FileCheck %s
+
+# CHECK: {vex} vpdpbusd %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd  268435456(%esp,%esi,8), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd  291(%edi,%eax,4), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  (%eax), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0x30
+
+# CHECK: {vex} vpdpbusd  -1024(,%ebp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusd  4064(%ecx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  -4096(%edx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusd  268435456(%esp,%esi,8), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd  291(%edi,%eax,4), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  (%eax), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0x30
+
+# CHECK: {vex} vpdpbusd  -512(,%ebp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusd  2032(%ecx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  -2048(%edx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpbusds %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds  268435456(%esp,%esi,8), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds  291(%edi,%eax,4), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  (%eax), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0x30
+
+# CHECK: {vex} vpdpbusds  -1024(,%ebp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusds  4064(%ecx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  -4096(%edx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusds  268435456(%esp,%esi,8), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds  291(%edi,%eax,4), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  (%eax), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0x30
+
+# CHECK: {vex} vpdpbusds  -512(,%ebp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusds  2032(%ecx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  -2048(%edx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssd %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd  268435456(%esp,%esi,8), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd  291(%edi,%eax,4), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  (%eax), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0x30
+
+# CHECK: {vex} vpdpwssd  -1024(,%ebp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssd  4064(%ecx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  -4096(%edx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssd  268435456(%esp,%esi,8), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd  291(%edi,%eax,4), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  (%eax), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0x30
+
+# CHECK: {vex} vpdpwssd  -512(,%ebp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssd  2032(%ecx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  -2048(%edx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssds %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds  268435456(%esp,%esi,8), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds  291(%edi,%eax,4), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  (%eax), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0x30
+
+# CHECK: {vex} vpdpwssds  -1024(,%ebp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssds  4064(%ecx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  -4096(%edx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssds  268435456(%esp,%esi,8), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds  291(%edi,%eax,4), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  (%eax), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0x30
+
+# CHECK: {vex} vpdpwssds  -512(,%ebp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssds  2032(%ecx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  -2048(%edx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff
+

llvm/test/MC/Disassembler/X86/intel-syntax-avx_vnni.txt

+# RUN: llvm-mc --disassemble %s -triple=i686 -x86-asm-syntax=intel --output-asm-variant=1 | FileCheck %s
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x55,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x55,0x50,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [eax]
+0xc4,0xe2,0x55,0x50,0x30
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [ecx + 4064]
+0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [edx - 4096]
+0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x51,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x51,0x50,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [eax]
+0xc4,0xe2,0x51,0x50,0x30
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [2*ebp - 512]
+0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [ecx + 2032]
+0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [edx - 2048]
+0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x55,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x55,0x51,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [eax]
+0xc4,0xe2,0x55,0x51,0x30
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [ecx + 4064]
+0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [edx - 4096]
+0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x51,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x51,0x51,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [eax]
+0xc4,0xe2,0x51,0x51,0x30
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [2*ebp - 512]
+0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [ecx + 2032]
+0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [edx - 2048]
+0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x55,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x55,0x52,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [eax]
+0xc4,0xe2,0x55,0x52,0x30
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [ecx + 4064]
+0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [edx - 4096]
+0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x51,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x51,0x52,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [eax]
+0xc4,0xe2,0x51,0x52,0x30
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [2*ebp - 512]
+0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [ecx + 2032]
+0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [edx - 2048]
+0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x55,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x55,0x53,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [eax]
+0xc4,0xe2,0x55,0x53,0x30
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [ecx + 4064]
+0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [edx - 4096]
+0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+0xc4,0xe2,0x51,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+0xc4,0xe2,0x51,0x53,0xb4,0x87,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [eax]
+0xc4,0xe2,0x51,0x53,0x30
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [2*ebp - 512]
+0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [ecx + 2032]
+0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [edx - 2048]
+0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff
+

llvm/test/MC/Disassembler/X86/intel-syntax-x86-64-avx_vnni.txt

+# RUN: llvm-mc --disassemble %s -triple=x86_64 -x86-asm-syntax=intel --output-asm-variant=1 | FileCheck %s
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x55,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x55,0x50,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rip]
+0xc4,0xe2,0x55,0x50,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rcx + 4064]
+0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rdx - 4096]
+0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x51,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x51,0x50,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rip]
+0xc4,0xe2,0x51,0x50,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [2*rbp - 512]
+0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rcx + 2032]
+0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rdx - 2048]
+0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x55,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x55,0x51,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rip]
+0xc4,0xe2,0x55,0x51,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rcx + 4064]
+0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rdx - 4096]
+0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x51,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x51,0x51,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rip]
+0xc4,0xe2,0x51,0x51,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [2*rbp - 512]
+0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rcx + 2032]
+0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rdx - 2048]
+0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x55,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x55,0x52,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rip]
+0xc4,0xe2,0x55,0x52,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rcx + 4064]
+0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rdx - 4096]
+0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x51,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x51,0x52,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rip]
+0xc4,0xe2,0x51,0x52,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [2*rbp - 512]
+0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rcx + 2032]
+0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rdx - 2048]
+0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymm4
+0xc4,0xe2,0x55,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmm4
+0xc4,0xe2,0x51,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x55,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x55,0x53,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rip]
+0xc4,0xe2,0x55,0x53,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rcx + 4064]
+0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rdx - 4096]
+0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+0xc4,0xa2,0x51,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+0xc4,0xc2,0x51,0x53,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rip]
+0xc4,0xe2,0x51,0x53,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [2*rbp - 512]
+0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rcx + 2032]
+0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rdx - 2048]
+0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff
+

llvm/test/MC/Disassembler/X86/x86-64-avx_vnni.txt

+# RUN: llvm-mc --disassemble %s -triple=x86_64-apple-darwin9 | FileCheck %s
+
+# CHECK: {vex} vpdpbusd %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xf4
+
+# CHECK: {vex} vpdpbusd  268435456(%rbp,%r14,8), %ymm5, %ymm6
+0xc4,0xa2,0x55,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd  291(%r8,%rax,4), %ymm5, %ymm6
+0xc4,0xc2,0x55,0x50,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  (%rip), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  -1024(,%rbp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusd  4064(%rcx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  -4096(%rdx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusd  268435456(%rbp,%r14,8), %xmm5, %xmm6
+0xc4,0xa2,0x51,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusd  291(%r8,%rax,4), %xmm5, %xmm6
+0xc4,0xc2,0x51,0x50,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  (%rip), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  -512(,%rbp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusd  2032(%rcx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusd  -2048(%rdx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpbusds %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xf4
+
+# CHECK: {vex} vpdpbusds  268435456(%rbp,%r14,8), %ymm5, %ymm6
+0xc4,0xa2,0x55,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds  291(%r8,%rax,4), %ymm5, %ymm6
+0xc4,0xc2,0x55,0x51,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  (%rip), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  -1024(,%rbp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpbusds  4064(%rcx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  -4096(%rdx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpbusds  268435456(%rbp,%r14,8), %xmm5, %xmm6
+0xc4,0xa2,0x51,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpbusds  291(%r8,%rax,4), %xmm5, %xmm6
+0xc4,0xc2,0x51,0x51,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  (%rip), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  -512(,%rbp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpbusds  2032(%rcx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpbusds  -2048(%rdx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssd %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xf4
+
+# CHECK: {vex} vpdpwssd  268435456(%rbp,%r14,8), %ymm5, %ymm6
+0xc4,0xa2,0x55,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd  291(%r8,%rax,4), %ymm5, %ymm6
+0xc4,0xc2,0x55,0x52,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  (%rip), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  -1024(,%rbp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssd  4064(%rcx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  -4096(%rdx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssd  268435456(%rbp,%r14,8), %xmm5, %xmm6
+0xc4,0xa2,0x51,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssd  291(%r8,%rax,4), %xmm5, %xmm6
+0xc4,0xc2,0x51,0x52,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  (%rip), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  -512(,%rbp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssd  2032(%rcx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssd  -2048(%rdx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff
+
+# CHECK: {vex} vpdpwssds %ymm4, %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds %xmm4, %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xf4
+
+# CHECK: {vex} vpdpwssds  268435456(%rbp,%r14,8), %ymm5, %ymm6
+0xc4,0xa2,0x55,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds  291(%r8,%rax,4), %ymm5, %ymm6
+0xc4,0xc2,0x55,0x53,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  (%rip), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  -1024(,%rbp,2), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff
+
+# CHECK: {vex} vpdpwssds  4064(%rcx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  -4096(%rdx), %ymm5, %ymm6
+0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff
+
+# CHECK: {vex} vpdpwssds  268435456(%rbp,%r14,8), %xmm5, %xmm6
+0xc4,0xa2,0x51,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10
+
+# CHECK: {vex} vpdpwssds  291(%r8,%rax,4), %xmm5, %xmm6
+0xc4,0xc2,0x51,0x53,0xb4,0x80,0x23,0x01,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  (%rip), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0x35,0x00,0x00,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  -512(,%rbp,2), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff
+
+# CHECK: {vex} vpdpwssds  2032(%rcx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00
+
+# CHECK: {vex} vpdpwssds  -2048(%rdx), %xmm5, %xmm6
+0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff
+

llvm/test/MC/X86/avx_vnni-encoding.s

+// RUN: llvm-mc -triple i686-unknown-unknown -mattr=+avxvnni --show-encoding < %s  | FileCheck %s
+
+// CHECK: {vex} vpdpbusd %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xf4]
+          {vex} vpdpbusd %ymm4, %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xf4]
+          {vex} vpdpbusd %xmm4, %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  268435456(%esp,%esi,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd  268435456(%esp,%esi,8), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  291(%edi,%eax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd  291(%edi,%eax,4), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  (%eax), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x30]
+          {vex} vpdpbusd  (%eax), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  -1024(,%ebp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusd  -1024(,%ebp,2), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  4064(%ecx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusd  4064(%ecx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  -4096(%edx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusd  -4096(%edx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  268435456(%esp,%esi,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd  268435456(%esp,%esi,8), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  291(%edi,%eax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd  291(%edi,%eax,4), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  (%eax), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x30]
+          {vex} vpdpbusd  (%eax), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  -512(,%ebp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusd  -512(,%ebp,2), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  2032(%ecx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusd  2032(%ecx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  -2048(%edx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusd  -2048(%edx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xf4]
+          {vex} vpdpbusds %ymm4, %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xf4]
+          {vex} vpdpbusds %xmm4, %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  268435456(%esp,%esi,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds  268435456(%esp,%esi,8), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  291(%edi,%eax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds  291(%edi,%eax,4), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  (%eax), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x30]
+          {vex} vpdpbusds  (%eax), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  -1024(,%ebp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusds  -1024(,%ebp,2), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  4064(%ecx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusds  4064(%ecx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  -4096(%edx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusds  -4096(%edx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  268435456(%esp,%esi,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds  268435456(%esp,%esi,8), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  291(%edi,%eax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds  291(%edi,%eax,4), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  (%eax), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x30]
+          {vex} vpdpbusds  (%eax), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  -512(,%ebp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusds  -512(,%ebp,2), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  2032(%ecx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusds  2032(%ecx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  -2048(%edx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusds  -2048(%edx), %xmm5, %xmm6
+
+// CHECK: vpdpwssd %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xf4]
+          {vex} vpdpwssd %ymm4, %ymm5, %ymm6
+
+// CHECK: vpdpwssd %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xf4]
+          {vex} vpdpwssd %xmm4, %xmm5, %xmm6
+
+// CHECK: vpdpwssd  268435456(%esp,%esi,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd  268435456(%esp,%esi,8), %ymm5, %ymm6
+
+// CHECK: vpdpwssd  291(%edi,%eax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd  291(%edi,%eax,4), %ymm5, %ymm6
+
+// CHECK: vpdpwssd  (%eax), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x30]
+          {vex} vpdpwssd  (%eax), %ymm5, %ymm6
+
+// CHECK: vpdpwssd  -1024(,%ebp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssd  -1024(,%ebp,2), %ymm5, %ymm6
+
+// CHECK: vpdpwssd  4064(%ecx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssd  4064(%ecx), %ymm5, %ymm6
+
+// CHECK: vpdpwssd  -4096(%edx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssd  -4096(%edx), %ymm5, %ymm6
+
+// CHECK: vpdpwssd  268435456(%esp,%esi,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd  268435456(%esp,%esi,8), %xmm5, %xmm6
+
+// CHECK: vpdpwssd  291(%edi,%eax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd  291(%edi,%eax,4), %xmm5, %xmm6
+
+// CHECK: vpdpwssd  (%eax), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x30]
+          {vex} vpdpwssd  (%eax), %xmm5, %xmm6
+
+// CHECK: vpdpwssd  -512(,%ebp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssd  -512(,%ebp,2), %xmm5, %xmm6
+
+// CHECK: vpdpwssd  2032(%ecx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssd  2032(%ecx), %xmm5, %xmm6
+
+// CHECK: vpdpwssd  -2048(%edx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssd  -2048(%edx), %xmm5, %xmm6
+
+// CHECK: vpdpwssds %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xf4]
+          {vex} vpdpwssds %ymm4, %ymm5, %ymm6
+
+// CHECK: vpdpwssds %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xf4]
+          {vex} vpdpwssds %xmm4, %xmm5, %xmm6
+
+// CHECK: vpdpwssds  268435456(%esp,%esi,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds  268435456(%esp,%esi,8), %ymm5, %ymm6
+
+// CHECK: vpdpwssds  291(%edi,%eax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds  291(%edi,%eax,4), %ymm5, %ymm6
+
+// CHECK: vpdpwssds  (%eax), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x30]
+          {vex} vpdpwssds  (%eax), %ymm5, %ymm6
+
+// CHECK: vpdpwssds  -1024(,%ebp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssds  -1024(,%ebp,2), %ymm5, %ymm6
+
+// CHECK: vpdpwssds  4064(%ecx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssds  4064(%ecx), %ymm5, %ymm6
+
+// CHECK: vpdpwssds  -4096(%edx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssds  -4096(%edx), %ymm5, %ymm6
+
+// CHECK: vpdpwssds  268435456(%esp,%esi,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds  268435456(%esp,%esi,8), %xmm5, %xmm6
+
+// CHECK: vpdpwssds  291(%edi,%eax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds  291(%edi,%eax,4), %xmm5, %xmm6
+
+// CHECK: vpdpwssds  (%eax), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x30]
+          {vex} vpdpwssds  (%eax), %xmm5, %xmm6
+
+// CHECK: vpdpwssds  -512(,%ebp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssds  -512(,%ebp,2), %xmm5, %xmm6
+
+// CHECK: vpdpwssds  2032(%ecx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssds  2032(%ecx), %xmm5, %xmm6
+
+// CHECK: vpdpwssds  -2048(%edx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssds  -2048(%edx), %xmm5, %xmm6
+

llvm/test/MC/X86/intel-syntax-avx_vnni.s

+// RUN: llvm-mc -triple i686-unknown-unknown -mattr=+avxvnni -x86-asm-syntax=intel -output-asm-variant=1 --show-encoding %s | FileCheck %s
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xf4]
+          {vex} vpdpbusd ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xf4]
+          {vex} vpdpbusd xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x30]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [eax]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [ecx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [ecx + 4064]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [edx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [edx - 4096]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x30]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [eax]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [2*ebp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [2*ebp - 512]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [ecx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [ecx + 2032]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [edx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [edx - 2048]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xf4]
+          {vex} vpdpbusds ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xf4]
+          {vex} vpdpbusds xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x30]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [eax]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [ecx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [ecx + 4064]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [edx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [edx - 4096]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x30]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [eax]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [2*ebp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [2*ebp - 512]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [ecx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [ecx + 2032]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [edx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [edx - 2048]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xf4]
+          {vex} vpdpwssd ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xf4]
+          {vex} vpdpwssd xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x30]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [eax]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [ecx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [ecx + 4064]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [edx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [edx - 4096]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x30]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [eax]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [2*ebp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [2*ebp - 512]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [ecx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [ecx + 2032]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [edx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [edx - 2048]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xf4]
+          {vex} vpdpwssds ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xf4]
+          {vex} vpdpwssds xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x30]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [eax]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [2*ebp - 1024]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [ecx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [ecx + 4064]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [edx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [edx - 4096]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb4,0xf4,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [esp + 8*esi + 268435456]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb4,0x87,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [edi + 4*eax + 291]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [eax]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x30]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [eax]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [2*ebp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [2*ebp - 512]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [ecx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [ecx + 2032]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [edx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [edx - 2048]
+

llvm/test/MC/X86/intel-syntax-x86-64-avx_vnni.s

+// RUN: llvm-mc -triple x86_64-unknown-unknown -mattr=+avxvnni -x86-asm-syntax=intel -output-asm-variant=1 --show-encoding %s | FileCheck %s
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xf4]
+          {vex} vpdpbusd ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xf4]
+          {vex} vpdpbusd xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x55,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x55,0x50,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rip]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rcx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rcx + 4064]
+
+// CHECK: {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rdx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusd ymm6, ymm5, ymmword ptr [rdx - 4096]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x51,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x51,0x50,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rip]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [2*rbp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [2*rbp - 512]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rcx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rcx + 2032]
+
+// CHECK: {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rdx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusd xmm6, xmm5, xmmword ptr [rdx - 2048]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xf4]
+          {vex} vpdpbusds ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xf4]
+          {vex} vpdpbusds xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x55,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x55,0x51,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rip]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rcx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rcx + 4064]
+
+// CHECK: {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rdx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusds ymm6, ymm5, ymmword ptr [rdx - 4096]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x51,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x51,0x51,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rip]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [2*rbp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [2*rbp - 512]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rcx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rcx + 2032]
+
+// CHECK: {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rdx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusds xmm6, xmm5, xmmword ptr [rdx - 2048]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xf4]
+          {vex} vpdpwssd ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xf4]
+          {vex} vpdpwssd xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x55,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x55,0x52,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rip]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rcx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rcx + 4064]
+
+// CHECK: {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rdx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssd ymm6, ymm5, ymmword ptr [rdx - 4096]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x51,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x51,0x52,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rip]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [2*rbp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [2*rbp - 512]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rcx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rcx + 2032]
+
+// CHECK: {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rdx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssd xmm6, xmm5, xmmword ptr [rdx - 2048]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymm4
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xf4]
+          {vex} vpdpwssds ymm6, ymm5, ymm4
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmm4
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xf4]
+          {vex} vpdpwssds xmm6, xmm5, xmm4
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x55,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x55,0x53,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rip]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [2*rbp - 1024]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rcx + 4064]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rcx + 4064]
+
+// CHECK: {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rdx - 4096]
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssds ymm6, ymm5, ymmword ptr [rdx - 4096]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+// CHECK: encoding: [0xc4,0xa2,0x51,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rbp + 8*r14 + 268435456]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+// CHECK: encoding: [0xc4,0xc2,0x51,0x53,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [r8 + 4*rax + 291]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rip]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rip]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [2*rbp - 512]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [2*rbp - 512]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rcx + 2032]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rcx + 2032]
+
+// CHECK: {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rdx - 2048]
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssds xmm6, xmm5, xmmword ptr [rdx - 2048]
+

llvm/test/MC/X86/x86-64-avx_vnni-encoding.s

+// RUN: llvm-mc -triple x86_64-unknown-unknown -mattr=+avxvnni --show-encoding < %s  | FileCheck %s
+
+// CHECK: {vex} vpdpbusd %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xf4]
+          {vex} vpdpbusd %ymm4, %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xf4]
+          {vex} vpdpbusd %xmm4, %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  268435456(%rbp,%r14,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xa2,0x55,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd  268435456(%rbp,%r14,8), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  291(%r8,%rax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xc2,0x55,0x50,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd  291(%r8,%rax,4), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  (%rip), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusd  (%rip), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  -1024(,%rbp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusd  -1024(,%rbp,2), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  4064(%rcx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusd  4064(%rcx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  -4096(%rdx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x50,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusd  -4096(%rdx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusd  268435456(%rbp,%r14,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xa2,0x51,0x50,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusd  268435456(%rbp,%r14,8), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  291(%r8,%rax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xc2,0x51,0x50,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusd  291(%r8,%rax,4), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  (%rip), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusd  (%rip), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  -512(,%rbp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusd  -512(,%rbp,2), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  2032(%rcx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusd  2032(%rcx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusd  -2048(%rdx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x50,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusd  -2048(%rdx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xf4]
+          {vex} vpdpbusds %ymm4, %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xf4]
+          {vex} vpdpbusds %xmm4, %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  268435456(%rbp,%r14,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xa2,0x55,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds  268435456(%rbp,%r14,8), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  291(%r8,%rax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xc2,0x55,0x51,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds  291(%r8,%rax,4), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  (%rip), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusds  (%rip), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  -1024(,%rbp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpbusds  -1024(,%rbp,2), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  4064(%rcx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpbusds  4064(%rcx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  -4096(%rdx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x51,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpbusds  -4096(%rdx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpbusds  268435456(%rbp,%r14,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xa2,0x51,0x51,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpbusds  268435456(%rbp,%r14,8), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  291(%r8,%rax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xc2,0x51,0x51,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpbusds  291(%r8,%rax,4), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  (%rip), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpbusds  (%rip), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  -512(,%rbp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpbusds  -512(,%rbp,2), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  2032(%rcx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpbusds  2032(%rcx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpbusds  -2048(%rdx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x51,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpbusds  -2048(%rdx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xf4]
+          {vex} vpdpwssd %ymm4, %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xf4]
+          {vex} vpdpwssd %xmm4, %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd  268435456(%rbp,%r14,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xa2,0x55,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd  268435456(%rbp,%r14,8), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd  291(%r8,%rax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xc2,0x55,0x52,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd  291(%r8,%rax,4), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd  (%rip), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssd  (%rip), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd  -1024(,%rbp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssd  -1024(,%rbp,2), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd  4064(%rcx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssd  4064(%rcx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd  -4096(%rdx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x52,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssd  -4096(%rdx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssd  268435456(%rbp,%r14,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xa2,0x51,0x52,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssd  268435456(%rbp,%r14,8), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd  291(%r8,%rax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xc2,0x51,0x52,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssd  291(%r8,%rax,4), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd  (%rip), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssd  (%rip), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd  -512(,%rbp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssd  -512(,%rbp,2), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd  2032(%rcx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssd  2032(%rcx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssd  -2048(%rdx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x52,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssd  -2048(%rdx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds %ymm4, %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xf4]
+          {vex} vpdpwssds %ymm4, %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds %xmm4, %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xf4]
+          {vex} vpdpwssds %xmm4, %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds  268435456(%rbp,%r14,8), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xa2,0x55,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds  268435456(%rbp,%r14,8), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds  291(%r8,%rax,4), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xc2,0x55,0x53,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds  291(%r8,%rax,4), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds  (%rip), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssds  (%rip), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds  -1024(,%rbp,2), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0x34,0x6d,0x00,0xfc,0xff,0xff]
+          {vex} vpdpwssds  -1024(,%rbp,2), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds  4064(%rcx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb1,0xe0,0x0f,0x00,0x00]
+          {vex} vpdpwssds  4064(%rcx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds  -4096(%rdx), %ymm5, %ymm6
+// CHECK: encoding: [0xc4,0xe2,0x55,0x53,0xb2,0x00,0xf0,0xff,0xff]
+          {vex} vpdpwssds  -4096(%rdx), %ymm5, %ymm6
+
+// CHECK: {vex} vpdpwssds  268435456(%rbp,%r14,8), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xa2,0x51,0x53,0xb4,0xf5,0x00,0x00,0x00,0x10]
+          {vex} vpdpwssds  268435456(%rbp,%r14,8), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds  291(%r8,%rax,4), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xc2,0x51,0x53,0xb4,0x80,0x23,0x01,0x00,0x00]
+          {vex} vpdpwssds  291(%r8,%rax,4), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds  (%rip), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x35,0x00,0x00,0x00,0x00]
+          {vex} vpdpwssds  (%rip), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds  -512(,%rbp,2), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0x34,0x6d,0x00,0xfe,0xff,0xff]
+          {vex} vpdpwssds  -512(,%rbp,2), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds  2032(%rcx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb1,0xf0,0x07,0x00,0x00]
+          {vex} vpdpwssds  2032(%rcx), %xmm5, %xmm6
+
+// CHECK: {vex} vpdpwssds  -2048(%rdx), %xmm5, %xmm6
+// CHECK: encoding: [0xc4,0xe2,0x51,0x53,0xb2,0x00,0xf8,0xff,0xff]
+          {vex} vpdpwssds  -2048(%rdx), %xmm5, %xmm6
+