[X86] Support AMX-FP16

[xiangzhangllvm]

Summary

Support Intel AMX-FP16 instruction

[pengfei]

clang/lib/Basic/Targets/X86.cpp

785

This can be removed.

clang/lib/Headers/immintrin.h

511

Remove

513

Remove

clang/lib/Sema/SemaChecking.cpp

5031

This can be merged together?

[pengfei]

clang/test/CodeGen/amx_fp16.c

8

Remove

clang/test/CodeGen/amx_fp16_errors.c

7 ↗

(On Diff #467683)

Remove

llvm/lib/Support/X86TargetParser.cpp

586

It should not relate to AVX512FP16.

llvm/lib/Target/X86/X86ISelLowering.cpp

36990–36991

Format the code?

llvm/test/CodeGen/X86/amx_fp16_intrinsics.ll

2

Maybe auto gen it?

2

Why need +avx512f

2

Why need -O0?

6

The comment seems meaningless.

10–11

ditto here.

16–17

ditto here.

[pengfei]

clang/test/Driver/x86-target-features.c

293

Change to --target=i386

294

Don't need here.

295–296

ditto.

llvm/include/llvm/IR/IntrinsicsX86.td

5121

Remove

[RKSimon]

clang/test/CodeGen/amx_fp16_errors.c

2 ↗

(On Diff #467683)

Add 32-bit test coverage to ensure the intrinsics aren't visible?

[xiangzhangllvm]

clang/test/CodeGen/amx_fp16_errors.c

2 ↗

(On Diff #467683)

That is better, but let me merge this test into X86/amx_errors.c first : )

llvm/lib/Target/X86/X86ISelLowering.cpp

36990–36991

Make sense, but let's sync with the upper code ? Seems that style is good to reduce line num of the big file. All other comments will be updated soon. thanks a lot!

[LuoYuanke]

llvm/lib/Target/X86/X86ISelLowering.cpp

36986

This code may be merged to int8 and bf16 case.

case X86::PTDPBSSD:
case X86::PTDPBSUD:
case X86::PTDPBUSD:
case X86::PTDPBUUD:
case X86::PTDPBF16PS:
case X86::PTDPFP16PS:

[xiangzhangllvm]

clang/test/Driver/x86-target-features.c

293

It is not good for amx use i386 testing. We need to update the other amx too.

[xiangzhangllvm]

llvm/lib/Target/X86/X86ISelLowering.cpp

36986

Good catch!

[RKSimon]

clang/docs/ReleaseNotes.rst

598

Probably mention that this is the for the _tile_dpfp16ps instruction?

llvm/test/MC/Disassembler/X86/x86-64AmxTileFP16-att.txt

4 ↗

(On Diff #468130)

in some MC tests we handle att + intel tests in the same file - should we be doing that for all new tests?

[LuoYuanke]

llvm/test/MC/X86/x86-64AmxFP16-att.s

2 ↗

(On Diff #468130)

There is "test/MC/X86/AMX/" folder. Probably move the test case to that folder or maybe merge the test case to test/MC/X86/AMX/x86-64-amx-bf16-att.s

[xiangzhangllvm]

clang/docs/ReleaseNotes.rst

598

Yes, that is more clear!

llvm/test/MC/Disassembler/X86/x86-64AmxTileFP16-att.txt

4 ↗

(On Diff #468130)

That is make sense before, but we begin to use tool to auto generate the tests, it is easy to split them for tool, and we have do that for x86-64-avx512bf16-att/intel.txt x86-64-avx512bf16vl-att/intel.txt x86-64-avx512vp2intersectvl-att/intel.txt and also KEYLOCKER tests.

llvm/test/MC/X86/x86-64AmxFP16-att.s

2 ↗

(On Diff #468130)

let me put it into "test/MC/X86/AMX/" folder. And keep the tools generated file. That is more clear.

[RKSimon]

llvm/test/MC/X86/AMX/x86-64-amx-fp16-att.s

6

merge att/intel testing into the same file and use --check-prefix to test them

[RKSimon]

llvm/test/MC/Disassembler/X86/x86-64AmxTileFP16-att.txt

4 ↗

(On Diff #468130)

OK - I don't suppose you could ads att/intel functionality to your tool?

[xiangzhangllvm]

llvm/test/MC/X86/AMX/x86-64-amx-fp16-att.s

6

Yes, that is our previous action, I think the most benefit is that we can easy to cmp them for same encoding (put them together) not reduce file number.
but now we are try to use tools auto generate/verify them. So split them is more easy/clear for tools generating/verify. (I think it is complex to let tools “by turns” generate same encoding for different style instructions.)
So should we change the old way to let tools easy to auto-gen tests? (And I found we have split them in llvm/test/MC/X86/AMX/ before.)
Thanks again for careful reviewing.

[xiangzhangllvm]

llvm/test/MC/Disassembler/X86/x86-64AmxTileFP16-att.txt

4 ↗

(On Diff #468130)

The tools is not developed by me, I just use it : )

[LuoYuanke]

llvm/test/MC/Disassembler/X86/x86-64AmxTileFP16-intel.txt

1 ↗

(On Diff #468421)

Move the test case to test/MC/Disassembler/X86/AMX/?

[xiangzhangllvm]

llvm/test/MC/Disassembler/X86/x86-64AmxTileFP16-att.txt

4 ↗

(On Diff #468130)

The tools is not developed by me, I just use it : )

Let me sync with the tools' developer, thanks : )

[LuoYuanke]

llvm/test/MC/X86/AMX/x86-64-amx-fp16-att.s

6

This is encoding test, it seems we are not able to merge them into one file as below?

// RUN: llvm-mc -triple x86_64-unknown-unknown --show-encoding %s | FileCheck %s --check-prefix=ATT-CHECK
// RUN: llvm-mc -triple x86_64-unknown-unknown -x86-asm-syntax=intel -output-asm-variant=1 --show-encoding %s | FileCheck %s --check-prefix=INTEL-CHECK

// ATT-CHECK:      tdpfp16ps       %tmm5, %tmm4, %tmm3
// ATT-CHECK: encoding: [0xc4,0xe2,0x53,0x5c,0xdc]
               tdpfp16ps       %tmm5, %tmm4, %tmm3

// INTEL-CHECK:      tdpfp16ps       tmm3, tmm4, tmm5
// INTEL-CHECK: encoding: [0xc4,0xe2,0x53,0x5c,0xdc]
               tdpfp16ps       tmm3, tmm4, tmm5

[pengfei]

llvm/test/MC/X86/AMX/x86-64-amx-fp16-att.s

6

Yes, the tool just fold disassemble test.

[xiangzhangllvm]

yes, so the *.s must be split with intel and att.
So let make same way for disassemble.
Let me first move them into AMX directory.

[xiangzhangllvm]

merge disassembler tests

[pengfei]

LGTM.

[pengfei]

clang/include/clang/Driver/Options.td

4527

This seems a typo?

4531–4532

Better to move them below amx_bf16 for ordering.

clang/lib/Basic/Targets/X86.cpp

891

ditto.

[xiangzhangllvm]

clang/include/clang/Driver/Options.td

4527

I think so, should fix in another patch.

4531–4532

let me reorder it when commit. thanks!

[LuoYuanke]

clang/include/clang/Driver/Options.td

4531–4532

I fixed it at rG3770d2b9cad9

[xiangzhangllvm]

Hi @RKSimon , I sync with Freddy who also use this tool, we decide to merge the disassembler tests. thanks : )

clang/include/clang/Driver/Options.td

4531–4532

quick action! : )

clang/docs/ReleaseNotes.rst

 -----------------------
 - ``sifive-7-rv32`` and ``sifive-7-rv64`` are no longer supported for ``-mcpu``.
   Use ``sifive-e76``, ``sifive-s76``, or ``sifive-u74`` instead.
 
 X86 Support in Clang
 --------------------
 - Support ``-mindirect-branch-cs-prefix`` for call and jmp to indirect thunk.
 - Fix 32-bit ``__fastcall`` and ``__vectorcall`` ABI mismatch with MSVC.
+- Add ISA of ``AMX-FP16`` which support ``_tile_dpfp16ps``.
 - Switch ``AVX512-BF16`` intrinsics types from ``short`` to ``__bf16``.
 - Add support for ``PREFETCHI`` instructions.
 
 DWARF Support in Clang
 ----------------------
 
 Arm and AArch64 Support in Clang
 --------------------------------

clang/include/clang/Basic/BuiltinsX86_64.def

 TARGET_BUILTIN(__builtin_ia32_tdpbsud, "vIUcIUcIUc", "n", "amx-int8")
 TARGET_BUILTIN(__builtin_ia32_tdpbusd, "vIUcIUcIUc", "n", "amx-int8")
 TARGET_BUILTIN(__builtin_ia32_tdpbuud, "vIUcIUcIUc", "n", "amx-int8")
 TARGET_BUILTIN(__builtin_ia32_tdpbf16ps, "vIUcIUcIUc", "n", "amx-bf16")
 TARGET_BUILTIN(__builtin_ia32_ptwrite64, "vUOi", "n", "ptwrite")
 
 TARGET_BUILTIN(__builtin_ia32_prefetchi, "vvC*Ui", "nc", "prefetchi")
 
+// AMX_FP16 FP16
+TARGET_BUILTIN(__builtin_ia32_tdpfp16ps, "vIUcIUcIUc", "n", "amx-fp16")
+
 #undef BUILTIN
 #undef TARGET_BUILTIN
 #undef TARGET_HEADER_BUILTIN

clang/include/clang/Driver/Options.td

 def mmmx : Flag<["-"], "mmmx">, Group<m_x86_Features_Group>;
 def mno_mmx : Flag<["-"], "mno-mmx">, Group<m_x86_Features_Group>;
 def m3dnow : Flag<["-"], "m3dnow">, Group<m_x86_Features_Group>;
 def mno_3dnow : Flag<["-"], "mno-3dnow">, Group<m_x86_Features_Group>;
 def m3dnowa : Flag<["-"], "m3dnowa">, Group<m_x86_Features_Group>;
 def mno_3dnowa : Flag<["-"], "mno-3dnowa">, Group<m_x86_Features_Group>;
 def mamx_bf16 : Flag<["-"], "mamx-bf16">, Group<m_x86_Features_Group>;
 def mno_amx_bf16 : Flag<["-"], "mno-amx-bf16">, Group<m_x86_Features_Group>;
+def mamx_fp16 : Flag<["-"], "mamx-fp16">, Group<m_x86_Features_Group>;
+def mno_amx_fp16 : Flag<["-"], "mno-amx-fp16">, Group<m_x86_Features_Group>;
 def mamx_int8 : Flag<["-"], "mamx-int8">, Group<m_x86_Features_Group>;
 def mno_amx_int8 : Flag<["-"], "mno-amx-int8">, Group<m_x86_Features_Group>;
 def mamx_tile : Flag<["-"], "mamx-tile">, Group<m_x86_Features_Group>;
 def mno_amx_tile : Flag<["-"], "mno-amx-tile">, Group<m_x86_Features_Group>;
 def msse : Flag<["-"], "msse">, Group<m_x86_Features_Group>;
 def mno_sse : Flag<["-"], "mno-sse">, Group<m_x86_Features_Group>;
 def msse2 : Flag<["-"], "msse2">, Group<m_x86_Features_Group>;
 def mno_sse2 : Flag<["-"], "mno-sse2">, Group<m_x86_Features_Group>;
 def msse3 : Flag<["-"], "msse3">, Group<m_x86_Features_Group>;
 def mno_sse3 : Flag<["-"], "mno-sse3">, Group<m_x86_Features_Group>;
 def mssse3 : Flag<["-"], "mssse3">, Group<m_x86_Features_Group>;
 def mno_ssse3 : Flag<["-"], "mno-ssse3">, Group<m_x86_Features_Group>;
 def msse4_1 : Flag<["-"], "msse4.1">, Group<m_x86_Features_Group>;
 def mno_sse4_1 : Flag<["-"], "mno-sse4.1">, Group<m_x86_Features_Group>;

clang/lib/Basic/Targets/X86.h

class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo {
   bool HasLAHFSAHF = false;
   bool HasWBNOINVD = false;
   bool HasWAITPKG = false;
   bool HasMOVDIRI = false;
   bool HasMOVDIR64B = false;
   bool HasPTWRITE = false;
   bool HasINVPCID = false;
   bool HasENQCMD = false;
+  bool HasAMXFP16 = false;
   bool HasKL = false;      // For key locker
   bool HasWIDEKL = false; // For wide key locker
   bool HasHRESET = false;
   bool HasAVXVNNI = false;
   bool HasAMXTILE = false;
   bool HasAMXINT8 = false;
   bool HasAMXBF16 = false;
   bool HasSERIALIZE = false;

clang/lib/Basic/Targets/X86.cpp

for (const auto &Feature : Features) {
     } else if (Feature == "+invpcid") {
       HasINVPCID = true;
     } else if (Feature == "+enqcmd") {
       HasENQCMD = true;
     } else if (Feature == "+hreset") {
       HasHRESET = true;
     } else if (Feature == "+amx-bf16") {
       HasAMXBF16 = true;
+    } else if (Feature == "+amx-fp16") {
+      HasAMXFP16 = true;
     } else if (Feature == "+amx-int8") {
       HasAMXINT8 = true;
     } else if (Feature == "+amx-tile") {
       HasAMXTILE = true;
     } else if (Feature == "+avxvnni") {
       HasAVXVNNI = true;
     } else if (Feature == "+serialize") {
       HasSERIALIZE = true;
void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
   if (HasHRESET)
     Builder.defineMacro("__HRESET__");
   if (HasAMXTILE)
     Builder.defineMacro("__AMXTILE__");
   if (HasAMXINT8)
     Builder.defineMacro("__AMXINT8__");
   if (HasAMXBF16)
     Builder.defineMacro("__AMXBF16__");
+  if (HasAMXFP16)
+    Builder.defineMacro("__AMXFP16__");
   if (HasAVXVNNI)
     Builder.defineMacro("__AVXVNNI__");
   if (HasSERIALIZE)
     Builder.defineMacro("__SERIALIZE__");
   if (HasTSXLDTRK)
     Builder.defineMacro("__TSXLDTRK__");
   if (HasUINTR)
     Builder.defineMacro("__UINTR__");
   if (HasCRC32)

 
 bool X86TargetInfo::isValidFeatureName(StringRef Name) const {
   return llvm::StringSwitch<bool>(Name)
       .Case("3dnow", true)
       .Case("3dnowa", true)
       .Case("adx", true)
       .Case("aes", true)
       .Case("amx-bf16", true)
+      .Case("amx-fp16", true)
       .Case("amx-int8", true)
       .Case("amx-tile", true)
       .Case("avx", true)
       .Case("avx2", true)
       .Case("avx512f", true)
       .Case("avx512cd", true)
       .Case("avx512vpopcntdq", true)
       .Case("avx512vnni", true)
       .Case("avx512bf16", true)
       .Case("avx512er", true)
       .Case("avx512fp16", true)
return llvm::StringSwitch<bool>(Name)
       .Default(false);
 }
 
 bool X86TargetInfo::hasFeature(StringRef Feature) const {
   return llvm::StringSwitch<bool>(Feature)
       .Case("adx", HasADX)
       .Case("aes", HasAES)
       .Case("amx-bf16", HasAMXBF16)
+      .Case("amx-fp16", HasAMXFP16)
       .Case("amx-int8", HasAMXINT8)
       .Case("amx-tile", HasAMXTILE)
       .Case("avxvnni", HasAVXVNNI)
       .Case("avx", SSELevel >= AVX)
       .Case("avx2", SSELevel >= AVX2)
       .Case("avx512f", SSELevel >= AVX512F)
       .Case("avx512cd", HasAVX512CD)
       .Case("avx512vpopcntdq", HasAVX512VPOPCNTDQ)

clang/lib/Headers/CMakeLists.txt

 set(webassembly_files
   wasm_simd128.h
   )
 
 set(x86_files
 # Intrinsics
   adxintrin.h
   ammintrin.h
+  amxfp16intrin.h
   amxintrin.h
   avx2intrin.h
   avx512bf16intrin.h
   avx512bitalgintrin.h
   avx512bwintrin.h
   avx512cdintrin.h
   avx512dqintrin.h
   avx512erintrin.h

clang/lib/Headers/amxfp16intrin.h

+/*===------------- amxfp16intrin.h - AMX_FP16 intrinsics -*- C++ -*---------===
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===------------------------------------------------------------------------===
+ */
+
+#ifndef __IMMINTRIN_H
+#error "Never use <amxfp16intrin.h> directly; use <immintrin.h> instead."
+#endif /* __IMMINTRIN_H */
+
+#ifndef __AMX_FP16INTRIN_H
+#define __AMX_FP16INTRIN_H
+#ifdef __x86_64__
+
+/// Compute dot-product of FP16 (16-bit) floating-point pairs in tiles \a a
+///    and \a b, accumulating the intermediate single-precision (32-bit)
+///    floating-point elements with elements in \a dst, and store the 32-bit
+///    result back to tile \a dst.
+///
+/// \headerfile <x86intrin.h>
+///
+/// \code
+/// void _tile_dpfp16ps (__tile dst, __tile a, __tile b)
+/// \endcode
+///
+/// \code{.operation}
+/// FOR m := 0 TO dst.rows - 1
+/// tmp := dst.row[m]
+/// FOR k := 0 TO (a.colsb / 4) - 1
+///   FOR n := 0 TO (dst.colsb / 4) - 1
+///     tmp.fp32[n] += FP32(a.row[m].fp16[2*k+0]) *
+///         FP32(b.row[k].fp16[2*n+0])
+///     tmp.fp32[n] += FP32(a.row[m].fp16[2*k+1]) *
+///         FP32(b.row[k].fp16[2*n+1])
+///   ENDFOR
+/// ENDFOR
+/// write_row_and_zero(dst, m, tmp, dst.colsb)
+/// ENDFOR
+/// zero_upper_rows(dst, dst.rows)
+/// zero_tileconfig_start()
+/// \endcode
+///
+/// This intrinsic corresponds to the \c TDPFP16PS instruction.
+///
+/// \param dst
+///    The destination tile. Max size is 1024 Bytes.
+/// \param a
+///    The 1st source tile. Max size is 1024 Bytes.
+/// \param b
+///    The 2nd source tile. Max size is 1024 Bytes.
+#define _tile_dpfp16ps(dst, a, b)                                \
+  __builtin_ia32_tdpfp16ps(dst, a, b)
+
+#endif /* __x86_64__ */
+#endif /* __AMX_FP16INTRIN_H */

clang/lib/Headers/cpuid.h

 #define bit_AMXBF16       0x00400000
 #define bit_AVX512FP16    0x00800000
 #define bit_AMXTILE       0x01000000
 #define bit_AMXINT8       0x02000000
 
 /* Features in %eax for leaf 7 sub-leaf 1 */
 #define bit_AVXVNNI       0x00000010
 #define bit_AVX512BF16    0x00000020
+#define bit_AMXFP16       0x00200000
 #define bit_HRESET        0x00400000
 
 /* Features in %edx for leaf 7 sub-leaf 1 */
 #define bit_PREFETCHI     0x00004000
 
 /* Features in %eax for leaf 13 sub-leaf 1 */
 #define bit_XSAVEOPT    0x00000001
 #define bit_XSAVEC      0x00000002

clang/lib/Headers/immintrin.h

#if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__PTWRITE__)
 #include <ptwriteintrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__INVPCID__)
 #include <invpcidintrin.h>
 #endif
+#if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
+    defined(__AMXFP16__)
+#include <amxfp16intrin.h>
+#endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__KL__) || defined(__WIDEKL__)
 #include <keylockerintrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__AMXTILE__) || defined(__AMXINT8__) || defined(__AMXBF16__)

clang/lib/Sema/SemaChecking.cpp

bool Sema::CheckX86BuiltinTileArguments(unsigned BuiltinID, CallExpr *TheCall) {
   case X86::BI__builtin_ia32_tilestored64:
   case X86::BI__builtin_ia32_tilezero:
     return CheckX86BuiltinTileArgumentsRange(TheCall, 0);
   case X86::BI__builtin_ia32_tdpbssd:
   case X86::BI__builtin_ia32_tdpbsud:
   case X86::BI__builtin_ia32_tdpbusd:
   case X86::BI__builtin_ia32_tdpbuud:
   case X86::BI__builtin_ia32_tdpbf16ps:
+  case X86::BI__builtin_ia32_tdpfp16ps:
     return CheckX86BuiltinTileRangeAndDuplicate(TheCall, {0, 1, 2});
   }
 }
 static bool isX86_32Builtin(unsigned BuiltinID) {
   // These builtins only work on x86-32 targets.
   switch (BuiltinID) {
   case X86::BI__builtin_ia32_readeflags_u32:
   case X86::BI__builtin_ia32_writeeflags_u32:
     return true;
   }

clang/test/CodeGen/X86/amx_errors.c

-// RUN: %clang_cc1 %s -ffreestanding -triple=x86_64-unknown-unknown -target-feature +amx-tile -target-feature +amx-int8 -target-feature +amx-bf16 -emit-llvm -fsyntax-only -verify
+// RUN: %clang_cc1 %s -ffreestanding -triple=x86_64-unknown-unknown -target-feature +amx-tile   \
+// RUN: -target-feature +amx-int8 -target-feature +amx-bf16 -target-feature +amx-fp16 -emit-llvm -fsyntax-only -verify
 
 #include <immintrin.h>
 
 void test_amx(void *data) {
   _tile_zero(16); // expected-error {{argument value 16 is outside the valid range [0, 7]}}
   _tile_loadd(19, data, 16); // expected-error {{argument value 19 is outside the valid range [0, 7]}}
   _tile_stream_loadd(23, data, 1); // expected-error {{argument value 23 is outside the valid range [0, 7]}}
   _tile_stored(88, data, 1); // expected-error {{argument value 88 is outside the valid range [0, 7]}}
   _tile_dpbssd(16, 2, 3); // expected-error {{argument value 16 is outside the valid range [0, 7]}}
   _tile_dpbssd(0, 16, 3); // expected-error {{argument value 16 is outside the valid range [0, 7]}}
   _tile_dpbuud(0, 2, 16); // expected-error {{argument value 16 is outside the valid range [0, 7]}}
   _tile_dpbsud(1, 1, 3); // expected-error {{tile arguments must refer to different tiles}}
   _tile_dpbsud(7, 1, 7); // expected-error {{tile arguments must refer to different tiles}}
   _tile_dpbsud(4, 3, 3); // expected-error {{tile arguments must refer to different tiles}}
   _tile_dpbf16ps(4, 3, 3); // expected-error {{tile arguments must refer to different tiles}}
+  _tile_dpfp16ps(1, 1, 3); // expected-error {{tile arguments must refer to different tiles}}
+  _tile_dpfp16ps(1, 2, 1); // expected-error {{tile arguments must refer to different tiles}}
+  _tile_dpfp16ps(1, 2, 2); // expected-error {{tile arguments must refer to different tiles}}
 }

clang/test/CodeGen/amx_fp16.c

+// RUN: %clang_cc1 %s -ffreestanding -triple=x86_64-unknown-unknown \
+// RUN: -target-feature +amx-tile -target-feature +amx-int8 -target-feature +amx-bf16 -target-feature +amx-fp16 -emit-llvm -o - -Wall -Werror -pedantic \
+// RUN: -Wno-gnu-statement-expression| FileCheck %s
+
+#include <immintrin.h>
+#include <stddef.h>
+void test_tile_dpfp16ps(void) {
+  // CHECK-LABEL: @test_tile_dpfp16ps
+  // CHECK: call void @llvm.x86.tdpfp16ps(i8 1, i8 2, i8 3)
+  _tile_dpfp16ps(1, 2, 3);
+}

clang/test/Driver/x86-target-features.c

 // AMX-BF16: "-target-feature" "+amx-bf16"
 // NO-AMX-BF16: "-target-feature" "-amx-bf16"
 
 // RUN: %clang --target=i386 -march=i386 -mamx-int8 %s -### 2>&1 | FileCheck --check-prefix=AMX-INT8 %s
 // RUN: %clang --target=i386 -march=i386 -mno-amx-int8 %s -### 2>&1 | FileCheck --check-prefix=NO-AMX-INT8 %s
 // AMX-INT8: "-target-feature" "+amx-int8"
 // NO-AMX-INT8: "-target-feature" "-amx-int8"
 
+// RUN: %clang --target=x86_64 -mamx-fp16 %s \
+// RUN: -### -o %t.o 2>&1 | FileCheck -check-prefix=AMX-FP16 %s
+// RUN: %clang --target=x86_64 -mno-amx-fp16 \
+// RUN: %s -### -o %t.o 2>&1 | FileCheck -check-prefix=NO-AMX-FP16 %s
+// AMX-FP16: "-target-feature" "+amx-fp16"
+// NO-AMX-FP16: "-target-feature" "-amx-fp16"
+
 // RUN: %clang --target=i386 -march=i386 -mhreset %s -### 2>&1 | FileCheck -check-prefix=HRESET %s
 // RUN: %clang --target=i386 -march=i386 -mno-hreset %s -### 2>&1 | FileCheck -check-prefix=NO-HRESET %s
 // HRESET: "-target-feature" "+hreset"
 // NO-HRESET: "-target-feature" "-hreset"
 
 // RUN: %clang --target=i386 -march=i386 -muintr %s -### 2>&1 | FileCheck -check-prefix=UINTR %s
 // RUN: %clang --target=i386 -march=i386 -mno-uintr %s -### 2>&1 | FileCheck -check-prefix=NO-UINTR %s
 // UINTR: "-target-feature" "+uintr"

clang/test/Preprocessor/x86_target_features.c

 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -muintr -x c -E -dM -o - %s | FileCheck -check-prefix=UINTR %s
 
 // UINTR: #define __UINTR__ 1
 
 // RUN: %clang -target i386-unknown-linux-gnu -march=i386 -mno-uintr -x c -E -dM -o - %s | FileCheck -check-prefix=NOUINTR %s
 
 // NOUINTR-NOT: #define __UINTR__ 1
 
+// RUN: %clang -target x86_64-unknown-linux-gnu -march=atom -mamx-fp16 -x c \
+// RUN: -E -dM -o - %s | FileCheck  -check-prefix=AMX-FP16 %s
+
+// AMX-FP16: #define __AMXFP16__ 1
+// AMX-FP16: #define __AMXTILE__ 1
+
+// RUN: %clang -target x86_64-unknown-linux-gnu -march=atom -mno-amx-fp16 \
+// RUN: -x c -E -dM -o - %s | FileCheck  -check-prefix=NO-AMX-FP16 %s
+// RUN: %clang -target x86_64-unknown-linux-gnu -march=atom -mamx-fp16 \
+// RUN: -mno-amx-tile -x c -E -dM -o - %s | FileCheck  -check-prefix=NO-AMX-FP16 %s
+
+// NO-AMX-FP16-NOT: #define __AMXFP16__ 1
+// NO-AMX-FP16-NOT: #define __AMXTILE__ 1
+
 // RUN: %clang -target i386-unknown-unknown -march=atom -mavxvnni -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=AVXVNNI %s
 
 // AVXVNNI: #define __AVX2__ 1
 // AVXVNNI: #define __AVXVNNI__ 1
 
 // RUN: %clang -target i386-unknown-unknown -march=atom -mno-avxvnni -x c -E -dM -o - %s | FileCheck -match-full-lines --check-prefix=NOAVXVNNI %s
 
 // NOAVXVNNI-NOT: #define __AVXVNNI__ 1

llvm/docs/ReleaseNotes.rst

* For MinGW, generate embedded ``-exclude-symbols:`` directives for symbols
   This roughly makes hidden visibility work like it does for other object
   file formats.
 
 Changes to the X86 Backend
 --------------------------
 
 * Add support for the ``RDMSRLIST and WRMSRLIST`` instructions.
 * Add support for the ``WRMSRNS`` instruction.
+* Support ISA of ``AMX-FP16`` which contains ``tdpfp16ps`` instruction.
 
 Changes to the OCaml bindings
 -----------------------------
 
 
 Changes to the C API
 --------------------
 

llvm/include/llvm/IR/IntrinsicsX86.td

def int_x86_tdpbf16ps_internal :
                          llvm_x86amx_ty], []>;
   def int_x86_cast_vector_to_tile:
               Intrinsic<[llvm_x86amx_ty], [llvm_anyvector_ty], [IntrNoMem]>;
   def int_x86_cast_tile_to_vector:
               Intrinsic<[llvm_anyvector_ty], [llvm_x86amx_ty], [IntrNoMem]>;
 }
 
 //===----------------------------------------------------------------------===//
+let TargetPrefix = "x86" in {
+// AMX_FP16 - Intel FP16 AMX extensions
+  def int_x86_tdpfp16ps : ClangBuiltin<"__builtin_ia32_tdpfp16ps">,
+                          Intrinsic<[], [llvm_i8_ty, llvm_i8_ty, llvm_i8_ty],
+                                    [ImmArg<ArgIndex<0>>,
+                                    ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>]>;
+}
+//===----------------------------------------------------------------------===//
 // UINTR - User Level Interrupt
 
 let TargetPrefix = "x86" in {
   def int_x86_clui : ClangBuiltin<"__builtin_ia32_clui">,
               Intrinsic<[], [], []>;
   def int_x86_stui : ClangBuiltin<"__builtin_ia32_stui">,
               Intrinsic<[], [], []>;
   def int_x86_testui : ClangBuiltin<"__builtin_ia32_testui">,

llvm/include/llvm/Support/X86TargetParser.def

 X86_FEATURE       (WBNOINVD,        "wbnoinvd")
 X86_FEATURE       (X87,             "x87")
 X86_FEATURE       (XSAVE,           "xsave")
 X86_FEATURE       (XSAVEC,          "xsavec")
 X86_FEATURE       (XSAVEOPT,        "xsaveopt")
 X86_FEATURE       (XSAVES,          "xsaves")
 X86_FEATURE       (HRESET,          "hreset")
 X86_FEATURE       (AVX512FP16,      "avx512fp16")
+X86_FEATURE       (AMX_FP16,        "amx-fp16")
 X86_FEATURE       (AVXVNNI,         "avxvnni")
 // These features aren't really CPU features, but the frontend can set them.
 X86_FEATURE       (RETPOLINE_EXTERNAL_THUNK,    "retpoline-external-thunk")
 X86_FEATURE       (RETPOLINE_INDIRECT_BRANCHES, "retpoline-indirect-branches")
 X86_FEATURE       (RETPOLINE_INDIRECT_CALLS,    "retpoline-indirect-calls")
 X86_FEATURE       (LVI_CFI,                     "lvi-cfi")
 X86_FEATURE       (LVI_LOAD_HARDENING,          "lvi-load-hardening")
 #undef X86_FEATURE_COMPAT

llvm/lib/Support/Host.cpp

#endif
   Features["amx-bf16"]   = HasLeaf7 && ((EDX >> 22) & 1) && HasAMXSave;
   Features["avx512fp16"] = HasLeaf7 && ((EDX >> 23) & 1) && HasAVX512Save;
   Features["amx-tile"]   = HasLeaf7 && ((EDX >> 24) & 1) && HasAMXSave;
   Features["amx-int8"]   = HasLeaf7 && ((EDX >> 25) & 1) && HasAMXSave;
   bool HasLeaf7Subleaf1 =
       MaxLevel >= 7 && !getX86CpuIDAndInfoEx(0x7, 0x1, &EAX, &EBX, &ECX, &EDX);
   Features["avxvnni"]    = HasLeaf7Subleaf1 && ((EAX >> 4) & 1) && HasAVXSave;
   Features["avx512bf16"] = HasLeaf7Subleaf1 && ((EAX >> 5) & 1) && HasAVX512Save;
+  Features["amxfp16"] =    HasLeaf7Subleaf1 && ((EAX >> 21) & 1) && HasAMXSave;
   Features["hreset"]     = HasLeaf7Subleaf1 && ((EAX >> 22) & 1);
   Features["prefetchi"]  = HasLeaf7Subleaf1 && ((EDX >> 14) & 1);
 
   bool HasLeafD = MaxLevel >= 0xd &&
                   !getX86CpuIDAndInfoEx(0xd, 0x1, &EAX, &EBX, &ECX, &EDX);
 
   // Only enable XSAVE if OS has enabled support for saving YMM state.
   Features["xsaveopt"] = HasLeafD && ((EAX >> 0) & 1) && HasAVXSave;

llvm/lib/Support/X86TargetParser.cpp

 // SSE4_A->FMA4->XOP chain.
 constexpr FeatureBitset ImpliedFeaturesSSE4_A = FeatureSSE3;
 constexpr FeatureBitset ImpliedFeaturesFMA4 = FeatureAVX | FeatureSSE4_A;
 constexpr FeatureBitset ImpliedFeaturesXOP = FeatureFMA4;
 
 // AMX Features
 constexpr FeatureBitset ImpliedFeaturesAMX_TILE = {};
 constexpr FeatureBitset ImpliedFeaturesAMX_BF16 = FeatureAMX_TILE;
+constexpr FeatureBitset ImpliedFeaturesAMX_FP16 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesAMX_INT8 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesHRESET = {};
 
 constexpr FeatureBitset ImpliedFeaturesPREFETCHI = {};
 constexpr FeatureBitset ImpliedFeaturesAVX512FP16 =
     FeatureAVX512BW | FeatureAVX512DQ | FeatureAVX512VL;
 // Key Locker Features
 constexpr FeatureBitset ImpliedFeaturesKL = FeatureSSE2;
 constexpr FeatureBitset ImpliedFeaturesWIDEKL = FeatureKL;
 
 // AVXVNNI Features
 constexpr FeatureBitset ImpliedFeaturesAVXVNNI = FeatureAVX2;
 

llvm/lib/Target/X86/X86.td

 def FeatureAMXTILE     : SubtargetFeature<"amx-tile", "HasAMXTILE", "true",
                                       "Support AMX-TILE instructions">;
 def FeatureAMXINT8     : SubtargetFeature<"amx-int8", "HasAMXINT8", "true",
                                       "Support AMX-INT8 instructions",
                                       [FeatureAMXTILE]>;
 def FeatureAMXBF16     : SubtargetFeature<"amx-bf16", "HasAMXBF16", "true",
                                       "Support AMX-BF16 instructions",
                                       [FeatureAMXTILE]>;
+def FeatureAMXFP16     : SubtargetFeature<"amx-fp16", "HasAMXFP16", "true",
+                                      "Support AMX amx-fp16 instructions",
+                                      [FeatureAMXTILE]>;
+
 def FeatureINVPCID : SubtargetFeature<"invpcid", "HasINVPCID", "true",
                                       "Invalidate Process-Context Identifier">;
 def FeatureSGX     : SubtargetFeature<"sgx", "HasSGX", "true",
                                       "Enable Software Guard Extensions">;
 def FeatureCLFLUSHOPT : SubtargetFeature<"clflushopt", "HasCLFLUSHOPT", "true",
                                       "Flush A Cache Line Optimized">;
 def FeatureCLWB    : SubtargetFeature<"clwb", "HasCLWB", "true",
                                       "Cache Line Write Back">;

llvm/lib/Target/X86/X86ISelLowering.cpp

addRegOffset(
         X86::ESP, false, ArgOffset);
     MI.eraseFromParent();
     return BB;
   }
   case X86::PTDPBSSD:
   case X86::PTDPBSUD:
   case X86::PTDPBUSD:
   case X86::PTDPBUUD:
-  case X86::PTDPBF16PS: {
+  case X86::PTDPBF16PS:
+  case X86::PTDPFP16PS: {
     unsigned Opc;
     switch (MI.getOpcode()) {
     default: llvm_unreachable("illegal opcode!");
     case X86::PTDPBSSD: Opc = X86::TDPBSSD; break;
     case X86::PTDPBSUD: Opc = X86::TDPBSUD; break;
     case X86::PTDPBUSD: Opc = X86::TDPBUSD; break;
     case X86::PTDPBUUD: Opc = X86::TDPBUUD; break;
     case X86::PTDPBF16PS: Opc = X86::TDPBF16PS; break;
+    case X86::PTDPFP16PS: Opc = X86::TDPFP16PS; break;
     }
 
     MachineInstrBuilder MIB = BuildMI(*BB, MI, DL, TII->get(Opc));
     MIB.addReg(TMMImmToTMMReg(MI.getOperand(0).getImm()), RegState::Define);
     MIB.addReg(TMMImmToTMMReg(MI.getOperand(0).getImm()), RegState::Undef);
     MIB.addReg(TMMImmToTMMReg(MI.getOperand(1).getImm()), RegState::Undef);
     MIB.addReg(TMMImmToTMMReg(MI.getOperand(2).getImm()), RegState::Undef);
 
case X86::PTILESTORED: {
 
     if (Opc == X86::TILESTORED)
       MIB.addReg(TMMImmToTMMReg(MI.getOperand(CurOp++).getImm()),
                  RegState::Undef);
 
     MI.eraseFromParent(); // The pseudo is gone now.
     return BB;
   }
   }
 }
 
 //===----------------------------------------------------------------------===//
 //                           X86 Optimization Hooks
 //===----------------------------------------------------------------------===//
 
 bool
 X86TargetLowering::targetShrinkDemandedConstant(SDValue Op,
                                                 const APInt &DemandedBits,
                                                 const APInt &DemandedElts,
                                                 TargetLoweringOpt &TLO) const {
   EVT VT = Op.getValueType();
   unsigned Opcode = Op.getOpcode();

llvm/lib/Target/X86/X86InstrAMX.td

let usesCustomInserter = 1 in {
       // To be translated to the actual instructions in X86ISelLowering.cpp
       def PTDPBF16PS : PseudoI<(outs), (ins u8imm:$src1,
                                u8imm:$src2, u8imm:$src3),
                                [(int_x86_tdpbf16ps timm:$src1,
                                  timm:$src2, timm:$src3)]>;
     }
   }
 } // HasAMXTILE, HasAMXBF16
+
+//AMX-FP16
+let Predicates = [HasAMXFP16, In64BitMode] in {
+  let SchedRW = [WriteSystem] in {
+    let Constraints = "$src1 = $dst" in {
+      def TDPFP16PS : I<0x5c, MRMSrcReg4VOp3, (outs TILE:$dst),
+                        (ins TILE:$src1, TILE:$src2, TILE:$src3),
+                        "tdpfp16ps\t{$src3, $src2, $src1|$src1, $src2, $src3}",
+                        []>, VEX_4V, T8XD;
+    }
+    let  usesCustomInserter = 1 in {
+      def PTDPFP16PS : PseudoI<(outs), (ins u8imm:$src1,
+                               u8imm:$src2, u8imm:$src3),
+                               [(int_x86_tdpfp16ps timm:$src1,
+                                 timm:$src2, timm:$src3)]>;
+    }
+  }
+} // HasAMXTILE, HasAMXFP16

llvm/lib/Target/X86/X86InstrInfo.td

 def HasRDPID     : Predicate<"Subtarget->hasRDPID()">;
 def HasRDPRU     : Predicate<"Subtarget->hasRDPRU()">;
 def HasWAITPKG   : Predicate<"Subtarget->hasWAITPKG()">;
 def HasINVPCID   : Predicate<"Subtarget->hasINVPCID()">;
 def HasCX8       : Predicate<"Subtarget->hasCX8()">;
 def HasCX16      : Predicate<"Subtarget->hasCX16()">;
 def HasPCONFIG   : Predicate<"Subtarget->hasPCONFIG()">;
 def HasENQCMD    : Predicate<"Subtarget->hasENQCMD()">;
+def HasAMXFP16   : Predicate<"Subtarget->hasAMXFP16()">;
 def HasKL        : Predicate<"Subtarget->hasKL()">;
 def HasWIDEKL    : Predicate<"Subtarget->hasWIDEKL()">;
 def HasHRESET    : Predicate<"Subtarget->hasHRESET()">;
 def HasSERIALIZE : Predicate<"Subtarget->hasSERIALIZE()">;
 def HasTSXLDTRK  : Predicate<"Subtarget->hasTSXLDTRK()">;
 def HasAMXTILE   : Predicate<"Subtarget->hasAMXTILE()">;
 def HasAMXBF16   : Predicate<"Subtarget->hasAMXBF16()">;
 def HasAMXINT8   : Predicate<"Subtarget->hasAMXINT8()">;

llvm/test/CodeGen/X86/amx_fp16_intrinsics.ll

+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+amx-tile,+amx-fp16 | FileCheck %s
+
+; CHECK-LABEL: test_amx:
+; CHECK:       # %bb.0:
+; CHECK:    tdpfp16ps       %tmm1, %tmm2, %tmm3
+
+define void @test_amx() {
+call void @llvm.x86.tdpfp16ps(i8 3, i8 2, i8 1)
+
+ret void
+}
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+declare void @llvm.x86.tdpfp16ps(i8 %tile3, i8 %tile2, i8 %tile1)

llvm/test/MC/Disassembler/X86/AMX/x86-64-amx-tile-fp16.txt

+# RUN: llvm-mc --disassemble %s -triple=x86_64 | FileCheck -check-prefix=ATT %s
+# RUN: llvm-mc --disassemble %s -triple=x86_64 -x86-asm-syntax=intel --output-asm-variant=1 | FileCheck -check-prefix=INTEL %s
+
+# ATT:   tdpfp16ps %tmm5, %tmm4, %tmm3
+# INTEL: tdpfp16ps tmm3, tmm4, tmm5
+0xc4,0xe2,0x53,0x5c,0xdc

llvm/test/MC/X86/AMX/x86-64-amx-fp16-att.s

+// RUN: llvm-mc -triple x86_64-unknown-unknown --show-encoding %s | FileCheck %s
+
+// CHECK:      tdpfp16ps       %tmm5, %tmm4, %tmm3
+// CHECK: encoding: [0xc4,0xe2,0x53,0x5c,0xdc]
+               tdpfp16ps       %tmm5, %tmm4, %tmm3

llvm/test/MC/X86/AMX/x86-64-amx-fp16-intel.s

+// RUN: llvm-mc -triple x86_64-unknown-unknown -x86-asm-syntax=intel -output-asm-variant=1 --show-encoding %s | FileCheck %s
+
+// CHECK:      tdpfp16ps       tmm3, tmm4, tmm5
+// CHECK: encoding: [0xc4,0xe2,0x53,0x5c,0xdc]
+               tdpfp16ps       tmm3, tmm4, tmm5