[X86][1/2] Support PREFETCHI instructions

[pengfei]

Summary

For more details about these instructions, please refer to the latest ISE document: https://www.intel.com/content/www/us/en/develop/download/intel-architecture-instruction-set-extensions-programming-reference.html

[pengfei]

Fix lit fails.

[pengfei]

Fix lit fails.

[LuoYuanke]

clang/include/clang/Driver/Options.td

4651

I notice in line 4655 the option name is "prfch", do we need to follow the naming convention?

llvm/lib/Target/X86/X86InstrInfo.td

3007

Could you add comments to explain the what the constant (1, 3, 0) means? I guess it is the same to the arguments that llvm.prefetch defines.

[pengfei]

1. Address review comments;
2. Optimize prefetchit0/1 to prefetcht0/1 for non-rip address;
3. Add semacheck for prefetch write to instruction cache;
4. Fix a bug that set rw = 1 for prefetchit0/1;

I think the affected ARM and SystemZ tests are not valid before. Could @t.p.northover and @uweigand help to have a look?

clang/include/clang/Driver/Options.td

4651

I don't know the reason why we used prfchw before, but we are now using the option the same as feature name so that user can easily guess the right option.

llvm/lib/Target/X86/X86InstrInfo.td

3007

Added short introduction. Full description is in https://llvm.org/docs/LangRef.html#llvm-prefetch-intrinsic

[uweigand]

In D136040#3862225, @pengfei wrote:

3. Add semacheck for prefetch write to instruction cache;

I think the affected ARM and SystemZ tests are not valid before. Could @t.p.northover and @uweigand help to have a look?

This seems to be a semantic change. The Language Reference does not spell out that a write prefetch on the instruction cache is prohibited. In fact, I read it to explicitly state that every flavor of the prefetch intrinsic that doesn't match anything supported on the target architecture should simply be a no-op. (I could imagine that on certain architectures, you might even have such a prefetch, e.g. to handle self-modifying code more efficiently.)

[pengfei]

In D136040#3862265, @uweigand wrote:

In D136040#3862225, @pengfei wrote:

3. Add semacheck for prefetch write to instruction cache;

I think the affected ARM and SystemZ tests are not valid before. Could @t.p.northover and @uweigand help to have a look?

This seems to be a semantic change. The Language Reference does not spell out that a write prefetch on the instruction cache is prohibited. In fact, I read it to explicitly state that every flavor of the prefetch intrinsic that doesn't match anything supported on the target architecture should simply be a no-op. (I could imagine that on certain architectures, you might even have such a prefetch, e.g. to handle self-modifying code more efficiently.)

Sure, it is possible. But at least for now, there's no real target requires it. Checked with grep -rwn 'llvm.prefetch.*i32 0\s*)' llvm/test/CodeGen/.
The reason I do it is it's ambiguity to backend when lower it to a target that has "write data", "read data" and "read instruction" prefetches. It's clear if we just support the semantic of "read instruction" and lower it to "read data" when available.
I think we can modify Language Reference for it.

[uweigand]

In D136040#3862386, @pengfei wrote:

Sure, it is possible. But at least for now, there's no real target requires it. Checked with grep -rwn 'llvm.prefetch.*i32 0\s*)' llvm/test/CodeGen/.

But that's just within the LLVM sources. My point was that -up to now- this was part of the public LLVM IR spec, so there could by other LLVM users out there creating this IR, which would now suddenly break. Not sure what the rules are for breaking IR changes, but that would presumably need some wider discussion.

[craig.topper]

There are too many things in this patch. Please up MC layer changes from intrinsic/builtin changes. The RIP optimization probably also be separate.

[pengfei]

In D136040#3862551, @uweigand wrote:

In D136040#3862386, @pengfei wrote:

Sure, it is possible. But at least for now, there's no real target requires it. Checked with grep -rwn 'llvm.prefetch.*i32 0\s*)' llvm/test/CodeGen/.

But that's just within the LLVM sources. My point was that -up to now- this was part of the public LLVM IR spec, so there could by other LLVM users out there creating this IR, which would now suddenly break. Not sure what the rules are for breaking IR changes, but that would presumably need some wider discussion.

Thanks @uweigand, you are right. I have split D136145 and put it in Discourse for broad discussion.

[pengfei]

Rebased on D136145 and split RIP optimization.

[craig.topper]

Can the intrinsics changes be split from this patch so they don't depend on D136145. There's no reason to block assembler/disassembler support for that.

clang/lib/Headers/prfchiintrin.h

17

It looks old that this indented differently than the "Loads" on the line above.

[craig.topper]

clang/lib/Headers/prfchiintrin.h

17

That should have said "It looks ODD that this indented differently..."

[pengfei]

Remove the dependence to D136145.

[pengfei]

clang/lib/Headers/prfchiintrin.h

17

We use this indention in most existing headers. I guess it's intended to align with parameters' description. We follow this rule in the ISAs.

[craig.topper]

LGTM

clang/docs/ReleaseNotes.rst

 -----------------------
 - ``sifive-7-rv32`` and ``sifive-7-rv64`` are no longer supported for ``-mcpu``.
   Use ``sifive-e76``, ``sifive-s76``, or ``sifive-u74`` instead.
 
 X86 Support in Clang
 --------------------
 - Support ``-mindirect-branch-cs-prefix`` for call and jmp to indirect thunk.
 - Fix 32-bit ``__fastcall`` and ``__vectorcall`` ABI mismatch with MSVC.
+- Add support for ``PREFETCHI`` instructions.
 
 DWARF Support in Clang
 ----------------------
 
 Arm and AArch64 Support in Clang
 --------------------------------
 
 - The target(..) function attributes for AArch64 now accept:

clang/include/clang/Driver/Options.td

 def mpku : Flag<["-"], "mpku">, Group<m_x86_Features_Group>;
 def mno_pku : Flag<["-"], "mno-pku">, Group<m_x86_Features_Group>;
 def mpclmul : Flag<["-"], "mpclmul">, Group<m_x86_Features_Group>;
 def mno_pclmul : Flag<["-"], "mno-pclmul">, Group<m_x86_Features_Group>;
 def mpconfig : Flag<["-"], "mpconfig">, Group<m_x86_Features_Group>;
 def mno_pconfig : Flag<["-"], "mno-pconfig">, Group<m_x86_Features_Group>;
 def mpopcnt : Flag<["-"], "mpopcnt">, Group<m_x86_Features_Group>;
 def mno_popcnt : Flag<["-"], "mno-popcnt">, Group<m_x86_Features_Group>;
+def mprefetchi : Flag<["-"], "mprefetchi">, Group<m_x86_Features_Group>;
+def mno_prefetchi : Flag<["-"], "mno-prefetchi">, Group<m_x86_Features_Group>;
 def mprefetchwt1 : Flag<["-"], "mprefetchwt1">, Group<m_x86_Features_Group>;
 def mno_prefetchwt1 : Flag<["-"], "mno-prefetchwt1">, Group<m_x86_Features_Group>;
 def mprfchw : Flag<["-"], "mprfchw">, Group<m_x86_Features_Group>;
 def mno_prfchw : Flag<["-"], "mno-prfchw">, Group<m_x86_Features_Group>;
 def mptwrite : Flag<["-"], "mptwrite">, Group<m_x86_Features_Group>;
 def mno_ptwrite : Flag<["-"], "mno-ptwrite">, Group<m_x86_Features_Group>;
 def mrdpid : Flag<["-"], "mrdpid">, Group<m_x86_Features_Group>;
 def mno_rdpid : Flag<["-"], "mno-rdpid">, Group<m_x86_Features_Group>;

clang/lib/Basic/Targets/X86.h

class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo {
   bool HasMWAITX = false;
   bool HasCLZERO = false;
   bool HasCLDEMOTE = false;
   bool HasPCONFIG = false;
   bool HasPKU = false;
   bool HasCLFLUSHOPT = false;
   bool HasCLWB = false;
   bool HasMOVBE = false;
+  bool HasPREFETCHI = false;
   bool HasPREFETCHWT1 = false;
   bool HasRDPID = false;
   bool HasRDPRU = false;
   bool HasRetpolineExternalThunk = false;
   bool HasLAHFSAHF = false;
   bool HasWBNOINVD = false;
   bool HasWAITPKG = false;
   bool HasMOVDIRI = false;

clang/lib/Basic/Targets/X86.cpp

for (const auto &Feature : Features) {
     } else if (Feature == "+pku") {
       HasPKU = true;
     } else if (Feature == "+clflushopt") {
       HasCLFLUSHOPT = true;
     } else if (Feature == "+clwb") {
       HasCLWB = true;
     } else if (Feature == "+wbnoinvd") {
       HasWBNOINVD = true;
+    } else if (Feature == "+prefetchi") {
+      HasPREFETCHI = true;
     } else if (Feature == "+prefetchwt1") {
       HasPREFETCHWT1 = true;
     } else if (Feature == "+clzero") {
       HasCLZERO = true;
     } else if (Feature == "+cldemote") {
       HasCLDEMOTE = true;
     } else if (Feature == "+rdpid") {
       HasRDPID = true;
void X86TargetInfo::getTargetDefines(const LangOptions &Opts,
   if (HasCLWB)
     Builder.defineMacro("__CLWB__");
   if (HasWBNOINVD)
     Builder.defineMacro("__WBNOINVD__");
   if (HasSHSTK)
     Builder.defineMacro("__SHSTK__");
   if (HasSGX)
     Builder.defineMacro("__SGX__");
+  if (HasPREFETCHI)
+    Builder.defineMacro("__PREFETCHI__");
   if (HasPREFETCHWT1)
     Builder.defineMacro("__PREFETCHWT1__");
   if (HasCLZERO)
     Builder.defineMacro("__CLZERO__");
   if (HasKL)
     Builder.defineMacro("__KL__");
   if (HasWIDEKL)
     Builder.defineMacro("__WIDEKL__");
return llvm::StringSwitch<bool>(Name)
       .Case("movbe", true)
       .Case("movdiri", true)
       .Case("movdir64b", true)
       .Case("mwaitx", true)
       .Case("pclmul", true)
       .Case("pconfig", true)
       .Case("pku", true)
       .Case("popcnt", true)
+      .Case("prefetchi", true)
       .Case("prefetchwt1", true)
       .Case("prfchw", true)
       .Case("ptwrite", true)
       .Case("rdpid", true)
       .Case("rdpru", true)
       .Case("rdrnd", true)
       .Case("rdseed", true)
       .Case("rtm", true)
return llvm::StringSwitch<bool>(Feature)
       .Case("movbe", HasMOVBE)
       .Case("movdiri", HasMOVDIRI)
       .Case("movdir64b", HasMOVDIR64B)
       .Case("mwaitx", HasMWAITX)
       .Case("pclmul", HasPCLMUL)
       .Case("pconfig", HasPCONFIG)
       .Case("pku", HasPKU)
       .Case("popcnt", HasPOPCNT)
+      .Case("prefetchi", HasPREFETCHI)
       .Case("prefetchwt1", HasPREFETCHWT1)
       .Case("prfchw", HasPRFCHW)
       .Case("ptwrite", HasPTWRITE)
       .Case("rdpid", HasRDPID)
       .Case("rdpru", HasRDPRU)
       .Case("rdrnd", HasRDRND)
       .Case("rdseed", HasRDSEED)
       .Case("retpoline-external-thunk", HasRetpolineExternalThunk)

clang/lib/CodeGen/CGBuiltin.cpp

case Builtin::BI__builtin_object_size: {
 
     // We pass this builtin onto the optimizer so that it can figure out the
     // object size in more complex cases.
     bool IsDynamic = BuiltinID == Builtin::BI__builtin_dynamic_object_size;
     return RValue::get(emitBuiltinObjectSize(E->getArg(0), Type, ResType,
                                              /*EmittedE=*/nullptr, IsDynamic));
   }
   case Builtin::BI__builtin_prefetch: {
-    Value *Locality, *RW, *Address = EmitScalarExpr(E->getArg(0));
+    Value *Locality, *RW, *Data, *Address = EmitScalarExpr(E->getArg(0));
     // FIXME: Technically these constants should of type 'int', yes?
     RW = (E->getNumArgs() > 1) ? EmitScalarExpr(E->getArg(1)) :
       llvm::ConstantInt::get(Int32Ty, 0);
     Locality = (E->getNumArgs() > 2) ? EmitScalarExpr(E->getArg(2)) :
       llvm::ConstantInt::get(Int32Ty, 3);
-    Value *Data = llvm::ConstantInt::get(Int32Ty, 1);
+    Data = (E->getNumArgs() > 3) ? EmitScalarExpr(E->getArg(3)) :
+      llvm::ConstantInt::get(Int32Ty, 1);
     Function *F = CGM.getIntrinsic(Intrinsic::prefetch, Address->getType());
     return RValue::get(Builder.CreateCall(F, {Address, RW, Locality, Data}));
   }
   case Builtin::BI__builtin_readcyclecounter: {
     Function *F = CGM.getIntrinsic(Intrinsic::readcyclecounter);
     return RValue::get(Builder.CreateCall(F));
   }
   case Builtin::BI__builtin___clear_cache: {

clang/lib/Headers/cpuid.h

 #define bit_AMXTILE       0x01000000
 #define bit_AMXINT8       0x02000000
 
 /* Features in %eax for leaf 7 sub-leaf 1 */
 #define bit_AVXVNNI       0x00000010
 #define bit_AVX512BF16    0x00000020
 #define bit_HRESET        0x00400000
 
+/* Features in %edx for leaf 7 sub-leaf 1 */
+#define bit_PREFETCHI     0x00004000
+
 /* Features in %eax for leaf 13 sub-leaf 1 */
 #define bit_XSAVEOPT    0x00000001
 #define bit_XSAVEC      0x00000002
 #define bit_XSAVES      0x00000008
 
 /* Features in %eax for leaf 0x14 sub-leaf 0 */
 #define bit_PTWRITE     0x00000010
 

clang/lib/Headers/prfchiintrin.h

+/*===---- prfchiintrin.h - PREFETCHI intrinsic -----------------------------===
+ *
+ * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+ * See https://llvm.org/LICENSE.txt for license information.
+ * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+ *
+ *===-----------------------------------------------------------------------===
+ */
+
+#ifndef __PRFCHIINTRIN_H
+#define __PRFCHIINTRIN_H
+
+#ifdef __x86_64__
+
+/// Loads an instruction sequence containing the specified memory address into
+///    all level cache.
+///
+///    Note that the effect of this intrinsic is dependent on the processor
+///    implementation.
+///
+/// \headerfile <x86intrin.h>
+///
+/// This intrinsic corresponds to the \c PREFETCHIT0 instruction.
+///
+/// \param __P
+///    A pointer specifying the memory address to be prefetched.
+static __inline__ void __attribute__((__always_inline__, __nodebug__))
+_m_prefetchit0(volatile const void *__P) {
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wcast-qual"
+  __builtin_prefetch((const void *)__P, 0, 3 /* _MM_HINT_T0 */, 0 /* inst */);
+#pragma clang diagnostic pop
+}
+
+/// Loads an instruction sequence containing the specified memory address into
+///    all but the first-level cache.
+///
+///    Note that the effect of this intrinsic is dependent on the processor
+///    implementation.
+///
+/// \headerfile <x86intrin.h>
+///
+/// This intrinsic corresponds to the \c PREFETCHIT1 instruction.
+///
+/// \param __P
+///    A pointer specifying the memory address to be prefetched.
+static __inline__ void __attribute__((__always_inline__, __nodebug__))
+_m_prefetchit1(volatile const void *__P) {
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wcast-qual"
+  __builtin_prefetch((const void *)__P, 0, 2 /* _MM_HINT_T1 */, 0 /* inst */);
+#pragma clang diagnostic pop
+}
+#endif /* __x86_64__ */
+
+#endif /* __PRFCHWINTRIN_H */

clang/lib/Headers/x86gprintrin.h

 #include <uintrintrin.h>
 #endif
 
 #if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
     defined(__CRC32__)
 #include <crc32intrin.h>
 #endif
 
+#if !(defined(_MSC_VER) || defined(__SCE__)) || __has_feature(modules) ||      \
+    defined(__PRFCHI__)
+#include <prfchiintrin.h>
+#endif
+
 #if defined(__i386__)
 #define __SAVE_GPRBX "mov {%%ebx, %%eax |eax, ebx};"
 #define __RESTORE_GPRBX "mov {%%eax, %%ebx |ebx, eax};"
 #define __TMPGPR "eax"
 #else
 // When in 64-bit target, the 32-bit operands generate a 32-bit result,
 // zero-extended to a 64-bit result in the destination general-purpose,
 // It means "mov x %ebx" will clobber the higher 32 bits of rbx, so we

clang/lib/Sema/SemaChecking.cpp

 }
 
 /// SemaBuiltinPrefetch - Handle __builtin_prefetch.
 // This is declared to take (const void*, ...) and can take two
 // optional constant int args.
 bool Sema::SemaBuiltinPrefetch(CallExpr *TheCall) {
   unsigned NumArgs = TheCall->getNumArgs();
 
-  if (NumArgs > 3)
+  if (NumArgs > 4)
     return Diag(TheCall->getEndLoc(),
                 diag::err_typecheck_call_too_many_args_at_most)
-           << 0 /*function call*/ << 3 << NumArgs << TheCall->getSourceRange();
+           << 0 /*function call*/ << 4 << NumArgs << TheCall->getSourceRange();
 
   // Argument 0 is checked for us and the remaining arguments must be
   // constant integers.
   for (unsigned i = 1; i != NumArgs; ++i)
-    if (SemaBuiltinConstantArgRange(TheCall, i, 0, i == 1 ? 1 : 3))
+    if (SemaBuiltinConstantArgRange(TheCall, i, 0, i == 2 ? 3 : 1))
       return true;
 
   return false;
 }
 
 /// SemaBuiltinArithmeticFence - Handle __arithmetic_fence.
 bool Sema::SemaBuiltinArithmeticFence(CallExpr *TheCall) {
   if (!Context.getTargetInfo().checkArithmeticFenceSupported())

clang/test/CodeGen/X86/prefetchi-builtins.c

+// RUN: %clang_cc1 -ffreestanding -triple x86_64-unknown-unknown -target-feature +prefetchi -emit-llvm -o - %s | FileCheck %s
+
+
+#include <x86intrin.h>
+
+void test_m_prefetch_it0(void *p) {
+  return _m_prefetchit0(p);
+  // CHECK-LABEL: define{{.*}} void @test_m_prefetch_it0
+  // CHECK: call void @llvm.prefetch.p0(ptr {{.*}}, i32 0, i32 3, i32 0)
+}
+
+void test_m_prefetch_it1(void *p) {
+  return _m_prefetchit1(p);
+  // CHECK-LABEL: define{{.*}} void @test_m_prefetch_it1
+  // CHECK: call void @llvm.prefetch.p0(ptr {{.*}}, i32 0, i32 2, i32 0)
+}

clang/test/Driver/x86-target-features.c

 // SGX: "-target-feature" "+sgx"
 // NO-SGX: "-target-feature" "-sgx"
 
 // RUN: %clang --target=i386 -march=i386 -mprefetchwt1 %s -### 2>&1 | FileCheck -check-prefix=PREFETCHWT1 %s
 // RUN: %clang --target=i386 -march=i386 -mno-prefetchwt1 %s -### 2>&1 | FileCheck -check-prefix=NO-PREFETCHWT1 %s
 // PREFETCHWT1: "-target-feature" "+prefetchwt1"
 // NO-PREFETCHWT1: "-target-feature" "-prefetchwt1"
 
+// RUN: %clang --target=i386 -march=i386 -mprefetchi %s -### -o %t.o 2>&1 | FileCheck -check-prefix=PREFETCHI %s
+// RUN: %clang --target=i386 -march=i386 -mno-prefetchi %s -### -o %t.o 2>&1 | FileCheck -check-prefix=NO-PREFETCHI %s
+// PREFETCHI: "-target-feature" "+prefetchi"
+// NO-PREFETCHI: "-target-feature" "-prefetchi"
+
 // RUN: %clang --target=i386 -march=i386 -mclzero %s -### 2>&1 | FileCheck -check-prefix=CLZERO %s
 // RUN: %clang --target=i386 -march=i386 -mno-clzero %s -### 2>&1 | FileCheck -check-prefix=NO-CLZERO %s
 // CLZERO: "-target-feature" "+clzero"
 // NO-CLZERO: "-target-feature" "-clzero"
 
 // RUN: %clang --target=i386 -march=i386 -mvaes %s -### 2>&1 | FileCheck -check-prefix=VAES %s
 // RUN: %clang --target=i386 -march=i386 -mno-vaes %s -### 2>&1 | FileCheck -check-prefix=NO-VAES %s
 // VAES: "-target-feature" "+vaes"

clang/test/Sema/builtin-prefetch.c

 // RUN: %clang_cc1 -fsyntax-only -verify %s
 
 void foo(void) {
   int a;
   __builtin_prefetch(&a);
   __builtin_prefetch(&a, 1);
-  __builtin_prefetch(&a, 1, 2);
-  __builtin_prefetch(&a, 1, 9, 3); // expected-error{{too many arguments to function}}
+  __builtin_prefetch(&a, 1, 2, 0);
+  __builtin_prefetch(&a, 1, 9, 8, 3); // expected-error{{too many arguments to function}}
   __builtin_prefetch(&a, "hello", 2); // expected-error{{argument to '__builtin_prefetch' must be a constant integer}}
   __builtin_prefetch(&a, a, 2); // expected-error{{argument to '__builtin_prefetch' must be a constant integer}}
   __builtin_prefetch(&a, 2); // expected-error{{argument value 2 is outside the valid range [0, 1]}}
   __builtin_prefetch(&a, 0, 4); // expected-error{{argument value 4 is outside the valid range [0, 3]}}
   __builtin_prefetch(&a, -1, 4); // expected-error{{argument value -1 is outside the valid range [0, 1]}}
+  __builtin_prefetch(&a, 1, 2, 3); // expected-error{{argument value 3 is outside the valid range [0, 1]}}
 }

llvm/include/llvm/Support/X86TargetParser.def

 X86_FEATURE       (LWP,             "lwp")
 X86_FEATURE       (LZCNT,           "lzcnt")
 X86_FEATURE       (MOVBE,           "movbe")
 X86_FEATURE       (MOVDIR64B,       "movdir64b")
 X86_FEATURE       (MOVDIRI,         "movdiri")
 X86_FEATURE       (MWAITX,          "mwaitx")
 X86_FEATURE       (PCONFIG,         "pconfig")
 X86_FEATURE       (PKU,             "pku")
+X86_FEATURE       (PREFETCHI,       "prefetchi")
 X86_FEATURE       (PREFETCHWT1,     "prefetchwt1")
 X86_FEATURE       (PRFCHW,          "prfchw")
 X86_FEATURE       (PTWRITE,         "ptwrite")
 X86_FEATURE       (RDPID,           "rdpid")
 X86_FEATURE       (RDPRU,           "rdpru")
 X86_FEATURE       (RDRND,           "rdrnd")
 X86_FEATURE       (RDSEED,          "rdseed")
 X86_FEATURE       (RTM,             "rtm")

llvm/lib/Support/Host.cpp

#endif
   Features["avx512fp16"] = HasLeaf7 && ((EDX >> 23) & 1) && HasAVX512Save;
   Features["amx-tile"]   = HasLeaf7 && ((EDX >> 24) & 1) && HasAMXSave;
   Features["amx-int8"]   = HasLeaf7 && ((EDX >> 25) & 1) && HasAMXSave;
   bool HasLeaf7Subleaf1 =
       MaxLevel >= 7 && !getX86CpuIDAndInfoEx(0x7, 0x1, &EAX, &EBX, &ECX, &EDX);
   Features["avxvnni"]    = HasLeaf7Subleaf1 && ((EAX >> 4) & 1) && HasAVXSave;
   Features["avx512bf16"] = HasLeaf7Subleaf1 && ((EAX >> 5) & 1) && HasAVX512Save;
   Features["hreset"]     = HasLeaf7Subleaf1 && ((EAX >> 22) & 1);
+  Features["prefetchi"]  = HasLeaf7Subleaf1 && ((EDX >> 14) & 1);
 
   bool HasLeafD = MaxLevel >= 0xd &&
                   !getX86CpuIDAndInfoEx(0xd, 0x1, &EAX, &EBX, &ECX, &EDX);
 
   // Only enable XSAVE if OS has enabled support for saving YMM state.
   Features["xsaveopt"] = HasLeafD && ((EAX >> 0) & 1) && HasAVXSave;
   Features["xsavec"]   = HasLeafD && ((EAX >> 1) & 1) && HasAVXSave;
   Features["xsaves"]   = HasLeafD && ((EAX >> 3) & 1) && HasAVXSave;

llvm/lib/Support/X86TargetParser.cpp

 constexpr FeatureBitset ImpliedFeaturesXOP = FeatureFMA4;
 
 // AMX Features
 constexpr FeatureBitset ImpliedFeaturesAMX_TILE = {};
 constexpr FeatureBitset ImpliedFeaturesAMX_BF16 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesAMX_INT8 = FeatureAMX_TILE;
 constexpr FeatureBitset ImpliedFeaturesHRESET = {};
 
+static constexpr FeatureBitset ImpliedFeaturesPREFETCHI = {};
 static constexpr FeatureBitset ImpliedFeaturesAVX512FP16 =
     FeatureAVX512BW | FeatureAVX512DQ | FeatureAVX512VL;
 // Key Locker Features
 constexpr FeatureBitset ImpliedFeaturesKL = FeatureSSE2;
 constexpr FeatureBitset ImpliedFeaturesWIDEKL = FeatureKL;
 
 // AVXVNNI Features
 constexpr FeatureBitset ImpliedFeaturesAVXVNNI = FeatureAVX2;

llvm/lib/Target/X86/X86.td

def FeatureCDI      : SubtargetFeature<"avx512cd", "HasCDI", "true",
                       "Enable AVX-512 Conflict Detection Instructions",
                                       [FeatureAVX512]>;
 def FeatureVPOPCNTDQ : SubtargetFeature<"avx512vpopcntdq", "HasVPOPCNTDQ",
                        "true", "Enable AVX-512 Population Count Instructions",
                                       [FeatureAVX512]>;
 def FeaturePFI      : SubtargetFeature<"avx512pf", "HasPFI", "true",
                       "Enable AVX-512 PreFetch Instructions",
                                       [FeatureAVX512]>;
+def FeaturePREFETCHI  : SubtargetFeature<"prefetchi", "HasPREFETCHI",
+                                   "true",
+                                   "Prefetch instruction with T0 or T1 Hint">;
 def FeaturePREFETCHWT1  : SubtargetFeature<"prefetchwt1", "HasPREFETCHWT1",
                                    "true",
                                    "Prefetch with Intent to Write and T1 Hint">;
 def FeatureDQI     : SubtargetFeature<"avx512dq", "HasDQI", "true",
                       "Enable AVX-512 Doubleword and Quadword Instructions",
                                       [FeatureAVX512]>;
 def FeatureBWI     : SubtargetFeature<"avx512bw", "HasBWI", "true",
                       "Enable AVX-512 Byte and Word Instructions",

llvm/lib/Target/X86/X86DiscriminateMemOps.cpp

public:
   static char ID;
 
   /// Default construct and initialize the pass.
   X86DiscriminateMemOps();
 };
 
 bool IsPrefetchOpcode(unsigned Opcode) {
   return Opcode == X86::PREFETCHNTA || Opcode == X86::PREFETCHT0 ||
-         Opcode == X86::PREFETCHT1 || Opcode == X86::PREFETCHT2;
+         Opcode == X86::PREFETCHT1 || Opcode == X86::PREFETCHT2 ||
+         Opcode == X86::PREFETCHIT0 || Opcode == X86::PREFETCHIT1;
 }
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 //            Implementation
 //===----------------------------------------------------------------------===//
 
 char X86DiscriminateMemOps::ID = 0;

llvm/lib/Target/X86/X86Instr3DNow.td

 def PrefetchWT1Level : ImmLeaf<i32, [{
   return Imm < 3;
 }]>;
 
 let SchedRW = [WriteLoad] in {
 let Predicates = [Has3DNow, NoSSEPrefetch] in
 def PREFETCH : I3DNow<0x0D, MRM0m, (outs), (ins i8mem:$addr),
                       "prefetch\t$addr",
-                      [(prefetch addr:$addr, imm, imm, (i32 1))]>, TB;
+                      [(prefetch addr:$addr, imm, imm, (i32 imm))]>, TB;
 
 def PREFETCHW : I<0x0D, MRM1m, (outs), (ins i8mem:$addr), "prefetchw\t$addr",
                   [(prefetch addr:$addr, (i32 1), (i32 PrefetchWLevel), (i32 1))]>,
                   TB, Requires<[HasPrefetchW]>;
 
 def PREFETCHWT1 : I<0x0D, MRM2m, (outs), (ins i8mem:$addr), "prefetchwt1\t$addr",
                     [(prefetch addr:$addr, (i32 1), (i32 PrefetchWT1Level), (i32 1))]>,
                     TB, Requires<[HasPREFETCHWT1]>;
 }
 
 // "3DNowA" instructions
 defm PF2IW    : I3DNow_conv_rm_int<0x1C, "pf2iw", WriteCvtPS2I, "a">;
 defm PI2FW    : I3DNow_conv_rm_int<0x0C, "pi2fw", WriteCvtI2PS, "a">;
 defm PFNACC   : I3DNow_binop_rm_int<0x8A, "pfnacc", WriteFAdd, 0, "a">;
 defm PFPNACC  : I3DNow_binop_rm_int<0x8E, "pfpnacc", WriteFAdd, 0, "a">;
 defm PSWAPD   : I3DNow_conv_rm_int<0xBB, "pswapd", SchedWriteShuffle.MMX, "a">;

llvm/lib/Target/X86/X86InstrInfo.td

 def HasIFMA      : Predicate<"Subtarget->hasIFMA()">;
 def HasRTM       : Predicate<"Subtarget->hasRTM()">;
 def HasADX       : Predicate<"Subtarget->hasADX()">;
 def HasSHA       : Predicate<"Subtarget->hasSHA()">;
 def HasSGX       : Predicate<"Subtarget->hasSGX()">;
 def HasRDSEED    : Predicate<"Subtarget->hasRDSEED()">;
 def HasSSEPrefetch : Predicate<"Subtarget->hasSSEPrefetch()">;
 def NoSSEPrefetch : Predicate<"!Subtarget->hasSSEPrefetch()">;
+def HasPREFETCHI : Predicate<"Subtarget->hasPREFETCHI()">;
 def HasPrefetchW : Predicate<"Subtarget->hasPrefetchW()">;
 def HasPREFETCHWT1 : Predicate<"Subtarget->hasPREFETCHWT1()">;
 def HasLAHFSAHF  : Predicate<"Subtarget->hasLAHFSAHF()">;
 def HasMWAITX    : Predicate<"Subtarget->hasMWAITX()">;
 def HasCLZERO    : Predicate<"Subtarget->hasCLZERO()">;
 def HasCLDEMOTE  : Predicate<"Subtarget->hasCLDEMOTE()">;
 def HasMOVDIRI   : Predicate<"Subtarget->hasMOVDIRI()">;
 def HasMOVDIR64B : Predicate<"Subtarget->hasMOVDIR64B()">;
def SENDUIPI : I<0xC7, MRM6r, (outs), (ins GR64:$arg), "senduipi\t$arg",
                    [(int_x86_senduipi GR64:$arg)]>, XS;
 
   let Defs = [EFLAGS] in
     def TESTUI : I<0x01, MRM_ED, (outs), (ins), "testui",
                    [(set EFLAGS, (X86testui))]>, XS;
 }
 
 //===----------------------------------------------------------------------===//
+// PREFETCHIT0 and PREFETCHIT1 Instructions
+//
+let Predicates = [HasPREFETCHI, In64BitMode], SchedRW = [WriteLoad] in {
+  def PREFETCHIT0 : I<0x18, MRM7m, (outs), (ins i8mem:$src),
+    "prefetchit0\t$src", [(prefetch addr:$src, (i32 1), (i32 3), (i32 0))]>, TB;
+  def PREFETCHIT1 : I<0x18, MRM6m, (outs), (ins i8mem:$src),
+    "prefetchit1\t$src", [(prefetch addr:$src, (i32 1), (i32 2), (i32 0))]>, TB;
+}
+
+//===----------------------------------------------------------------------===//
 // Pattern fragments to auto generate TBM instructions.
 //===----------------------------------------------------------------------===//
 
 let Predicates = [HasTBM] in {
   // FIXME: patterns for the load versions are not implemented
   def : Pat<(and GR32:$src, (add GR32:$src, 1)),
             (BLCFILL32rr GR32:$src)>;
   def : Pat<(and GR64:$src, (add GR64:$src, 1)),

llvm/lib/Target/X86/X86InstrSSE.td

 
 //===----------------------------------------------------------------------===//
 // SSE 1 & 2 - Prefetch and memory fence
 //===----------------------------------------------------------------------===//
 
 // Prefetch intrinsic.
 let Predicates = [HasSSEPrefetch], SchedRW = [WriteLoad] in {
 def PREFETCHT0   : I<0x18, MRM1m, (outs), (ins i8mem:$src),
-    "prefetcht0\t$src", [(prefetch addr:$src, imm, (i32 3), (i32 1))]>, TB;
+    "prefetcht0\t$src", [(prefetch addr:$src, imm, (i32 3), imm)]>, TB;
 def PREFETCHT1   : I<0x18, MRM2m, (outs), (ins i8mem:$src),
-    "prefetcht1\t$src", [(prefetch addr:$src, imm, (i32 2), (i32 1))]>, TB;
+    "prefetcht1\t$src", [(prefetch addr:$src, imm, (i32 2), imm)]>, TB;
 def PREFETCHT2   : I<0x18, MRM3m, (outs), (ins i8mem:$src),
-    "prefetcht2\t$src", [(prefetch addr:$src, imm, (i32 1), (i32 1))]>, TB;
+    "prefetcht2\t$src", [(prefetch addr:$src, imm, (i32 1), imm)]>, TB;
 def PREFETCHNTA  : I<0x18, MRM0m, (outs), (ins i8mem:$src),
-    "prefetchnta\t$src", [(prefetch addr:$src, imm, (i32 0), (i32 1))]>, TB;
+    "prefetchnta\t$src", [(prefetch addr:$src, imm, (i32 0), imm)]>, TB;
 }
 
 // FIXME: How should flush instruction be modeled?
 let SchedRW = [WriteLoad] in {
 // Flush cache
 def CLFLUSH : I<0xAE, MRM7m, (outs), (ins i8mem:$src),
                "clflush\t$src", [(int_x86_sse2_clflush addr:$src)]>,
                PS, Requires<[HasSSE2]>;

llvm/lib/Target/X86/X86Subtarget.h

bool hasPrefetchW() const {
     // it and KNL has another that prefetches to L2 cache. We assume the
     // L1 version exists if the L2 version does.
     return hasThreeDNow() || hasPRFCHW() || hasPREFETCHWT1();
   }
   bool hasSSEPrefetch() const {
     // We implicitly enable these when we have a write prefix supporting cache
     // level OR if we have prfchw, but don't already have a read prefetch from
     // 3dnow.
-    return hasSSE1() || (hasPRFCHW() && !hasThreeDNow()) || hasPREFETCHWT1();
+    return hasSSE1() || (hasPRFCHW() && !hasThreeDNow()) || hasPREFETCHWT1() ||
+           hasPREFETCHI();
   }
   bool canUseLAHFSAHF() const { return hasLAHFSAHF64() || !is64Bit(); }
   // These are generic getters that OR together all of the thunk types
   // supported by the subtarget. Therefore useIndirectThunk*() will return true
   // if any respective thunk feature is enabled.
   bool useIndirectThunkCalls() const {
     return useRetpolineIndirectCalls() || useLVIControlFlowIntegrity();
   }

llvm/test/CodeGen/X86/prefetch.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- -mattr=+sse | FileCheck %s --check-prefix=SSE
 ; RUN: llc < %s -mtriple=i686-- -mattr=+avx | FileCheck %s --check-prefix=SSE
 ; RUN: llc < %s -mtriple=i686-- -mattr=+sse,+prfchw | FileCheck %s -check-prefix=PRFCHWSSE
 ; RUN: llc < %s -mtriple=i686-- -mattr=+prfchw | FileCheck %s -check-prefix=PRFCHWSSE
 ; RUN: llc < %s -mtriple=i686-- -mcpu=slm | FileCheck %s -check-prefix=PRFCHWSSE
 ; RUN: llc < %s -mtriple=i686-- -mcpu=btver2 | FileCheck %s -check-prefix=PRFCHWSSE
 ; RUN: llc < %s -mtriple=i686-- -mcpu=btver2 -mattr=-prfchw | FileCheck %s -check-prefix=SSE
 ; RUN: llc < %s -mtriple=i686-- -mattr=+sse,+prefetchwt1 | FileCheck %s -check-prefix=PREFETCHWT1
 ; RUN: llc < %s -mtriple=i686-- -mattr=-sse,+prefetchwt1 | FileCheck %s -check-prefix=PREFETCHWT1
 ; RUN: llc < %s -mtriple=i686-- -mattr=-sse,+3dnow,+prefetchwt1 | FileCheck %s -check-prefix=PREFETCHWT1
 ; RUN: llc < %s -mtriple=i686-- -mattr=+3dnow | FileCheck %s -check-prefix=3DNOW
 ; RUN: llc < %s -mtriple=i686-- -mattr=+3dnow,+prfchw | FileCheck %s -check-prefix=3DNOW
+; RUN: llc < %s -mtriple=i686-- -mattr=+sse,+prefetchwt1,+prefetchi | FileCheck %s -check-prefix=PREFETCHWT1
+; RUN: llc < %s -mtriple=x86_64-- -mattr=+prefetchwt1,+prefetchi | FileCheck %s -check-prefix=PREFETCHI
 
 ; Rules:
 ; 3dnow by itself get you just the single prefetch instruction with no hints
 ; sse provides prefetch0/1/2/nta
 ; supporting prefetchw, but not 3dnow implicitly provides prefetcht0/1/2/nta regardless of sse setting as we need something to fall back to for the non-write hint.
 ; supporting prefetchwt1 implies prefetcht0/1/2/nta and prefetchw regardless of other settings. this allows levels for non-write and gives us an instruction for write+T0
 ; 3dnow prefetch instruction will only get used if you have no other prefetch instructions enabled
 
 ; rdar://10538297
 
 define void @t(ptr %ptr) nounwind  {
 ; SSE-LABEL: t:
 ; SSE:       # %bb.0: # %entry
 ; SSE-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SSE-NEXT:    prefetcht2 (%eax)
 ; SSE-NEXT:    prefetcht1 (%eax)
 ; SSE-NEXT:    prefetcht0 (%eax)
 ; SSE-NEXT:    prefetchnta (%eax)
 ; SSE-NEXT:    prefetcht2 (%eax)
 ; SSE-NEXT:    prefetcht1 (%eax)
 ; SSE-NEXT:    prefetcht0 (%eax)
 ; SSE-NEXT:    prefetchnta (%eax)
+; SSE-NEXT:    prefetcht0 (%eax)
+; SSE-NEXT:    prefetcht1 (%eax)
 ; SSE-NEXT:    retl
 ;
 ; PRFCHWSSE-LABEL: t:
 ; PRFCHWSSE:       # %bb.0: # %entry
 ; PRFCHWSSE-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; PRFCHWSSE-NEXT:    prefetcht2 (%eax)
 ; PRFCHWSSE-NEXT:    prefetcht1 (%eax)
 ; PRFCHWSSE-NEXT:    prefetcht0 (%eax)
 ; PRFCHWSSE-NEXT:    prefetchnta (%eax)
 ; PRFCHWSSE-NEXT:    prefetchw (%eax)
 ; PRFCHWSSE-NEXT:    prefetchw (%eax)
 ; PRFCHWSSE-NEXT:    prefetchw (%eax)
 ; PRFCHWSSE-NEXT:    prefetchw (%eax)
+; PRFCHWSSE-NEXT:    prefetcht0 (%eax)
+; PRFCHWSSE-NEXT:    prefetcht1 (%eax)
 ; PRFCHWSSE-NEXT:    retl
 ;
 ; PREFETCHWT1-LABEL: t:
 ; PREFETCHWT1:       # %bb.0: # %entry
 ; PREFETCHWT1-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; PREFETCHWT1-NEXT:    prefetcht2 (%eax)
 ; PREFETCHWT1-NEXT:    prefetcht1 (%eax)
 ; PREFETCHWT1-NEXT:    prefetcht0 (%eax)
 ; PREFETCHWT1-NEXT:    prefetchnta (%eax)
 ; PREFETCHWT1-NEXT:    prefetchwt1 (%eax)
 ; PREFETCHWT1-NEXT:    prefetchwt1 (%eax)
 ; PREFETCHWT1-NEXT:    prefetchw (%eax)
 ; PREFETCHWT1-NEXT:    prefetchwt1 (%eax)
+; PREFETCHWT1-NEXT:    prefetcht0 (%eax)
+; PREFETCHWT1-NEXT:    prefetcht1 (%eax)
 ; PREFETCHWT1-NEXT:    retl
 ;
 ; 3DNOW-LABEL: t:
 ; 3DNOW:       # %bb.0: # %entry
 ; 3DNOW-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; 3DNOW-NEXT:    prefetch (%eax)
 ; 3DNOW-NEXT:    prefetch (%eax)
 ; 3DNOW-NEXT:    prefetch (%eax)
 ; 3DNOW-NEXT:    prefetch (%eax)
 ; 3DNOW-NEXT:    prefetchw (%eax)
 ; 3DNOW-NEXT:    prefetchw (%eax)
 ; 3DNOW-NEXT:    prefetchw (%eax)
 ; 3DNOW-NEXT:    prefetchw (%eax)
+; 3DNOW-NEXT:    prefetch (%eax)
+; 3DNOW-NEXT:    prefetch (%eax)
 ; 3DNOW-NEXT:    retl
+;
+; PREFETCHI-LABEL: t:
+; PREFETCHI:       # %bb.0: # %entry
+; PREFETCHI-NEXT:    prefetcht2 (%rdi)
+; PREFETCHI-NEXT:    prefetcht1 (%rdi)
+; PREFETCHI-NEXT:    prefetcht0 (%rdi)
+; PREFETCHI-NEXT:    prefetchnta (%rdi)
+; PREFETCHI-NEXT:    prefetchwt1 (%rdi)
+; PREFETCHI-NEXT:    prefetchwt1 (%rdi)
+; PREFETCHI-NEXT:    prefetchw (%rdi)
+; PREFETCHI-NEXT:    prefetchwt1 (%rdi)
+; PREFETCHI-NEXT:    prefetchit0 (%rdi)
+; PREFETCHI-NEXT:    prefetchit1 (%rdi)
+; PREFETCHI-NEXT:    retq
 entry:
   tail call void @llvm.prefetch( ptr %ptr, i32 0, i32 1, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 0, i32 2, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 0, i32 3, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 0, i32 0, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 1, i32 1, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 1, i32 2, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 1, i32 3, i32 1 )
   tail call void @llvm.prefetch( ptr %ptr, i32 1, i32 0, i32 1 )
+  tail call void @llvm.prefetch( ptr %ptr, i32 1, i32 3, i32 0 )
+  tail call void @llvm.prefetch( ptr %ptr, i32 1, i32 2, i32 0 )
   ret void
 }
 
 declare void @llvm.prefetch(ptr, i32, i32, i32) nounwind

llvm/test/MC/Disassembler/X86/x86-64.txt

 # CHECK: senduipi %r8
 0xf3,0x41,0x0f,0xc7,0xf0
 
 # CHECK: senduipi %r13
 0xf3,0x41,0x0f,0xc7,0xf5
 
 # CHECK: rdpru
 0x0f,0x01,0xfd
+
+# CHECK: prefetchit0 (%rip)
+0x0f,0x18,0x3d,0x00,0x00,0x00,0x00
+
+# CHECK: prefetchit1 (%rip)
+0x0f,0x18,0x35,0x00,0x00,0x00,0x00

llvm/test/MC/X86/PREFETCH-64.s

 // CHECK: prefetchwt1 64(%rdx,%rax) 
 // CHECK: encoding: [0x0f,0x0d,0x54,0x02,0x40]        
 prefetchwt1 64(%rdx,%rax) 
 
 // CHECK: prefetchwt1 (%rdx) 
 // CHECK: encoding: [0x0f,0x0d,0x12]        
 prefetchwt1 (%rdx) 
 
+// CHECK: prefetchit0 485498096
+// CHECK: encoding: [0x0f,0x18,0x3c,0x25,0xf0,0x1c,0xf0,0x1c]
+prefetchit0 485498096
+
+// CHECK: prefetchit0 64(%rdx)
+// CHECK: encoding: [0x0f,0x18,0x7a,0x40]
+prefetchit0 64(%rdx)
+
+// CHECK: prefetchit0 64(%rdx,%rax,4)
+// CHECK: encoding: [0x0f,0x18,0x7c,0x82,0x40]
+prefetchit0 64(%rdx,%rax,4)
+
+// CHECK: prefetchit0 -64(%rdx,%rax,4)
+// CHECK: encoding: [0x0f,0x18,0x7c,0x82,0xc0]
+prefetchit0 -64(%rdx,%rax,4)
+
+// CHECK: prefetchit0 64(%rdx,%rax)
+// CHECK: encoding: [0x0f,0x18,0x7c,0x02,0x40]
+prefetchit0 64(%rdx,%rax)
+
+// CHECK: prefetchit0 (%rdx)
+// CHECK: encoding: [0x0f,0x18,0x3a]
+prefetchit0 (%rdx)
+
+// CHECK: prefetchit1 485498096
+// CHECK: encoding: [0x0f,0x18,0x34,0x25,0xf0,0x1c,0xf0,0x1c]
+prefetchit1 485498096
+
+// CHECK: prefetchit1 64(%rdx)
+// CHECK: encoding: [0x0f,0x18,0x72,0x40]
+prefetchit1 64(%rdx)
+
+// CHECK: prefetchit1 64(%rdx,%rax,4)
+// CHECK: encoding: [0x0f,0x18,0x74,0x82,0x40]
+prefetchit1 64(%rdx,%rax,4)
+
+// CHECK: prefetchit1 -64(%rdx,%rax,4)
+// CHECK: encoding: [0x0f,0x18,0x74,0x82,0xc0]
+prefetchit1 -64(%rdx,%rax,4)
+
+// CHECK: prefetchit1 64(%rdx,%rax)
+// CHECK: encoding: [0x0f,0x18,0x74,0x02,0x40]
+prefetchit1 64(%rdx,%rax)
+
+// CHECK: prefetchit1 (%rdx)
+// CHECK: encoding: [0x0f,0x18,0x32]
+prefetchit1 (%rdx)