[X86] Support AMX fast register allocation

[xiangzhangllvm]

Summary

[X86] Support AMX fast register allocation
The amx programming model that discussed in llvm-dev
(http://lists.llvm.org/pipermail/llvm-dev/2020-August/144302.html).
The amx fast register allocation that discussed in llvm-dev
(https://lists.llvm.org/pipermail/llvm-dev/2021-April/149770.html)
1>
In O0 level, for the customers usually means clang –O0 –S/-c (Front End and Back end both compile in O0 level):
The tile data of amx intrinsic must be loaded before uses, and store into mem after define a tile register.
Some like

----------------------------------------------------------------------
%t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)
%t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)
%t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)
%td = call x86_amx @llvm.x86.tdpbssd.internal(m, n, k, t1, t2, t3)    // key amx intrinsic
call void @llvm.x86.tilestored64.internal(... td)
----------------------------------------------------------------------

Because the life range of tile register is very short (from tileload to tilestore, impossible to spill), we let fast register allocation directly allocate tile registers for them.

As the AMX programming model above show, we need ldtilecfg for each tile register before using them.
So we insert ldtilecfg for every key amx intrinsic (There are 2 reasons do it:
1,we don't much care about the performance at O0. 2,The shapes are very hard to compare at O0 level )
e.g.

----------------------------------------------------------------------
%cfgmem = alloca <16 x i32>, align 4
store <16 x i32> zeroinitializer, <16 x i32>* %cfgmem
call void @llvm.x86.ldtilecfg.internal(i8* %cfgmem)
---------------------------------------------------------------------
%t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)
%t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)
%t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)
%td = call x86_amx @llvm.x86.tdpbssd.internal(m, n, k, t1, t2, t3)    // key amx intrinsic
call void @llvm.x86.tilestored64.internal(... td)
-------------------------------------------------------------------------

But the ldtilecfg need to write the shapes of tile register in its config mem, then we write the shapes before fast register allocation. (it is trouble to do it after register allocation, because the shapes register relocated for AMXinstrinsics may not live at writing position.) But currently, we don’t know for which physic tile register we write the virtual register of shapes ,(because it is before register allocation). So, we just orderly write these shapes into config memory:
e.g.

----------------------------------------------------------------------
%cfgmem = alloca <16 x i32>, align 4                                 * allocate mem
store <16 x i32> zeroinitializer, <16 x i32>* %cfgmem       * zero init
...
//pre-config shape of %t1                                                     *
store volatile i8 %m, i8* %amx.tmm.0.shape.row, align 1     *
store volatile i16 %k, i16* %amx.tmm.0.shape.col, align 2    * pre-config
// pre-config shape of %t2                                                    * shapes
store volatile i8 %k, i8* %amx.tmm.1.shape.row, align 1       *
store volatile i16 %n, i16* %amx.tmm.1.shape.col, align 2    *
// pre-config shape of %t3, %td                                            *
            ….
call void @llvm.x86.ldtilecfg.internal(i8* %cfgmem)              * tile config
-------------------------------------------------------------------------

And then adjust them after fast register allocation.
e.g.
We supposed written the first shape into %amx.tmm.0.shape.row (base + 48), but after fast register allocation if we find the first shape is not corresponding to the first tile register (tmm0), it is corresponding to the 2nd tile register (tmm1), we will adjust the written mem to %amx.tmm.1.shape.row (base + 48 +1).

---------------------------------------------------------------------------
MOV8mi %stack.5, 1, $noreg, 49, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.0.shape.row)
MOV16mr %stack.5, 1, $noreg, 18, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col)
     …
PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
--------------------------------------------------------------------------

2>
For the customers, they usually use clang –O0 –S/-c (Front End and Back end both compile in O0 level).
But for llvm developers, we may usually let Front End build with O1/2/… and Back End build in O0 (e.g.: clang –O0 –S –emit-llvm + llc –O0)

Considering this way is not the main way of building program and let the upper algorithm works too, I “volatiles” the tile data of key AMX intrinsic in pass “Lower AMX type for load/store”, just let it like in clang –O0, all tile data of key AMX intrinsic must be loaded before uses, and stored into mem after define a tile register. Because the Back End build it in O0, so here we don’t consider the performance, just care about the correctness.

[pengfei]

llvm/lib/Target/X86/X86ExpandPseudo.cpp

483

Maybe don't need pseudo instruction?

llvm/lib/Target/X86/X86FastTileConfig.cpp

26

You can remove it if you don't use it.

28

Same above.

37

Same above.

52–55

You can remove them if you don't use.

63

Naming conversions: function should start with lower case letter.

111–114

You can remove it if you don't require any analysis.

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

4705

You can move it to .td file.

llvm/lib/Target/X86/X86LowerAMXType.cpp

521

You may need to exclude debug intrinsics.

529

I maybe not the last phi.

596

Do you also insert a store for load intrinsic?

632

We may need to postpone it after we find a AMX intrinsic.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

60

You may need to exclude debug intrinsics.

173

Should be V512Ty?

193

Better add assert for this case.

291

Better add assert to check I is load.

356

Do we need to check it since the pass is only created under O0?

llvm/test/CodeGen/X86/AMX/amx-configO2toO0.ll

2

Better change one case to use avx or sse to check if stack cleared correctly.

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

416

The shapes for tmm0 is ax and cx, but the stored shape in stack is $sil and 8?

441

I don't find where we shore this shape.

llvm/utils/gn/secondary/llvm/lib/Target/X86/BUILD.gn

113

Why don't add X86FastTileConfig?

[xiangzhangllvm]

Address Pengfei's comments:

llvm/lib/Target/X86/X86ExpandPseudo.cpp

483

Yes, I'll change it (duo to some history I used pseudo instruction.)

llvm/lib/Target/X86/X86LowerAMXType.cpp

521

PHI's operands shouldn't be debug intrinsics.

529

Here no need domination relation, I'll remove it, (for some history reason, I put allocation instruction in dominated BB)

596

If the tileload comes from user's source code, O0 or my patch will generate for it.
We shouldn't generate tilestore for auto generated tileload.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

60

debug IRs are not IntrinsicInst.

173

we set the cfg mem align to 4 Bytes before, so here sync with it.

291

BB.end() is possible.

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

416

Here is a mistake, I'll fix it, thanks!

441

above mov8/16* are pre-generated before register allocation, they will adjust their store position after this pass tested.

[pengfei]

Just one thought: Is there a case user specifies --regalloc=fast with O2 in llc? Can we handle this case?

Thanks
Pengfei

[xiangzhangllvm]

We didn't cover this case currently, also O0 + greedy is not support.

[xiangzhangllvm]

llvm/lib/Target/X86/X86PreAMXConfig.cpp

193

Multi-TileStores of tile definition is not fit current Volatile Model, it will failed in assert.

llvm/test/CodeGen/X86/AMX/amx-configO2toO0.ll

2

I'll add it, thanks!

[pengfei]

llvm/lib/CodeGen/TargetPassConfig.cpp

1321

Will this affect other targets? I think being called by FastRA might not be expected for them.

llvm/lib/Target/X86/X86FastTileConfig.cpp

45–46

A bit strange leave them uninitialized.

132

You should exclude debug MI here.

135

Extra spaces.

182

Do you need to check MO's register class? If the KeyMI is store, you will save its tile to shape?

193

If the number is consecutive, use SmallVector should have better performance.

198

Is it possible the shapes are not in current BB? E.g. The previous BB been split etc.

230

Should palette = 1?

273

Nit. Maybe better define it return void and check ShapedTiles.size() in assert.

284

I think it's better to collect the shape config here.

llvm/lib/Target/X86/X86LowerAMXIntrinsics.cpp

56–58

You can move it into the namespace. By the way, clang-format.

llvm/lib/Target/X86/X86LowerAMXType.cpp

20

necessary

389

Better use cast to help check for failures.

393

Will II be nullptr?

511

Can we handle the phi used by another phi?

634

Is this necessary for O0?

[pengfei]

llvm/lib/Target/X86/X86PreAMXConfig.cpp

206

You can check Loads[0] directly.

217

Why can't be DPSSD etc.?

llvm/test/CodeGen/X86/AMX/amx-configO0toO0.ll

3

You don't need prefix for the single RUN. The same below.

llvm/test/CodeGen/X86/AMX/amx-configO2toO0.ll

40

You should always set palette = 1 after it.

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

32

But the alignment of store and alloca is not match. You may cause runtime crush due the the alignment.

[xiangzhangllvm]

Address pengfei's comments:

llvm/lib/Target/X86/X86FastTileConfig.cpp

182

Checked at 176, MO must be TilePhysReg, Yes, KeyMI's all tile operands' shapes should be saved.
KeyMI can never be a tilestore, because for a volatile model, tile data in tilestore must comes from tileload.
So the KeyMI prefer tileload. line 154 in getKeyAMXInstr should never happen, I'll replace it with a assert.

193

because we don't know the num of the shapes at first, we may meet the shapes not in order.

198

I think it is impossible in O0

230

Yes, I miss it, thanks!

284

Do you mean materializeTileCfg(MI) here ? It will modify the MBB, that will broken the iterators of MBB

llvm/lib/Target/X86/X86LowerAMXIntrinsics.cpp

56–58

I think that is not sensitive for a static opt.

llvm/lib/Target/X86/X86LowerAMXType.cpp

389

the caller passed IsPHI will make sure it is phi.

393

It can't be nullptr, Current all tile def should comes from IntrinsicInst. (PHI has specially handled)

511

I begin to think this case, I think it should never happened, do we have meet Recursive PHI before? I think the " Recursive PHI" should be PHI which has more than 2 operands.

634

Greedy also need this AMX Lower Type pass.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

206

What we care is that there should be only 1 tileload for tilestore.

llvm/test/CodeGen/X86/AMX/amx-configO0toO0.ll

3

you mean --check-prefix=AMX_O0 ? I just thought it is more clear for this test.

[xiangzhangllvm]

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

32

Make sense, thanks!

[xiangzhangllvm]

The patch is big, thanks for Pengfei's review!

llvm/lib/Target/X86/X86PreAMXConfig.cpp

217

// here "be stored" means "be stored into mem" not "be TileStore instruction", I'll refine this comments.

356

it checked at its caller.

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

32

updated in other tests, e.g. llvm/test/CodeGen/X86/AMX/amx-configO2toO0-precfg.ll : line 30

[xiangzhangllvm]

Refine Shape info:
In AMX intrinsics we let Shape = {Row, Col (Bytes) }, but the RealCol = Col / ElementSize. We may use the RealCol as a new Row for other new created AMX intrinsics.

[LuoYuanke]

llvm/lib/Target/X86/X86LowerAMXType.cpp

92

Do we need to calculate the value in compile time if V is constant?

llvm/lib/Target/X86/X86PreAMXConfig.cpp

70

I don't understand the function. Does it mean only tilezero intrinsic return true?

[xiangzhangllvm]

llvm/lib/Target/X86/X86LowerAMXType.cpp

92

CreateUDiv already handle it.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

70

And currently tileload too, this function return is "intrinsic only def tile (not use tile)" or not.

[pengfei]

llvm/include/llvm/CodeGen/CodeGenPassBuilder.h

1042 ↗

(On Diff #338692)

Is this function actually used?

llvm/lib/Target/X86/X86LowerAMXType.cpp

69

Better move it to class X86LowerAMXType

348–350

You can make it be member of X86VolatileTileData. Then you don't need to calculate F here.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

206–208

return preWriteTileCfg(I8Ptr, Cfg, Shapes); ?

325

Does it cause warning in release build?

[xiangzhangllvm]

I'll try build it in release, make sure it no warnings, thanks!

llvm/include/llvm/CodeGen/CodeGenPassBuilder.h

1042 ↗

(On Diff #338692)

I think I mix with the same named function "bool addPostFastRegAllocRewrite( )", thank you!

llvm/lib/Target/X86/X86LowerAMXType.cpp

69

I thought it before, here I just won't to pass it into static function (getRowFromCol).

348–350

this is in static function, not function of X86VolatileTileData.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

206–208

Let's return Init0, cover the assert(Init0 && ...), thanks!

325

I think yes, good catch, thanks!

[xiangzhangllvm]

Refined: No new warning in Release build.

[pengfei]

You can find more information about coding style here: https://llvm.org/docs/CodingStandards.html#don-t-use-braces-on-simple-single-statement-bodies-of-if-else-loop-statements.

llvm/lib/Target/X86/X86FastTileConfig.cpp

298

Should this return true based on tilecfg rewrite?

llvm/lib/Target/X86/X86LowerAMXType.cpp

391

Coding style: Use {} when if uses it.

629–634

Remove unused code.

llvm/lib/Target/X86/X86PreAMXConfig.cpp

268

Maybe better to use BasicBlock::iterator &, then you don't need to return it.

282

Coding style: Use {} when the else uses it.

372–374

Coding style: Don't use {} for single line.

[pengfei]

llvm/lib/Target/X86/X86LowerAMXType.cpp

112

Can we always get the shape by the def of a AMX. Then we can avoid to add calculation for it.
We can add assert here to make sure of it.

[xiangzhangllvm]

llvm/lib/Target/X86/X86LowerAMXType.cpp

112

We just can pass Element Size into the AMX intrinsic def. The calculation still need.

[xiangzhangllvm]

llvm/lib/Target/X86/X86PreAMXConfig.cpp

268

Let I = getShapesAndConfigPosEnd(...) in caller is more readable, In my eye.

[pengfei]

The implementation is good to me in general. Let's wait a few days to see opinions from community and other reviewers.

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

32

Nit: the source lacks palette initialization. It depends on you add it or not.

[xiangzhangllvm]

Thanks for your reviewing!

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

32

Yes, here doesn't matter, this test focus on checking rewiting shapes after fast register allocation.
So I didn't write the palette in the test.
In other tests (amx-configO2toO0.ll, amx-configO0toO0.ll) we can see the palette is set to 1.

[xiangzhangllvm]

rebase

[xiangzhangllvm]

Use clang-format refine.
And add Fixme: at LowAMXType

[xiangzhangllvm]

llvm/lib/Target/X86/X86LowerAMXType.cpp

111

And add Fixme: at LowAMXType

[bkramer]

llvm/lib/Target/X86/X86LowerAMXType.cpp

69

You can't simply have global state here, it doesn't work in a multithreaded environment. I reverted this change in df323ba445f7fc4d29def8950e80dec6ba487961 because it breaks us.

[xiangzhangllvm]

Fix bkramer's multithread problem.

[bkramer]

That's better, thanks!

[xiangzhangllvm]

refine, compatible with old AMX API

llvm/include/llvm/CodeGen/Passes.h

namespace llvm {
   /// The pass fixups statepoint machine instruction to replace usage of
   /// caller saved registers with stack slots.
   extern char &FixupStatepointCallerSavedID;
 
   /// The pass transforms load/store <256 x i32> to AMX load/store intrinsics
   /// or split the data to two <128 x i32>.
   FunctionPass *createX86LowerAMXTypePass();
 
+  /// The pass insert tile config intrinsics for AMX fast register allocation.
+  FunctionPass *createX86PreAMXConfigPass();
+
   /// The pass transforms amx intrinsics to scalar operation if the function has
   /// optnone attribute or it is O0.
   FunctionPass *createX86LowerAMXIntrinsicsPass();
 } // End llvm namespace
 
 #endif

llvm/include/llvm/IR/IntrinsicsX86.td

def int_x86_tdpbuud : GCCBuiltin<"__builtin_ia32_tdpbuud">,
               Intrinsic<[], [llvm_i8_ty, llvm_i8_ty, llvm_i8_ty],
                         [ImmArg<ArgIndex<0>>, ImmArg<ArgIndex<1>>,
                          ImmArg<ArgIndex<2>>]>;
   def int_x86_tdpbf16ps : GCCBuiltin<"__builtin_ia32_tdpbf16ps">,
               Intrinsic<[], [llvm_i8_ty, llvm_i8_ty, llvm_i8_ty],
                         [ImmArg<ArgIndex<0>>, ImmArg<ArgIndex<1>>,
                          ImmArg<ArgIndex<2>>]>;
   // AMX - internal intrinsics
+  def int_x86_ldtilecfg_internal :
+              Intrinsic<[], [llvm_ptr_ty], []>;
   def int_x86_tileloadd64_internal :
               GCCBuiltin<"__builtin_ia32_tileloadd64_internal">,
               Intrinsic<[llvm_x86amx_ty],
                         [llvm_i16_ty, llvm_i16_ty, llvm_ptr_ty, llvm_i64_ty],
                         []>;
   def int_x86_tdpbssd_internal :
               GCCBuiltin<"__builtin_ia32_tdpbssd_internal">,
               Intrinsic<[llvm_x86amx_ty],

llvm/lib/CodeGen/TargetPassConfig.cpp

 }
 
 bool TargetPassConfig::addRegAssignAndRewriteFast() {
   if (RegAlloc != &useDefaultRegisterAllocator &&
       RegAlloc != &createFastRegisterAllocator)
     report_fatal_error("Must use fast (default) register allocator for unoptimized regalloc.");
 
   addPass(createRegAllocPass(false));
+
+  // Allow targets to change the register assignments before rewriting.
+  addPreRewrite();
   return true;
 }
 
 bool TargetPassConfig::addRegAssignAndRewriteOptimized() {
   // Add the selected register allocation pass.
   addPass(createRegAllocPass(true));
 
   // Allow targets to change the register assignments before rewriting.

llvm/lib/Target/X86/CMakeLists.txt

set(sources
   X86CallFrameOptimization.cpp
   X86CallingConv.cpp
   X86CallLowering.cpp
   X86CmovConversion.cpp
   X86DomainReassignment.cpp
   X86DiscriminateMemOps.cpp
   X86LowerTileCopy.cpp
   X86LowerAMXType.cpp
+  X86PreAMXConfig.cpp
   X86LowerAMXIntrinsics.cpp
   X86TileConfig.cpp
+  X86FastTileConfig.cpp
   X86PreTileConfig.cpp
   X86ExpandPseudo.cpp
   X86FastISel.cpp
   X86FixupBWInsts.cpp
   X86FixupLEAs.cpp
   X86AvoidStoreForwardingBlocks.cpp
   X86FixupSetCC.cpp
   X86FlagsCopyLowering.cpp

llvm/lib/Target/X86/X86.h

 FunctionPass *createX86FlagsCopyLoweringPass();
 
 /// Return a pass that expands WinAlloca pseudo-instructions.
 FunctionPass *createX86WinAllocaExpander();
 
 /// Return a pass that config the tile registers.
 FunctionPass *createX86TileConfigPass();
 
+/// Return a pass that config the tile registers after fast reg allocation.
+FunctionPass *createX86FastTileConfigPass();
+
 /// Return a pass that insert pseudo tile config instruction.
 FunctionPass *createX86PreTileConfigPass();
 
 /// Return a pass that lower the tile copy instruction.
 FunctionPass *createX86LowerTileCopyPass();
 
 /// Return a pass that inserts int3 at the end of the function if it ends with a
 /// CALL instruction. The pass does the same for each funclet as well. This

 void initializeX86FlagsCopyLoweringPassPass(PassRegistry &);
 void initializeX86LoadValueInjectionLoadHardeningPassPass(PassRegistry &);
 void initializeX86LoadValueInjectionRetHardeningPassPass(PassRegistry &);
 void initializeX86OptimizeLEAPassPass(PassRegistry &);
 void initializeX86PartialReductionPass(PassRegistry &);
 void initializeX86SpeculativeLoadHardeningPassPass(PassRegistry &);
 void initializeX86SpeculativeExecutionSideEffectSuppressionPass(PassRegistry &);
 void initializeX86PreTileConfigPass(PassRegistry &);
+void initializeX86FastTileConfigPass(PassRegistry &);
 void initializeX86TileConfigPass(PassRegistry &);
 void initializeX86LowerAMXTypeLegacyPassPass(PassRegistry &);
+void initializeX86PreAMXConfigPassPass(PassRegistry &);
 void initializeX86LowerTileCopyPass(PassRegistry &);
 void initializeX86LowerAMXIntrinsicsLegacyPassPass(PassRegistry &);
 
 namespace X86AS {
 enum : unsigned {
   GS = 256,
   FS = 257,
   SS = 258,

llvm/lib/Target/X86/X86ExpandPseudo.cpp

case TargetOpcode::ICALL_BRANCH_FUNNEL:
     ExpandICallBranchFunnel(&MBB, MBBI);
     return true;
   case X86::PTILELOADDV: {
     for (unsigned i = 2; i > 0; --i)
       MI.RemoveOperand(i);
     MI.setDesc(TII->get(X86::TILELOADD));
     return true;
   }
+  case X86::PLDTILECFGV: {
+    MI.setDesc(TII->get(X86::LDTILECFG));
+    return true;
+  }
   case X86::PTDPBSSDV:
   case X86::PTDPBSUDV:
   case X86::PTDPBUSDV:
   case X86::PTDPBUUDV:
   case X86::PTDPBF16PSV: {
     MI.untieRegOperand(4);
     for (unsigned i = 3; i > 0; --i)
       MI.RemoveOperand(i);

llvm/lib/Target/X86/X86FastTileConfig.cpp

+//===-- X86FastTileConfig.cpp - Fast Tile Register Configure---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file Pass to config the shape of AMX physical registers
+/// AMX register need to be configured before use. Before FastRegAllocation pass
+/// the pldtilecfg instruction is inserted, however at that time we don't
+/// know the shape of each physical tile registers, because the register
+/// allocation is not done yet. This pass runs after egister allocation
+/// pass. It collects the shape information of each physical tile register
+/// and store the shape in the stack slot that is allocated for load config
+/// to tile config register.
+//
+//===----------------------------------------------------------------------===//
+
+#include "X86.h"
+#include "X86InstrBuilder.h"
+#include "X86MachineFunctionInfo.h"
+#include "X86RegisterInfo.h"
+#include "X86Subtarget.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/PostOrderIterator.h"
+#include "llvm/CodeGen/LiveIntervals.h"
+#include "llvm/CodeGen/MachineDominators.h"
+#include "llvm/CodeGen/MachineFrameInfo.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstr.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/TargetInstrInfo.h"
+#include "llvm/CodeGen/TargetRegisterInfo.h"
+#include "llvm/CodeGen/TileShapeInfo.h"
+#include "llvm/CodeGen/VirtRegMap.h"
+#include "llvm/InitializePasses.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "fasttileconfig"
+
+namespace {
+
+class X86FastTileConfig : public MachineFunctionPass {
+  // context
+  MachineFunction *MF = nullptr;
+  const X86Subtarget *ST = nullptr;
+  const TargetRegisterInfo *TRI;
+  const TargetInstrInfo *TII;
+  MachineDominatorTree *DomTree = nullptr;
+  MachineRegisterInfo *MRI = nullptr;
+  VirtRegMap *VRM = nullptr;
+  LiveIntervals *LIS = nullptr;
+
+  MachineInstr *getTileConfigPoint();
+  void tileConfig();
+
+public:
+  X86FastTileConfig() : MachineFunctionPass(ID) {}
+
+  void FastTileConfig();
+  bool IsTileLoad(MachineInstr &MI);
+  bool IsTileStore(MachineInstr &MI);
+  bool IsAMXInstr(MachineInstr &MI);
+  void getTileStoreShape(MachineInstr &MI,
+                         SmallVector<MachineOperand *> &ShapedTiles);
+
+  MachineInstr *getKeyAMXInstr(MachineInstr *MI);
+  unsigned getTileShapesCfg(MachineInstr *MI,
+                            SmallVector<MachineOperand *> &ShapedTiles);
+  void getShapeCfgInstrs(MachineInstr *MI,
+                         std::map<unsigned, MachineInstr *> &RowCfgs,
+                         std::map<unsigned, MachineInstr *> &ColCfgs);
+
+  /// Return the pass name.
+  StringRef getPassName() const override {
+    return "Fast Tile Register Configure";
+  }
+
+  /// X86FastTileConfig analysis usage.
+  void getAnalysisUsage(AnalysisUsage &AU) const override;
+
+  void materializeTileCfg(MachineInstr *MI);
+
+  void rewriteTileCfg(SmallVector<MachineOperand *> &ShapedTiles,
+                      std::map<unsigned, MachineInstr *> &RowCfgs,
+                      std::map<unsigned, MachineInstr *> &ColCfgs);
+
+  /// Perform register allocation.
+  bool runOnMachineFunction(MachineFunction &mf) override;
+
+  MachineFunctionProperties getRequiredProperties() const override {
+    return MachineFunctionProperties().set(
+        MachineFunctionProperties::Property::NoPHIs);
+  }
+
+  static char ID;
+};
+
+} // end anonymous namespace
+
+char X86FastTileConfig::ID = 0;
+
+INITIALIZE_PASS_BEGIN(X86FastTileConfig, DEBUG_TYPE,
+                      "Fast Tile Register Configure", false, false)
+INITIALIZE_PASS_END(X86FastTileConfig, DEBUG_TYPE,
+                    "Fast Tile Register Configure", false, false)
+
+void X86FastTileConfig::getAnalysisUsage(AnalysisUsage &AU) const {
+  AU.setPreservesAll();
+  MachineFunctionPass::getAnalysisUsage(AU);
+}
+
+static bool isTilePhysReg(MachineOperand &Op) {
+  if (!Op.isReg())
+    return false;
+
+  Register Reg = Op.getReg();
+  if (Reg >= X86::TMM0 && Reg <= X86::TMM7)
+    return true;
+  return false;
+}
+
+static unsigned getTilePhysRegIdx(MachineOperand *Op) {
+  assert(isTilePhysReg(*Op) && "Tile Operand is invalid");
+  return Op->getReg() - X86::TMM0;
+}
+
+static inline void adjustRowCfg(unsigned TIdx, MachineInstr *MI) {
+  unsigned Offset = 48 + TIdx;
+  MI->getOperand(3).ChangeToImmediate(Offset);
+}
+
+static inline void adjustColCfg(unsigned TIdx, MachineInstr *MI) {
+  unsigned Offset = 16 + TIdx * 2;
+  MI->getOperand(3).ChangeToImmediate(Offset);
+}
+
+bool X86FastTileConfig::IsTileLoad(MachineInstr &MI) {
+  return MI.getOpcode() == X86::PTILELOADDV;
+}
+bool X86FastTileConfig::IsTileStore(MachineInstr &MI) {
+  return MI.getOpcode() == X86::PTILESTOREDV;
+}
+bool X86FastTileConfig::IsAMXInstr(MachineInstr &MI) {
+  for (MachineOperand &MO : MI.operands())
+    if (isTilePhysReg(MO))
+      return true;
+  // TODO: May need to handle some special nontile amx instrucion.
+  return false;
+}
+
+MachineInstr *X86FastTileConfig::getKeyAMXInstr(MachineInstr *MI) {
+  auto Cfg = MachineBasicBlock::iterator(MI);
+  MachineBasicBlock *MBB = MI->getParent();
+  MachineInstr *KeyMI = nullptr;
+  int KeyAMXNum = 0;
+
+  for (auto II = Cfg; II != MBB->end(); II++) {
+    if (IsTileLoad(*II)) {
+      KeyMI = &*II;
+      continue;
+    }
+    if (IsTileStore(*II)) {
+      if (!KeyMI)
+        KeyMI = &*II;
+      else
+        break;
+    }
+    if (IsAMXInstr(*II)) {
+      assert((KeyAMXNum == 0) && "Too many Key AMX instruction!");
+      KeyAMXNum++;
+      KeyMI = &*II;
+    }
+  }
+  assert(KeyMI && "There must be an AMX instruction.");
+  return KeyMI;
+}
+
+// Orderly get the tiles in key amx instruction, uses before defs.
+unsigned X86FastTileConfig::getTileShapesCfg(
+    MachineInstr *CfgMI, SmallVector<MachineOperand *> &ShapedTiles) {
+  MachineInstr *KeyMI = getKeyAMXInstr(CfgMI);
+
+  SmallVector<MachineOperand *> DefTiles;
+  for (MachineOperand &MO : KeyMI->operands()) {
+    if (!isTilePhysReg(MO))
+      continue;
+    if (MO.isDef())
+      DefTiles.push_back(&MO);
+    else
+      ShapedTiles.push_back(&MO);
+  }
+  ShapedTiles.append(DefTiles);
+
+  return ShapedTiles.size();
+}
+
+// We pre-config the shapes at position named with "amx.tmm.N.shape.row* and
+// amx.shape.N.col*" at pass "Pre AMX Tile Config".
+// The 'N' implies the order of tiles in key amx intrinsic.
+void X86FastTileConfig::getShapeCfgInstrs(
+    MachineInstr *MI, std::map<unsigned, MachineInstr *> &RowCfgs,
+    std::map<unsigned, MachineInstr *> &ColCfgs) {
+  auto Cfg = MachineBasicBlock::iterator(MI);
+  MachineBasicBlock *MBB = MI->getParent();
+
+  for (auto II = Cfg; II != MBB->begin(); II--) {
+    if (IsAMXInstr(*II) || II->isTerminator() || II->isCall())
+      break;
+    if (!II->mayStore() || !II->hasOneMemOperand())
+      continue;
+    const Value *MemPtr = II->memoperands()[0]->getValue();
+    if (!MemPtr)
+      continue;
+
+    StringRef Name = MemPtr->getName();
+    if (!Name.startswith("amx.tmm."))
+      continue;
+
+    // Get the 'N'th tile shape config in key amx instruction.
+    auto N = Name.find(".shape");
+    StringRef STileIdx = Name.slice(8, N);
+    unsigned Idx;
+    STileIdx.getAsInteger(10, Idx);
+
+    // And related them with their store instructions.
+    if (Name.contains("row"))
+      RowCfgs[Idx] = &*II;
+    else if (Name.contains("col"))
+      ColCfgs[Idx] = &*II;
+    else
+      llvm_unreachable("Invalid tile shape info!");
+  }
+  assert((RowCfgs.size() == ColCfgs.size()) &&
+         "The number of tile row and col must be equal!");
+}
+
+// Here is the data format for the tile config.
+// 0      palette   = 0 now.
+// 1      start_row = 0 now.
+// 2-15   reserved, must be zero
+// 16-17  tile0.colsb Tile 0 bytes per row.
+// 18-19  tile1.colsb Tile 1 bytes per row.
+// 20-21  tile2.colsb Tile 2 bytes per row.
+// ... (sequence continues)
+// 30-31  tile7.colsb Tile 7 bytes per row.
+// 32-47  reserved, must be zero
+// 48     tile0.rows Tile 0 rows.
+// 49     tile1.rows Tile 1 rows.
+// 50     tile2.rows Tile 2 rows.
+// ... (sequence continues)
+// 55     tile7.rows Tile 7 rows.
+// 56-63  reserved, must be zero
+void X86FastTileConfig::rewriteTileCfg(
+    SmallVector<MachineOperand *> &ShapedTiles,
+    std::map<unsigned, MachineInstr *> &RowCfgs,
+    std::map<unsigned, MachineInstr *> &ColCfgs) {
+  assert((RowCfgs.size() == ShapedTiles.size()) &&
+         "The number of tile shapes not equal with the number of tiles!");
+
+  // Orderly get the tiles and adjust the shape config.
+  for (unsigned i = 0, e = ShapedTiles.size(); i < e; i++) {
+    MachineOperand *MO = ShapedTiles[i];
+    unsigned TmmIdx = getTilePhysRegIdx(MO);
+    if (i == TmmIdx)
+      continue;
+    adjustRowCfg(TmmIdx, RowCfgs[i]);
+    adjustColCfg(TmmIdx, ColCfgs[i]);
+  }
+}
+
+// We have already preconfig the shapes before fast register allocation at
+// X86PreAMXConfig::preWriteTileCfg(). Now, we have done fast register
+// allocation, the shapes pre-written before may not rightly corresponding
+// to the correct tmm registers, so we need adjust them.
+void X86FastTileConfig::materializeTileCfg(MachineInstr *CfgMI) {
+  SmallVector<MachineOperand *> ShapedTiles;
+  std::map<unsigned, MachineInstr *> RowCfgs;
+  std::map<unsigned, MachineInstr *> ColCfgs;
+
+  // Orderly keep the tile uses and def in ShapedTiles;
+  unsigned NumCfg = getTileShapesCfg(CfgMI, ShapedTiles);
+  assert(NumCfg && "Not find shapes config!");
+
+  getShapeCfgInstrs(CfgMI, RowCfgs, ColCfgs);
+
+  rewriteTileCfg(ShapedTiles, RowCfgs, ColCfgs);
+}
+
+void X86FastTileConfig::FastTileConfig() {
+  for (MachineBasicBlock &MBB : *MF) {
+    SmallVector<MachineInstr *, 2> CFGs;
+    for (MachineInstr &MI : MBB)
+      if (MI.getOpcode() == X86::PLDTILECFGV)
+        CFGs.push_back(&MI);
+    for (auto *MI : CFGs)
+      materializeTileCfg(MI);
+  }
+}
+
+bool X86FastTileConfig::runOnMachineFunction(MachineFunction &mf) {
+  MF = &mf;
+  MRI = &mf.getRegInfo();
+  ST = &mf.getSubtarget<X86Subtarget>();
+  TRI = ST->getRegisterInfo();
+  TII = mf.getSubtarget().getInstrInfo();
+
+  FastTileConfig();
+  return true;
+}
+
+FunctionPass *llvm::createX86FastTileConfigPass() {
+  return new X86FastTileConfig();
+}

llvm/lib/Target/X86/X86ISelDAGToDAG.cpp

case Intrinsic::x86_tilestored64_internal: {
                        Disp,
                        Segment,
                        Node->getOperand(6),
                        Chain};
       CNode = CurDAG->getMachineNode(Opc, dl, MVT::Other, Ops);
       ReplaceNode(Node, CNode);
       return;
     }
+    case Intrinsic::x86_ldtilecfg_internal: {
+      unsigned Opc = X86::PLDTILECFGV;
+      SDValue Base = Node->getOperand(2);
+      SDValue Scale = getI8Imm(1, dl);
+      SDValue Index = CurDAG->getRegister(0, MVT::i32);
+      SDValue Disp = CurDAG->getTargetConstant(0, dl, MVT::i32);
+      SDValue Segment = CurDAG->getRegister(0, MVT::i16);
+      SDValue Chain = Node->getOperand(0);
+      MachineSDNode *CNode;
+      SDValue Ops[] = {Base,
+                       Scale,
+                       Index,
+                       Disp,
+                       Segment,
+                       Chain};
+      CNode = CurDAG->getMachineNode(Opc, dl, MVT::Other, Ops);
+      ReplaceNode(Node, CNode);
+      return;
+    }
     case Intrinsic::x86_tileloadd64:
     case Intrinsic::x86_tileloaddt164:
     case Intrinsic::x86_tilestored64: {
       if (!Subtarget->hasAMXTILE())
         break;
       unsigned Opc;
       switch (IntNo) {
       default: llvm_unreachable("Unexpected intrinsic!");

llvm/lib/Target/X86/X86InstrAMX.td

let SchedRW = [WriteSystem] in {
     def TILEZERO : I<0x49, MRMr0, (outs TILE:$dst), (ins),
                      "tilezero\t$dst", []>,
                      VEX, T8XD;
 
     // Pseduo instruction for RA.
     def PTILELOADDV : PseudoI<(outs TILE:$dst), (ins GR16:$src1,
                                                      GR16:$src2,
                                                      opaquemem:$src3), []>;
+    def PLDTILECFGV: PseudoI<(outs), (ins opaquemem:$src1),[]>;
     def PTILESTOREDV : PseudoI<(outs), (ins GR16:$src1,
                                             GR16:$src2, opaquemem:$src3,
                                             TILE:$src4), []>;
     let isReMaterializable = 1, isAsCheapAsAMove = 1, canFoldAsLoad = 1 in
       def PTILEZEROV : PseudoI<(outs TILE:$dst), (ins GR16:$src1, GR16:$src2),
                                 [(set TILE:$dst, (int_x86_tilezero_internal
                                   GR16:$src1, GR16:$src2))]>;
 

llvm/lib/Target/X86/X86LowerAMXIntrinsics.cpp

 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/IntrinsicsX86.h"
 #include "llvm/IR/PatternMatch.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
+#include "llvm/Support/CommandLine.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/LoopUtils.h"
 
 using namespace llvm;
 using namespace PatternMatch;
 
 #define DEBUG_TYPE "lower-amx-intrinsics"
 
 #ifndef NDEBUG
 static bool isV256I32Ty(Type *Ty) {
   if (auto *FVT = dyn_cast<FixedVectorType>(Ty))
     return FVT->getNumElements() == 256 &&
            FVT->getElementType()->isIntegerTy(32);
   return false;
 }
 #endif
 
+static cl::opt<bool> X86ScalarizeAMX("enable-x86-scalar-amx", cl::init(false),
+                       cl::Hidden, cl::desc("X86: enable AMX scalarizition."));
+
 namespace {
 class X86LowerAMXIntrinsics {
   Function &Func;
 
 public:
   X86LowerAMXIntrinsics(Function &F, DomTreeUpdater &DomTU, LoopInfo *LoopI)
       : Func(F), DTU(DomTU), LI(LoopI) {}
   bool visit();
private:
   bool lowerTileLoadStore(Instruction *TileLoadStore);
   template <Intrinsic::ID IntrID>
   typename std::enable_if<IntrID == Intrinsic::x86_tdpbssd_internal ||
                               IntrID == Intrinsic::x86_tdpbf16ps_internal,
                           bool>::type
   lowerTileDP(Instruction *TileDP);
   bool lowerTileZero(Instruction *TileZero);
 };
+} // anonymous namespace
 
 BasicBlock *X86LowerAMXIntrinsics::createLoop(BasicBlock *Preheader,
                                               BasicBlock *Exit, Value *Bound,
                                               Value *Step, StringRef Name,
                                               IRBuilderBase &B, Loop *L) {
   LLVMContext &Ctx = Preheader->getContext();
   BasicBlock *Header =
       BasicBlock::Create(Ctx, Name + ".header", Preheader->getParent(), Exit);
case Intrinsic::x86_tilezero_internal:
       break;
     default:
       llvm_unreachable("invalid amx intrinsics!");
     }
   }
 
   return C;
 }
-} // anonymous namespace
-
-namespace {
 
 class X86LowerAMXIntrinsicsLegacyPass : public FunctionPass {
 public:
   static char ID;
 
   X86LowerAMXIntrinsicsLegacyPass() : FunctionPass(ID) {
     initializeX86LowerAMXIntrinsicsLegacyPassPass(
         *PassRegistry::getPassRegistry());
   }
 
   bool runOnFunction(Function &F) override {
+    if (!X86ScalarizeAMX)
+      return false;
     TargetMachine *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
     if (!F.hasFnAttribute(Attribute::OptimizeNone) &&
         TM->getOptLevel() != CodeGenOpt::None)
       return false;
 
     auto *DTWP = getAnalysisIfAvailable<DominatorTreeWrapperPass>();
     auto *DT = DTWP ? &DTWP->getDomTree() : nullptr;
     auto *LIWP = getAnalysisIfAvailable<LoopInfoWrapperPass>();
     auto *LI = LIWP ? &LIWP->getLoopInfo() : nullptr;
     DomTreeUpdater DTU(DT, DomTreeUpdater::UpdateStrategy::Lazy);
 
     X86LowerAMXIntrinsics LAT(F, DTU, LI);
     return LAT.visit();
   }
   StringRef getPassName() const override { return "Lower AMX intrinsics"; }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addPreserved<DominatorTreeWrapperPass>();
     AU.addPreserved<LoopInfoWrapperPass>();
     AU.addRequired<TargetPassConfig>();
   }
 };
 
-} // anonymous namespace
-
 static const char PassName[] = "Lower AMX intrinsics";
 char X86LowerAMXIntrinsicsLegacyPass::ID = 0;
 INITIALIZE_PASS_BEGIN(X86LowerAMXIntrinsicsLegacyPass, DEBUG_TYPE, PassName,
                       false, false)
 INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
 INITIALIZE_PASS_END(X86LowerAMXIntrinsicsLegacyPass, DEBUG_TYPE, PassName,
                     false, false)
 
 FunctionPass *llvm::createX86LowerAMXIntrinsicsPass() {
   return new X86LowerAMXIntrinsicsLegacyPass();
 }

llvm/lib/Target/X86/X86LowerAMXType.cpp

-//===- llvm/CodeGen/TileShapeInfo.h - ---------------------------*- C++ -*-===//
+//===- Target/X86/X86LowerAMXType.cpp - -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file Pass to transform <256 x i32> load/store
 /// <256 x i32> is bitcasted to x86_amx on X86, and AMX instruction set only
 /// provides simple operation on x86_amx. The basic elementwise operation
 /// is not supported by AMX. Since x86_amx is bitcasted from vector <256 x i32>
 /// and only AMX intrinsics can operate on the type, we need transform
 /// load/store <256 x i32> instruction to AMX load/store. If the bitcast can
 /// not be combined with load/store, we transform the bitcast to amx load/store
 /// and <256 x i32> store/load.
+///
+/// If Front End not use O0 but the Mid/Back end use O0, (e.g. "Clang -O2 -S
+/// -emit-llvm t.c" + "llc t.ll") we should make sure the amx data is volatile,
+/// because that is nessary for AMX fast register allocation. (In Fast register
+/// allocation, register will be allocated before spill/reload, so there is no
+/// additional register for amx to identify the step in spill/reload.)
+/// The VolatileTileData() will handle this case.
+/// e.g.
+/// ----------------------------------------------------------
+/// | def %td = ...                                          |
+/// | ...                                                    |
+/// | "use %td"                                              |
+/// ----------------------------------------------------------
+/// will transfer to -->
+/// ----------------------------------------------------------
+/// | def %td = ...                                          |
+/// | call void @llvm.x86.tilestored64.internal(mem, %td)    |
+/// | ...                                                    |
+/// | %td2 = call x86_amx @llvm.x86.tileloadd64.internal(mem)|
+/// | "use %td2"                                             |
+/// ----------------------------------------------------------
 //
 //===----------------------------------------------------------------------===//
 //
 #include "X86.h"
 #include "llvm/ADT/PostOrderIterator.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/IntrinsicsX86.h"
 #include "llvm/IR/PatternMatch.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Target/TargetMachine.h"
 
 using namespace llvm;
 using namespace PatternMatch;
 
 #define DEBUG_TYPE "lower-amx-type"
 
-static AllocaInst *CreateAllocaInst(IRBuilder<> &Builder, BasicBlock *BB) {
+static AllocaInst *CreateAllocaInstAtEntry(IRBuilder<> &Builder,
+                                           BasicBlock *BB) {
   Function &F = *BB->getParent();
   Module *M = BB->getModule();
   const DataLayout &DL = M->getDataLayout();
 
   Type *V256I32Ty = VectorType::get(Builder.getInt32Ty(), 256, false);
   LLVMContext &Ctx = Builder.getContext();
   auto AllocaAlignment = DL.getPrefTypeAlign(Type::getX86_AMXTy(Ctx));
   unsigned AllocaAS = DL.getAllocaAddrSpace();
   AllocaInst *AllocaRes =
       new AllocaInst(V256I32Ty, AllocaAS, "", &F.getEntryBlock().front());
   AllocaRes->setAlignment(AllocaAlignment);
   return AllocaRes;
 }
 
 static std::pair<Value *, Value *> getShape(IntrinsicInst *II, unsigned OpNo) {
   Value *Row = nullptr, *Col = nullptr;
   switch (II->getIntrinsicID()) {
   default:
     llvm_unreachable("Expect amx intrinsics");
   case Intrinsic::x86_tileloadd64_internal:
   case Intrinsic::x86_tilestored64_internal: {
     Row = II->getArgOperand(0);
     Col = II->getArgOperand(1);
     break;
   }
   // a * b + c
   // The shape depends on which operand.
   case Intrinsic::x86_tdpbssd_internal:
   case Intrinsic::x86_tdpbsud_internal:
   case Intrinsic::x86_tdpbusd_internal:
   case Intrinsic::x86_tdpbuud_internal:
   case Intrinsic::x86_tdpbf16ps_internal: {
     switch (OpNo) {
     case 3:
       Row = II->getArgOperand(0);
       Col = II->getArgOperand(1);
       break;
     case 4:
       Row = II->getArgOperand(0);
       Col = II->getArgOperand(2);
       break;
     case 5:
       Row = II->getArgOperand(2);
       Col = II->getArgOperand(1);
       break;
     }
     break;
   }
   }
 
   return std::make_pair(Row, Col);
 }
 

 // transform bitcast to <store, load> instructions.
 static bool transformBitcast(BitCastInst *Bitcast) {
   IRBuilder<> Builder(Bitcast);
   AllocaInst *AllocaAddr;
   Value *I8Ptr, *Stride;
   auto *Src = Bitcast->getOperand(0);
 
   auto Prepare = [&]() {
-    AllocaAddr = CreateAllocaInst(Builder, Bitcast->getParent());
+    AllocaAddr = CreateAllocaInstAtEntry(Builder, Bitcast->getParent());
     I8Ptr = Builder.CreateBitCast(AllocaAddr, Builder.getInt8PtrTy());
     Stride = Builder.getInt64(64);
   };
 
   if (Bitcast->getType()->isX86_AMXTy()) {
     // %2 = bitcast <256 x i32> %src to x86_amx
     // -->
     // %addr = alloca <256 x i32>, align 64
bool X86LowerAMXType::visit() {
 
   for (auto *Inst : DeadInsts)
     Inst->eraseFromParent();
 
   return C;
 }
 } // anonymous namespace
 
+static AllocaInst *CreateAllocaInst(IRBuilder<> &Builder, BasicBlock *BB) {
+  Module *M = BB->getModule();
+  Function *F = BB->getParent();
+  const DataLayout &DL = M->getDataLayout();
+  unsigned AllocaAS = DL.getAllocaAddrSpace();
+  Type *V256I32Ty = VectorType::get(Builder.getInt32Ty(), 256, false);
+  AllocaInst *AllocaRes =
+      new AllocaInst(V256I32Ty, AllocaAS, "", &F->getEntryBlock().front());
+  // AllocaRes->setAlignment(alignment);
+  return AllocaRes;
+}
+
+static Instruction *CreateTileStore(Instruction *TileDef, Value *Ptr) {
+  assert(TileDef->getType()->isX86_AMXTy() && "Not define tile!");
+  auto *II = cast<IntrinsicInst>(TileDef);
+  assert(II && "Not tile intrinsic!");
+  Value *Row = II->getOperand(0);
+  Value *Col = II->getOperand(1);
+
+  BasicBlock *BB = TileDef->getParent();
+  BasicBlock::iterator Iter = TileDef->getIterator();
+  IRBuilder<> Builder(BB, ++Iter);
+  Value *Stride = Builder.getInt64(64);
+  std::array<Value *, 5> Args = {Row, Col, Ptr, Stride, TileDef};
+
+  Instruction *TileStore =
+      Builder.CreateIntrinsic(Intrinsic::x86_tilestored64_internal, None, Args);
+  return TileStore;
+}
+
+static void ReplaceWithTileLoad(Use &U, Value *Ptr, bool IsPHI = false) {
+  Value *V = U.get();
+  assert(V->getType()->isX86_AMXTy() && "Not define tile!");
+
+  // Get tile shape.
+  IntrinsicInst *II = nullptr;
+  if (IsPHI) {
+    Value *PhiOp = dyn_cast<PHINode>(V)->getIncomingValue(0);
+    II = cast<IntrinsicInst>(PhiOp);
+  } else
+    II = cast<IntrinsicInst>(V);
+  Value *Row = II->getOperand(0);
+  Value *Col = II->getOperand(1);
+
+  Instruction *UserI = dyn_cast<Instruction>(U.getUser());
+  IRBuilder<> Builder(UserI);
+  Value *Stride = Builder.getInt64(64);
+  std::array<Value *, 4> Args = {Row, Col, Ptr, Stride};
+
+  Value *TileLoad =
+      Builder.CreateIntrinsic(Intrinsic::x86_tileloadd64_internal, None, Args);
+  UserI->replaceUsesOfWith(V, TileLoad);
+}
+
+static bool IsIncomingOfPHI(Instruction *I) {
+  for (Use &U : I->uses()) {
+    User *V = U.getUser();
+    if (isa<PHINode>(V))
+      return true;
+  }
+  return false;
+}
+
+// Let all AMX tile data become volatile data, shorten the life range
+// of each tile register before fast register allocation.
+namespace {
+class X86VolatileTileData {
+  Function &F;
+  DominatorTree *DT = nullptr;
+
+public:
+  X86VolatileTileData(Function &Func, DominatorTree *DomTree)
+      : F(Func), DT(DomTree) {}
+  Value *updatePhiIncomings(BasicBlock *DomBB, Instruction *Pos,
+                            SmallVector<Instruction *, 2> &Imcomings);
+  void replacePhiDefWithLoad(Instruction *PHI, Value *StorePtr);
+  bool VolatileTileData();
+  void VolatileTilePHI(PHINode *Inst);
+  void VolatileTileNonPHI(Instruction *I);
+};
+
+Value *X86VolatileTileData::updatePhiIncomings(
+    BasicBlock *DomBB, Instruction *Pos,
+    SmallVector<Instruction *, 2> &Imcomings) {
+  IRBuilder<> Builder(DomBB->getTerminator());
+  if (Pos)
+    Builder.SetInsertPoint(Pos);
+  AllocaInst *AllocaAddr = CreateAllocaInst(Builder, DomBB);
+  Value *I8Ptr = Builder.CreateBitCast(AllocaAddr, Builder.getInt8PtrTy());
+
+  for (auto *I : Imcomings) {
+    User *Store = CreateTileStore(I, I8Ptr);
+
+    // All its uses (except phi) should load from stored mem.
+    for (Use &U : I->uses()) {
+      User *V = U.getUser();
+      if (isa<PHINode>(V) || V == Store)
+        continue;
+      ReplaceWithTileLoad(U, I8Ptr);
+    }
+  }
+  return I8Ptr;
+}
+
+void X86VolatileTileData::replacePhiDefWithLoad(Instruction *PHI,
+                                                Value *StorePtr) {
+  for (Use &U : PHI->uses())
+    ReplaceWithTileLoad(U, StorePtr, true);
+  PHI->eraseFromParent();
+}
+
+// Smilar with VolatileTileNonPHI, this function only handle PHI Nodes
+// and their related AMX intrinsics.
+// 1) PHI Def should change to tileload.
+// 2) PHI Incoming Values should tilestored in just after their def.
+// 3) The mem of these tileload and tilestores should be same.
+// e.g.
+// ------------------------------------------------------
+// bb_dom:
+//   ...
+//   br i1 %bool.cond, label %if.else, label %if.then
+//
+// if.then:
+//   def %t0 = ...
+//   ...
+//   use %t0
+//   ...
+//   br label %if.end
+//
+// if.else:
+//   def %t1 = ...
+//   br label %if.end
+//
+// if.end:
+//   %td = phi x86_amx [ %t1, %if.else ], [ %t0, %if.then ]
+//   ...
+//   use %td
+// ------------------------------------------------------
+// -->
+// ------------------------------------------------------
+// bb_entry:
+//   %mem = alloca <256 x i32>, align 1024                  *
+//   ...
+// bb_dom:
+//   ...
+//   br i1 %bool.cond, label %if.else, label %if.then
+//
+// if.then:
+//   def %t0 = ...
+//   call void @llvm.x86.tilestored64.internal(mem, %t0)    *
+//   ...
+//   %t0` = call x86_amx @llvm.x86.tileloadd64.internal(mem)*
+//   use %t0`                                               *
+//   ...
+//   br label %if.end
+//
+// if.else:
+//   def %t1 = ...
+//   call void @llvm.x86.tilestored64.internal(mem, %t1)    *
+//   br label %if.end
+//
+// if.end:
+//   ...
+//   %td = call x86_amx @llvm.x86.tileloadd64.internal(mem) *
+//   use %td
+// ------------------------------------------------------
+void X86VolatileTileData::VolatileTilePHI(PHINode *PHI) {
+  Instruction *Pos = nullptr;
+  BasicBlock *DomBB = PHI->getParent();
+  SmallVector<Instruction *, 2> Imcomings;
+
+  for (unsigned i = 0, e = PHI->getNumIncomingValues(); i != e; ++i) {
+    Value *Op = PHI->getIncomingValue(i);
+    Instruction *I = dyn_cast<Instruction>(Op);
+    assert(I && "We shouldn't fold AMX instrution!");
+
+    BasicBlock *BB = I->getParent();
+    BasicBlock *LastDomBB = DomBB;
+    DomBB = DT->findNearestCommonDominator(LastDomBB, BB);
+
+    if (DomBB == BB)
+      Pos = I;
+    else if (DomBB != LastDomBB)
+      Pos = nullptr;
+    Imcomings.push_back(I);
+  }
+
+  Value *StorePtr = updatePhiIncomings(DomBB, Pos, Imcomings);
+  replacePhiDefWithLoad(PHI, StorePtr);
+}
+
+// Store the defined tile and load it before use.
+// All its users are not PHI.
+// e.g.
+// ------------------------------------------------------
+// def %td = ...
+// ...
+// "use %td"
+// ------------------------------------------------------
+// -->
+// ------------------------------------------------------
+// def %td = ...
+// call void @llvm.x86.tilestored64.internal(mem, %td)
+// ...
+// %td2 = call x86_amx @llvm.x86.tileloadd64.internal(mem)
+// "use %td2"
+// ------------------------------------------------------
+void X86VolatileTileData::VolatileTileNonPHI(Instruction *I) {
+  BasicBlock *BB = I->getParent();
+  IRBuilder<> Builder(I);
+
+  AllocaInst *AllocaAddr = CreateAllocaInst(Builder, BB);
+  Value *I8Ptr = Builder.CreateBitCast(AllocaAddr, Builder.getInt8PtrTy());
+  User *Store = CreateTileStore(I, I8Ptr);
+
+  // All its uses should load from stored mem.
+  for (Use &U : I->uses()) {
+    User *V = U.getUser();
+    assert(!isa<PHINode>(V) && "PHI Nodes should be excluded!");
+    if (V != Store)
+      ReplaceWithTileLoad(U, I8Ptr);
+  }
+}
+
+// Volatile Tile Model:
+// 1) All the uses of tile data comes from tileload in time.
+// 2) All the defs of tile data tilestore into mem immediately.
+// For example:
+// --------------------------------------------------------------------------
+// %t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)          key
+// %t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)
+// %t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)          amx
+// %td = tail call x86_amx @llvm.x86.tdpbssd.internal(m, n, k, t1, t2, t3)
+// call void @llvm.x86.tilestored64.internal(... td)                     area
+// --------------------------------------------------------------------------
+// 3) No terminator, call or other amx instructions in the key amx area.
+bool X86VolatileTileData::VolatileTileData() {
+  bool Changed = false;
+  for (BasicBlock &BB : F) {
+    SmallVector<Instruction *, 2> PHIInsts;
+    SmallVector<Instruction *, 8> AMXDefInsts;
+
+    for (Instruction &I : BB) {
+      if (!I.getType()->isX86_AMXTy())
+        continue;
+      if (isa<PHINode>(&I))
+        PHIInsts.push_back(&I);
+      else
+        AMXDefInsts.push_back(&I);
+    }
+
+    // First we "volatile" the non-phi related amx intrinsics.
+    for (Instruction *I : AMXDefInsts) {
+      if (IsIncomingOfPHI(I))
+        continue;
+      VolatileTileNonPHI(I);
+      Changed = true;
+    }
+
+    for (Instruction *I : PHIInsts) {
+      VolatileTilePHI(dyn_cast<PHINode>(I));
+      Changed = true;
+    }
+  }
+  return Changed;
+}
+
+} // anonymous namespace
+
 namespace {
 
 class X86LowerAMXTypeLegacyPass : public FunctionPass {
 public:
   static char ID;
 
   X86LowerAMXTypeLegacyPass() : FunctionPass(ID) {
     initializeX86LowerAMXTypeLegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnFunction(Function &F) override {
     TargetMachine *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
-    if (F.hasFnAttribute(Attribute::OptimizeNone) ||
-        TM->getOptLevel() == CodeGenOpt::None)
-      return false;
+    // if (F.hasFnAttribute(Attribute::OptimizeNone) ||
+    //     TM->getOptLevel() == CodeGenOpt::None)
+    //   return false;
+    DominatorTree *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
+
     X86LowerAMXType LAT(F);
     bool C = LAT.visit();
+
+    // Prepare for fast register allocation at O0.
+    // Todo: May better check the volatile model of AMX code, not just
+    // by checking Attribute::OptimizeNone and CodeGenOpt::None.
+    if (TM->getOptLevel() == CodeGenOpt::None) {
+      // If Front End not use O0 but the Mid/Back end use O0, (e.g.
+      // "Clang -O2 -S -emit-llvm t.c" + "llc t.ll") we should make
+      // sure the amx data is volatile, that is nessary for AMX fast
+      // register allocation.
+      if (!F.hasFnAttribute(Attribute::OptimizeNone)) {
+        X86VolatileTileData VTD(F, DT);
+        C = VTD.VolatileTileData() || C;
+      }
+    }
+
     return C;
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     AU.addRequired<TargetPassConfig>();
+    AU.addRequired<DominatorTreeWrapperPass>();
+    AU.addPreserved<DominatorTreeWrapperPass>();
   }
 };
 
 } // anonymous namespace
 
 static const char PassName[] = "Lower AMX type for load/store";
 char X86LowerAMXTypeLegacyPass::ID = 0;
 INITIALIZE_PASS_BEGIN(X86LowerAMXTypeLegacyPass, DEBUG_TYPE, PassName, false,
                       false)
 INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
 INITIALIZE_PASS_END(X86LowerAMXTypeLegacyPass, DEBUG_TYPE, PassName, false,
                     false)
 
 FunctionPass *llvm::createX86LowerAMXTypePass() {
   return new X86LowerAMXTypeLegacyPass();
 }

llvm/lib/Target/X86/X86PreAMXConfig.cpp

+//===- Target/X86/X86PreAMXConfig.cpp - ------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// Insert tilecfg for each area of key AMX intrinsic.
+/// All the key AMX intrinsic's tile operand must come from tileload. And the
+/// def tile of key AMX intrinsic must be tilestored.
+/// take tdpbssd for example:
+/// --------------------------------------------------------------------------
+/// %t1 = call x86_amx @llvm.x86.tileloadd64.internal(...)                key
+/// %t2 = call x86_amx @llvm.x86.tileloadd64.internal(...)                 |
+/// %t3 = call x86_amx @llvm.x86.tileloadd64.internal(...)                amx
+/// %td = tail call x86_amx @llvm.x86.tdpbssd.internal(t1, t2, t3)         |
+/// call void @llvm.x86.tilestored64.internal(... td)                     area
+/// --------------------------------------------------------------------------
+/// This pass will insert tilecfg before every key-amx-area, some like:
+/// --------------------------------------------------------------------------
+/// %cfgmem = alloca <16 x i32>, align 4                        * allocate mem
+/// store <16 x i32> zeroinitializer, <16 x i32>* %cfgmem       * zero init
+/// ...
+/// ... pre-config shape of %t1                                 *
+/// store volatile i8 %m, i8* %amx.tmm.0.shape.row, align 1     *
+/// store volatile i16 %k, i16* %amx.tmm.0.shape.col, align 2   * pre-config
+/// ...                                                         *
+/// ... pre-config shape of %t2                                 * shapes
+/// store volatile i8 %k, i8* %amx.tmm.1.shape.row, align 1     *
+/// store volatile i16 %n, i16* %amx.tmm.1.shape.col, align 2   *
+/// ...
+/// call void @llvm.x86.ldtilecfg.internal(i8* %cfgmem)         * tile config
+//
+//===----------------------------------------------------------------------===//
+//
+#include "X86.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
+#include "llvm/CodeGen/ValueTypes.h"
+#include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/IntrinsicsX86.h"
+#include "llvm/IR/PatternMatch.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Target/TargetMachine.h"
+
+using namespace llvm;
+using namespace PatternMatch;
+
+#define DEBUG_TYPE "pre-amx-config"
+
+static bool IsAMXIntrinsic(IntrinsicInst *II) {
+  for (Value *Operand : II->operands())
+    if (Operand->getType()->isX86_AMXTy())
+      return true;
+  return II->getType()->isX86_AMXTy();
+}
+
+static bool IsTileLoad(IntrinsicInst *II) {
+  return II->getIntrinsicID() == Intrinsic::x86_tileloadd64_internal;
+}
+
+static bool IsTileStore(IntrinsicInst *II) {
+  return II->getIntrinsicID() == Intrinsic::x86_tilestored64_internal;
+}
+
+static bool brokenVolatile(Instruction *I) {
+  // Todo: it is weak to identify a normal call here.
+  if ((isa<CallInst>(I) && !isa<IntrinsicInst>(I)) || I->isTerminator())
+    return true;
+  return false;
+}
+
+namespace {
+class X86PreAMXConfig {
+  Function &F;
+
+public:
+  X86PreAMXConfig(Function &Func) : F(Func) {}
+  bool PreTileConfig();
+  bool addTileConfig(Instruction *FirstLoad, SmallVector<Value *, 8> &Shapes);
+  bool findConfigShapes(
+      DenseMap<Instruction *, SmallVector<Value *, 8>> &PosAndShapes);
+  bool preWriteTileCfg(Value *I8Ptr, Instruction *Pos,
+                       SmallVector<Value *, 8> &Shapes);
+  BasicBlock::iterator getConfigPosEnd(BasicBlock::iterator Iter,
+                                       SmallVector<Value *, 8> &Shapes);
+  bool checkVolatileModel(SmallSet<Value *, 4> &Loads, IntrinsicInst *Store,
+                          IntrinsicInst *KeyAMX);
+};
+
+// Orderly write the shapes in tilecfg's mem. This maybe not right.
+// Because the first shape may not corresponding to the first tmm register,
+// so we need to handle at at X86FastTileConfig::materializeTileCfg()
+// after register allocation.
+// For example:
+// --------------------------------------------------------------------------
+// zeroinitialize tilecfg's mem (of ldtilecfg)
+// --------------------------------------------------------------------------
+// ... pre-config shape of %t1                                 *
+// %amx.tmm.0.shape.row = getelementptr i8, i8* %mem, i64 48   *
+// %amx.tmm.0.shape.col = getelementptr i16, i16* %mem, i64 16 *
+// store volatile i8 %m, i8* %amx.tmm.0.shape.row, align 1     *
+// store volatile i16 %k, i16* %amx.tmm.0.shape.col, align 2   * pre-config
+// ...                                                         *
+// ... pre-config shape of %t2                                 *
+// %amx.tmm.1.shape.row = getelementptr i8, i8* %mem, i64 49   *
+// %amx.tmm.1.shape.col = getelementptr i16, i16* %mem, i64 18 *
+// store volatile i8 %k, i8* %amx.tmm.1.shape.row, align 1     * shapes
+// store volatile i16 %n, i16* %amx.tmm.1.shape.col, align 2   *
+// ...                                                         *
+// ... pre-config shape of %t3                                 * of
+// %amx.tmm.2.shape.row = getelementptr i8, i8* %mem, i64 50   *
+// %amx.tmm.2.shape.col = getelementptr i16, i16* %mem, i64 20 *
+// store volatile i8 %m, i8* %amx.tmm.2.shape.row, align 1     *
+// store volatile i16 %n, i16* %amx.tmm.2.shape.col, align 2   *
+// ...                                                         * tiles
+// ... pre-config shape of %td                                 *
+// %amx.tmm.3.shape.row = getelementptr i8, i8* %mem, i64 51   *
+// %amx.tmm.3.shape.col = getelementptr i16, i16* %mem, i64 22 *
+// store volatile i8 %m, i8* %amx.tmm.3.shape.row, align 1     *
+// store volatile i16 %n, i16* %amx.tmm.3.shape.col, align 2   *
+// --------------------------------------------------------------------------
+// call void @llvm.x86.ldtilecfg.internal(i8* %mem)            * tile config
+// --------------------------------------------------------------------------
+// %t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)          key
+// %t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)
+// %t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)          amx
+// %td = tail call x86_amx @llvm.x86.tdpbssd.internal(m, n, k, t1, t2, t3)
+// call void @llvm.x86.tilestored64.internal(... td)                     area
+// --------------------------------------------------------------------------
+bool X86PreAMXConfig::preWriteTileCfg(Value *I8Ptr, Instruction *Pos,
+                                      SmallVector<Value *, 8> &Shapes) {
+  bool Write = false;
+  LLVMContext &Ctx = Pos->getParent()->getContext();
+  Type *I8Ty = Type::getInt8Ty(Ctx);
+  Type *I16Ty = Type::getInt16Ty(Ctx);
+
+  for (int i = 0, e = Shapes.size() / 2; i < e; i++) {
+    Value *RowOffset = ConstantInt::get(Type::getInt64Ty(Ctx), 48 + i);
+    Value *ColOffset = ConstantInt::get(Type::getInt64Ty(Ctx), 16 + i * 2);
+    const std::string ShapeName = "amx.tmm." + itostr(i);
+    Value *RowPos = GetElementPtrInst::Create(I8Ty, I8Ptr, RowOffset,
+                                              ShapeName + ".shape.row", Pos);
+    Value *ColPos = GetElementPtrInst::Create(I8Ty, I8Ptr, ColOffset, "", Pos);
+    ColPos = new BitCastInst(ColPos, PointerType::get(I16Ty, 0),
+                             ShapeName + ".shape.col", Pos);
+    Value *Row = Shapes[i * 2];
+    Value *Col = Shapes[i * 2 + 1];
+    Row = new TruncInst(Row, I8Ty, "", Pos);
+    new StoreInst(Row, RowPos, "", Pos);
+    new StoreInst(Col, ColPos, "", Pos);
+    Write = true;
+  }
+  return Write;
+}
+
+bool X86PreAMXConfig::addTileConfig(Instruction *FirstLoad,
+                                    SmallVector<Value *, 8> &Shapes) {
+  Module *M = F.getParent();
+  IRBuilder<> Builder(FirstLoad);
+  const DataLayout &DL = M->getDataLayout();
+  unsigned AddrSpace = DL.getAllocaAddrSpace();
+  LLVMContext &Ctx = Builder.getContext();
+  Type *V512Ty = VectorType::get(Builder.getInt32Ty(), 16, false);
+  Align Alignment = DL.getPrefTypeAlign(Type::getInt32Ty(Ctx));
+
+  AllocaInst *Addr =
+      new AllocaInst(V512Ty, AddrSpace, "", &F.getEntryBlock().front());
+  Addr->setAlignment(Alignment);
+  Value *I8Ptr = Builder.CreateBitCast(Addr, Builder.getInt8PtrTy());
+
+  std::array<Value *, 1> Args = {I8Ptr};
+  Instruction *Cfg =
+      Builder.CreateIntrinsic(Intrinsic::x86_ldtilecfg_internal, None, Args);
+
+  Value *Val0 = Constant::getNullValue(V512Ty);
+  Instruction *Init0 = new StoreInst(Val0, Addr, Cfg);
+  assert(Init0 && "Not Zero initilizate the cfg mem!");
+
+  preWriteTileCfg(I8Ptr, Cfg, Shapes);
+
+  return Cfg;
+}
+
+// Todo: We may need to handle "more than one store" case in the future.
+bool X86PreAMXConfig::checkVolatileModel(SmallSet<Value *, 4> &Loads,
+                                         IntrinsicInst *Store,
+                                         IntrinsicInst *KeyAMX) {
+  Value *ST = Store->getOperand(4);
+
+  // Only has tileload and tilestore.
+  if (!KeyAMX) {
+    if (Loads.size() != 1)
+      return false;
+    return Loads.contains(ST);
+  }
+
+  // All Loads should be operands of KeyAMX.
+  // All tile operands of KeyAMX should come from Loads.
+  for (Value *Op : KeyAMX->operands()) {
+    if (Op->getType()->isX86_AMXTy())
+      if (!Loads.erase(Op))
+        return false;
+  }
+
+  // The def of KeyAMX should be stored.
+  // Todo: is it key amx can be no def?
+  return Loads.empty() && (ST == cast<Value>(KeyAMX));
+}
+
+// Collect the shapes and skip the area of current key amx intrinsic.
+//
+// For example:
+// ...
+// --------------------------------------------------------------------------
+// %t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)  record (m,k)
+// %t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)  record (m,k)
+// %t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)  record (m,k)
+// %td = call x86_amx @llvm.x86.tdpbssd.internal(...t1, t2, t3)
+// call void @llvm.x86.tilestored64.internal(m, n,... td) <--PosEnd record (m,k)
+// --------------------------------------------------------------------------
+BasicBlock::iterator
+X86PreAMXConfig::getConfigPosEnd(BasicBlock::iterator Iter,
+                                 SmallVector<Value *, 8> &Shapes) {
+  IntrinsicInst *KeyAMX = nullptr;
+  BasicBlock *BB = Iter->getParent();
+  SmallSet<Value *, 4> Loads;
+
+  assert(Shapes.empty() && "Shapes should be clean.");
+  for (auto I = Iter, E = BB->end(); I != E; ++I) {
+    assert(!brokenVolatile(&*I) && "Not reach tile store!");
+    IntrinsicInst *II = dyn_cast<IntrinsicInst>(&*I);
+    if (!II || !IsAMXIntrinsic(II))
+      continue;
+
+    if (IsTileLoad(II)) {
+      Loads.insert(II);
+      Shapes.push_back(II->getOperand(0));
+      Shapes.push_back(II->getOperand(1));
+    } else if (IsTileStore(II)) {
+      bool Res = checkVolatileModel(Loads, II, KeyAMX);
+      assert(Res && "Not Volatile AMX Model!");
+      // If only has tileload before, we have already got tilestore's shape.
+      if (KeyAMX) {
+        Shapes.push_back(II->getOperand(0));
+        Shapes.push_back(II->getOperand(1));
+      }
+      return I;
+    } else {
+      assert(!KeyAMX && "More than one non-load/store amx intrinsic!");
+      KeyAMX = II;
+    }
+  }
+  // TODO: May need to handle non-tile def key AMX intrinsic in the future.
+  llvm_unreachable("Without store AMX data!");
+  return BB->end();
+}
+
+// Record a key amx area's shapes with its position.
+// Use the first tileload as its position.
+// For example:
+// ...
+// --------------------------------------------------------------------------
+// %t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)   <--  pos
+// %t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)        /
+// %t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)     shapes:
+// %td = call x86_amx @llvm.x86.tdpbssd.internal(...t1, t2, t3)    (m,k)(k,n)
+// call void @llvm.x86.tilestored64.internal(m, n,... td)          (m,n)(m,n)
+// --------------------------------------------------------------------------
+bool X86PreAMXConfig::findConfigShapes(
+    DenseMap<Instruction *, SmallVector<Value *, 8>> &PosAndShapes) {
+  bool Find = false;
+  for (BasicBlock &BB : F) {
+    for (BasicBlock::iterator I = BB.begin(), E = BB.end(); I != E; ++I) {
+      IntrinsicInst *II = dyn_cast<IntrinsicInst>(&*I);
+      if (!II)
+        continue;
+      if (!IsAMXIntrinsic(II))
+        continue;
+      assert(II->getIntrinsicID() == Intrinsic::x86_tileloadd64_internal &&
+             "Not volatile model for AMX at O0!");
+
+      I = getConfigPosEnd(I, PosAndShapes[&*I]);
+      Find = true;
+    }
+  }
+  return Find;
+}
+
+// Insert ldtilecfg and preconfig the shapes for each area of key AMX intrinsic.
+// e.g. (key amx = tdpbssd)
+// --------------------------------------------------------------------------
+// %cfgmem = alloca <16 x i32>, align 4                        * allocate mem
+// store <16 x i32> zeroinitializer, <16 x i32>* %cfgmem       * zero init
+// ...
+// ... pre-config shape of %t1                                 *
+// store volatile i8 %m, i8* %amx.tmm.0.shape.row, align 1     *
+// store volatile i16 %k, i16* %amx.tmm.0.shape.col, align 2   * pre-config
+// ...                                                         *
+// ... pre-config shape of %t2                                 *
+// store volatile i8 %k, i8* %amx.tmm.1.shape.row, align 1     * shapes
+// store volatile i16 %n, i16* %amx.tmm.1.shape.col, align 2   *
+// ...                                                         *
+// ... pre-config shape of %t3                                 * of
+// store volatile i8 %m, i8* %amx.tmm.2.shape.row, align 1     *
+// store volatile i16 %n, i16* %amx.tmm.2.shape.col, align 2   *
+// ...                                                         * tiles
+// ... pre-config shape of %td                                 *
+// store volatile i8 %m, i8* %amx.tmm.3.shape.row, align 1     *
+// store volatile i16 %n, i16* %amx.tmm.3.shape.col, align 2   *
+//
+// call void @llvm.x86.ldtilecfg.internal(i8* %cfgmem)         * pre-config
+// --------------------------------------------------------------------------
+// %t1 = call x86_amx @llvm.x86.tileloadd64.internal(m, k, ...)          key
+// %t2 = call x86_amx @llvm.x86.tileloadd64.internal(k, n, ...)
+// %t3 = call x86_amx @llvm.x86.tileloadd64.internal(m, n, ...)          amx
+// %td = tail call x86_amx @llvm.x86.tdpbssd.internal(m, n, k, t1, t2, t3)
+// call void @llvm.x86.tilestored64.internal(... td)                     area
+// --------------------------------------------------------------------------
+bool X86PreAMXConfig::PreTileConfig() {
+  DenseMap<Instruction *, SmallVector<Value *, 8>> PosAndShapes;
+  bool NeedCfg = findConfigShapes(PosAndShapes);
+  if (!NeedCfg)
+    return false;
+  for (auto &IPAndShapes : PosAndShapes) {
+    addTileConfig(IPAndShapes.first, IPAndShapes.second);
+  }
+
+  return true;
+}
+} // anonymous namespace
+
+namespace {
+
+class X86PreAMXConfigPass : public FunctionPass {
+public:
+  static char ID;
+
+  X86PreAMXConfigPass() : FunctionPass(ID) {
+    initializeX86PreAMXConfigPassPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnFunction(Function &F) override {
+    TargetMachine *TM = &getAnalysis<TargetPassConfig>().getTM<TargetMachine>();
+    bool C = false;
+
+    // Prepare for fast register allocation at O0.
+    if (TM->getOptLevel() == CodeGenOpt::None) {
+
+      // We pre-config each key AMX intrinsic at O0.
+      // In theory, one tile config can cover several AMX intrinsics, but
+      // it is very diffcult to classify the tile shapes at O0. So here we
+      // let thing be easy, pre-config every key AMX intrinsic.
+      X86PreAMXConfig PCFG(F);
+      C = PCFG.PreTileConfig();
+    }
+
+    return C;
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    AU.addRequired<TargetPassConfig>();
+  }
+};
+
+} // anonymous namespace
+
+static const char PassName[] = "Pre AMX Tile Config";
+char X86PreAMXConfigPass::ID = 0;
+INITIALIZE_PASS_BEGIN(X86PreAMXConfigPass, DEBUG_TYPE, PassName, false, false)
+INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)
+INITIALIZE_PASS_END(X86PreAMXConfigPass, DEBUG_TYPE, PassName, false, false)
+
+FunctionPass *llvm::createX86PreAMXConfigPass() {
+  return new X86PreAMXConfigPass();
+}

llvm/lib/Target/X86/X86TargetMachine.cpp

 extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeX86Target() {
   // Register the target.
   RegisterTargetMachine<X86TargetMachine> X(getTheX86_32Target());
   RegisterTargetMachine<X86TargetMachine> Y(getTheX86_64Target());
 
   PassRegistry &PR = *PassRegistry::getPassRegistry();
   initializeX86LowerAMXIntrinsicsLegacyPassPass(PR);
   initializeX86LowerAMXTypeLegacyPassPass(PR);
+  initializeX86PreAMXConfigPassPass(PR);
   initializeGlobalISel(PR);
   initializeWinEHStatePassPass(PR);
   initializeFixupBWInstPassPass(PR);
   initializeEvexToVexInstPassPass(PR);
   initializeFixupLEAPassPass(PR);
   initializeFPSPass(PR);
   initializeX86FixupSetCCPassPass(PR);
   initializeX86CallFrameOptimizationPass(PR);
   initializeX86CmovConverterPassPass(PR);
   initializeX86TileConfigPass(PR);
+  initializeX86FastTileConfigPass(PR);
   initializeX86LowerTileCopyPass(PR);
   initializeX86ExpandPseudoPass(PR);
   initializeX86ExecutionDomainFixPass(PR);
   initializeX86DomainReassignmentPass(PR);
   initializeX86AvoidSFBPassPass(PR);
   initializeX86AvoidTrailingCallPassPass(PR);
   initializeX86SpeculativeLoadHardeningPassPass(PR);
   initializeX86SpeculativeExecutionSideEffectSuppressionPass(PR);

 void X86PassConfig::addIRPasses() {
   addPass(createAtomicExpandPass());
 
   // We add both pass anyway and when these two passes run, we skip the pass
   // based on the option level and option attribute.
   addPass(createX86LowerAMXIntrinsicsPass());
   addPass(createX86LowerAMXTypePass());
 
+  if (TM->getOptLevel() == CodeGenOpt::None)
+    addPass(createX86PreAMXConfigPass());
+
   TargetPassConfig::addIRPasses();
 
   if (TM->getOptLevel() != CodeGenOpt::None) {
     addPass(createInterleavedAccessPass());
     addPass(createX86PartialReductionPass());
   }
 
   // Add passes that handle indirect branch removal and insertion of a retpoline
if (TT.isOSWindows()) {
     addPass(createCFGuardLongjmpPass());
     // Identify valid eh continuation targets for Windows EHCont Guard.
     addPass(createEHContGuardCatchretPass());
   }
   addPass(createX86LoadValueInjectionRetHardeningPass());
 }
 
 bool X86PassConfig::addPreRewrite() {
+  //Fixme: should fast reg allocation.
+  if (getOptLevel() == CodeGenOpt::None)
+    addPass(createX86FastTileConfigPass());
+  else
     addPass(createX86TileConfigPass());
   return true;
 }
 
 std::unique_ptr<CSEConfigBase> X86PassConfig::getCSEConfig() const {
   return getStandardCSEConfigForOpt(TM->getOptLevel());
 }

llvm/test/CodeGen/X86/AMX/amx-configO0toO0.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -O0 -mtriple=x86_64-unknown-unknown -mattr=+amx-int8 -mattr=+avx512f | FileCheck %s --check-prefix=AMX_O0
+source_filename = "amx_api.c"
+
+%struct.__tile1024i_str = type <{ i16, i16, [60 x i8], <256 x i32> }>
+
+@buf = dso_local global [1024 x i8] zeroinitializer, align 16
+@buf2 = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @test_api(i32 %cond, i16 signext %row, i16 signext %col) #0 {
+; AMX_O0-LABEL: test_api:
+; AMX_O0:       # %bb.0: # %entry
+; AMX_O0-NEXT:    pushq %rbp
+; AMX_O0-NEXT:    .cfi_def_cfa_offset 16
+; AMX_O0-NEXT:    .cfi_offset %rbp, -16
+; AMX_O0-NEXT:    movq %rsp, %rbp
+; AMX_O0-NEXT:    .cfi_def_cfa_register %rbp
+; AMX_O0-NEXT:    andq $-1024, %rsp # imm = 0xFC00
+; AMX_O0-NEXT:    subq $25600, %rsp # imm = 0x6400
+; AMX_O0-NEXT:    movw %dx, %ax
+; AMX_O0-NEXT:    movw %si, %cx
+; AMX_O0-NEXT:    movl %edi, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    xorl %esi, %esi
+; AMX_O0-NEXT:    movl $1088, %edx # imm = 0x440
+; AMX_O0-NEXT:    callq memset@PLT
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    xorl %esi, %esi
+; AMX_O0-NEXT:    movl $1088, %edx # imm = 0x440
+; AMX_O0-NEXT:    callq memset@PLT
+; AMX_O0-NEXT:    movw $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    xorl %esi, %esi
+; AMX_O0-NEXT:    movl $1088, %edx # imm = 0x440
+; AMX_O0-NEXT:    callq memset@PLT
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    cmpl $0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    je .LBB0_2
+; AMX_O0-NEXT:  # %bb.1: # %if.then
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $buf, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw (%rax), %si
+; AMX_O0-NEXT:    movw 2(%rax), %dx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw %si, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %dil
+; AMX_O0-NEXT:    movb %dil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    ldtilecfg (%rdi)
+; AMX_O0-NEXT:    tileloadd (%rdx,%rsi), %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    addq $64, %rdx
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $buf, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw (%rax), %di
+; AMX_O0-NEXT:    movw 2(%rax), %dx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw %di, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %r8b
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r8
+; AMX_O0-NEXT:    ldtilecfg (%r8)
+; AMX_O0-NEXT:    tileloadd (%rdx,%rdi), %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    addq $64, %rdx
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $buf, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw (%rax), %si
+; AMX_O0-NEXT:    movw 2(%rax), %dx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw %si, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %r8b
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    ldtilecfg (%rdi)
+; AMX_O0-NEXT:    tileloadd (%rdx,%rsi), %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    addq $64, %rdx
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    jmp .LBB0_3
+; AMX_O0-NEXT:  .LBB0_2: # %if.else
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $buf2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw (%rax), %si
+; AMX_O0-NEXT:    movw 2(%rax), %dx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw %si, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %dil
+; AMX_O0-NEXT:    movb %dil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    ldtilecfg (%rdi)
+; AMX_O0-NEXT:    tileloadd (%rdx,%rsi), %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    addq $64, %rdx
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $buf2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw (%rax), %di
+; AMX_O0-NEXT:    movw 2(%rax), %dx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw %di, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %r8b
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r8
+; AMX_O0-NEXT:    ldtilecfg (%r8)
+; AMX_O0-NEXT:    tileloadd (%rdx,%rdi), %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    addq $64, %rdx
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $buf2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw (%rax), %si
+; AMX_O0-NEXT:    movw 2(%rax), %dx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rcx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movw %si, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %r8b
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    ldtilecfg (%rdi)
+; AMX_O0-NEXT:    tileloadd (%rdx,%rsi), %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    addq $64, %rdx
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:  .LBB0_3: # %if.end
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    movl $1088, %edx # imm = 0x440
+; AMX_O0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    vzeroupper
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    vmovdqa64 64(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 128(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 192(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 256(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 320(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 384(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 448(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 512(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 576(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 640(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 704(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 768(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 832(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 896(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 960(%rax), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 1024(%rax), %zmm0
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm16
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm17
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm18
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm19
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm20
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm21
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm22
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm23
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm24
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm25
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm26
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm27
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm28
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm29
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm30
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm31
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm1
+; AMX_O0-NEXT:    vmovaps %zmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm1
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm2
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm3
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm4
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm5
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm6
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm7
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm8
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm9
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm10
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm11
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm12
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm13
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm14
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm15
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm31, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm30, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm29, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm28, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm27, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm26, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm25, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm24, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm23, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm22, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm21, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm20, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm19, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm18, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm17, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm16, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm15, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm14, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm13, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm12, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm11, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm10, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm9, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm7, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm6, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm5, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm4, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm3, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm1, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    movl $1024, %edx # imm = 0x400
+; AMX_O0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    vzeroupper
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %di # 2-byte Reload
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %cx # 2-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    # kill: def $r8 killed $rax
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %ax # 2-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm16
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm17
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm18
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm19
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm20
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm21
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm22
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm23
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm24
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm25
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm26
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm27
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm28
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm29
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm30
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm31
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm1
+; AMX_O0-NEXT:    vmovaps %zmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm1
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm2
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm3
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm4
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm5
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm6
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm7
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm8
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm9
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm10
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm11
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm12
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm13
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm14
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm15
+; AMX_O0-NEXT:    movw %di, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %ax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 %zmm31, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm30, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm29, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm28, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm27, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm26, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm25, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm24, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm23, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm22, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm21, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm20, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm19, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm18, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm17, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm16, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm15, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm14, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm13, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm12, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm11, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm10, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm9, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm7, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm6, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm5, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm4, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm3, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm1, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %di
+; AMX_O0-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AMX_O0-NEXT:    vmovaps %zmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %r8b
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %di, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %dil, %r9b
+; AMX_O0-NEXT:    movb %r9b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r8
+; AMX_O0-NEXT:    ldtilecfg (%r8)
+; AMX_O0-NEXT:    movl $64, %r8d
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r9
+; AMX_O0-NEXT:    tileloadd (%r9,%r8), %tmm0
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r9
+; AMX_O0-NEXT:    tileloadd (%r9,%r8), %tmm1
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r9
+; AMX_O0-NEXT:    tileloadd (%r9,%r8), %tmm2
+; AMX_O0-NEXT:    tdpbssd %tmm2, %tmm1, %tmm0
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    addq $64, %rdi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdi,%r8)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    vzeroupper
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    movq $buf, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq $32, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm0
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm1
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm2
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm3
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm4
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm5
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm6
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm7
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm8
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm9
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm10
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm11
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm12
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm13
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm14
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm15
+; AMX_O0-NEXT:    vmovdqa64 %zmm15, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm14, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm13, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm12, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm11, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm10, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm9, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm7, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm6, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm5, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm4, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm3, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm1, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    vzeroupper
+; AMX_O0-NEXT:    callq memcpy@PLT
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %si # 2-byte Reload
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %dx # 2-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; AMX_O0-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm0 # 64-byte Reload
+; AMX_O0-NEXT:    # kill: def $rdi killed $rax
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm1
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm2
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm3
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm4
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm5
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm6
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm7
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm8
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm9
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm10
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm11
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm12
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm13
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm14
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm15
+; AMX_O0-NEXT:    vmovdqa64 {{[0-9]+}}(%rsp), %zmm16
+; AMX_O0-NEXT:    movw %si, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm16, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm15, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm14, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm13, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm12, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm11, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm10, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm9, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm7, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm6, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm5, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm4, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm3, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm2, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    vmovdqa64 %zmm1, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %ax
+; AMX_O0-NEXT:    movw {{[0-9]+}}(%rsp), %cx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    movq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdi
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r8
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %r9b
+; AMX_O0-NEXT:    movb %r9b, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    ldtilecfg (%r8)
+; AMX_O0-NEXT:    movl $64, %r8d
+; AMX_O0-NEXT:    tileloadd (%rdi,%r8), %tmm0
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    movq %rbp, %rsp
+; AMX_O0-NEXT:    popq %rbp
+; AMX_O0-NEXT:    .cfi_def_cfa %rsp, 8
+; AMX_O0-NEXT:    tilerelease
+; AMX_O0-NEXT:    vzeroupper
+; AMX_O0-NEXT:    retq
+entry:
+  %m.addr.i85 = alloca i16, align 2
+  %n.addr.i86 = alloca i16, align 2
+  %base.addr.i87 = alloca i8*, align 8
+  %stride.addr.i88 = alloca i64, align 8
+  %tile.addr.i = alloca <256 x i32>, align 64
+  %indirect-arg-temp.i5284 = alloca <256 x i32>, align 1024
+  %m.addr.i81 = alloca i16, align 2
+  %n.addr.i82 = alloca i16, align 2
+  %k.addr.i = alloca i16, align 2
+  %dst.addr.i83 = alloca <256 x i32>, align 64
+  %src1.addr.i = alloca <256 x i32>, align 64
+  %src2.addr.i = alloca <256 x i32>, align 64
+  %indirect-arg-temp5.i80 = alloca <256 x i32>, align 1024
+  %indirect-arg-temp4.i79 = alloca <256 x i32>, align 1024
+  %indirect-arg-temp.i78 = alloca <256 x i32>, align 1024
+  %m.addr.i74 = alloca i16, align 2
+  %n.addr.i75 = alloca i16, align 2
+  %base.addr.i76 = alloca i8*, align 8
+  %stride.addr.i77 = alloca i64, align 8
+  %m.addr.i70 = alloca i16, align 2
+  %n.addr.i71 = alloca i16, align 2
+  %base.addr.i72 = alloca i8*, align 8
+  %stride.addr.i73 = alloca i64, align 8
+  %m.addr.i66 = alloca i16, align 2
+  %n.addr.i67 = alloca i16, align 2
+  %base.addr.i68 = alloca i8*, align 8
+  %stride.addr.i69 = alloca i64, align 8
+  %m.addr.i62 = alloca i16, align 2
+  %n.addr.i63 = alloca i16, align 2
+  %base.addr.i64 = alloca i8*, align 8
+  %stride.addr.i65 = alloca i64, align 8
+  %m.addr.i58 = alloca i16, align 2
+  %n.addr.i59 = alloca i16, align 2
+  %base.addr.i60 = alloca i8*, align 8
+  %stride.addr.i61 = alloca i64, align 8
+  %m.addr.i = alloca i16, align 2
+  %n.addr.i = alloca i16, align 2
+  %base.addr.i56 = alloca i8*, align 8
+  %stride.addr.i57 = alloca i64, align 8
+  %base.addr.i50 = alloca i8*, align 8
+  %stride.addr.i51 = alloca i64, align 8
+  %indirect-arg-temp.i52 = alloca <256 x i32>, align 1024
+  %c49 = alloca %struct.__tile1024i_str, align 64
+  %dst.addr.i44 = alloca %struct.__tile1024i_str*, align 8
+  %indirect-arg-temp.i = alloca <256 x i32>, align 1024
+  %indirect-arg-temp4.i = alloca <256 x i32>, align 1024
+  %indirect-arg-temp5.i = alloca <256 x i32>, align 1024
+  %b43 = alloca %struct.__tile1024i_str, align 64
+  %a42 = alloca %struct.__tile1024i_str, align 64
+  %dst.addr.i35 = alloca %struct.__tile1024i_str*, align 8
+  %base.addr.i36 = alloca i8*, align 8
+  %stride.addr.i37 = alloca i64, align 8
+  %dst.addr.i28 = alloca %struct.__tile1024i_str*, align 8
+  %base.addr.i29 = alloca i8*, align 8
+  %stride.addr.i30 = alloca i64, align 8
+  %dst.addr.i21 = alloca %struct.__tile1024i_str*, align 8
+  %base.addr.i22 = alloca i8*, align 8
+  %stride.addr.i23 = alloca i64, align 8
+  %dst.addr.i14 = alloca %struct.__tile1024i_str*, align 8
+  %base.addr.i15 = alloca i8*, align 8
+  %stride.addr.i16 = alloca i64, align 8
+  %dst.addr.i7 = alloca %struct.__tile1024i_str*, align 8
+  %base.addr.i8 = alloca i8*, align 8
+  %stride.addr.i9 = alloca i64, align 8
+  %dst.addr.i = alloca %struct.__tile1024i_str*, align 8
+  %base.addr.i = alloca i8*, align 8
+  %stride.addr.i = alloca i64, align 8
+  %cond.addr = alloca i32, align 4
+  %row.addr = alloca i16, align 2
+  %col.addr = alloca i16, align 2
+  %a = alloca %struct.__tile1024i_str, align 64
+  %b = alloca %struct.__tile1024i_str, align 64
+  %c = alloca %struct.__tile1024i_str, align 64
+  store i32 %cond, i32* %cond.addr, align 4
+  store i16 %row, i16* %row.addr, align 2
+  store i16 %col, i16* %col.addr, align 2
+  %0 = bitcast %struct.__tile1024i_str* %a to i8*
+  call void @llvm.memset.p0i8.i64(i8* align 64 %0, i8 0, i64 1088, i1 false)
+  %row1 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %a, i32 0, i32 0
+  %1 = load i16, i16* %row.addr, align 2
+  store i16 %1, i16* %row1, align 64
+  %col2 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %a, i32 0, i32 1
+  store i16 8, i16* %col2, align 2
+  %2 = bitcast %struct.__tile1024i_str* %b to i8*
+  call void @llvm.memset.p0i8.i64(i8* align 64 %2, i8 0, i64 1088, i1 false)
+  %row3 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %b, i32 0, i32 0
+  store i16 8, i16* %row3, align 64
+  %col4 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %b, i32 0, i32 1
+  %3 = load i16, i16* %col.addr, align 2
+  store i16 %3, i16* %col4, align 2
+  %4 = bitcast %struct.__tile1024i_str* %c to i8*
+  call void @llvm.memset.p0i8.i64(i8* align 64 %4, i8 0, i64 1088, i1 false)
+  %row5 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %c, i32 0, i32 0
+  %5 = load i16, i16* %row.addr, align 2
+  store i16 %5, i16* %row5, align 64
+  %col6 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %c, i32 0, i32 1
+  %6 = load i16, i16* %col.addr, align 2
+  store i16 %6, i16* %col6, align 2
+  %7 = load i32, i32* %cond.addr, align 4
+  %tobool = icmp ne i32 %7, 0
+  br i1 %tobool, label %if.then, label %if.else
+
+if.then:                                          ; preds = %entry
+  store %struct.__tile1024i_str* %a, %struct.__tile1024i_str** %dst.addr.i35, align 8
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i8** %base.addr.i36, align 8
+  store i64 32, i64* %stride.addr.i37, align 8
+  %8 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i35, align 8
+  %row.i38 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %8, i32 0, i32 0
+  %9 = load i16, i16* %row.i38, align 64
+  %10 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i35, align 8
+  %col.i39 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %10, i32 0, i32 1
+  %11 = load i16, i16* %col.i39, align 2
+  %12 = load i8*, i8** %base.addr.i36, align 8
+  %13 = load i64, i64* %stride.addr.i37, align 8
+  store i16 %9, i16* %m.addr.i, align 2
+  store i16 %11, i16* %n.addr.i, align 2
+  store i8* %12, i8** %base.addr.i56, align 8
+  store i64 %13, i64* %stride.addr.i57, align 8
+  %14 = load i16, i16* %m.addr.i, align 2
+  %15 = load i16, i16* %n.addr.i, align 2
+  %16 = load i8*, i8** %base.addr.i56, align 8
+  %17 = load i64, i64* %stride.addr.i57, align 8
+  %18 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %14, i16 %15, i8* %16, i64 %17) #2
+  %19 = bitcast x86_amx %18 to <256 x i32>
+  %20 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i35, align 8
+  %tile.i41 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %20, i32 0, i32 3
+  store <256 x i32> %19, <256 x i32>* %tile.i41, align 64
+  store %struct.__tile1024i_str* %b, %struct.__tile1024i_str** %dst.addr.i28, align 8
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i8** %base.addr.i29, align 8
+  store i64 32, i64* %stride.addr.i30, align 8
+  %21 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i28, align 8
+  %row.i31 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %21, i32 0, i32 0
+  %22 = load i16, i16* %row.i31, align 64
+  %23 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i28, align 8
+  %col.i32 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %23, i32 0, i32 1
+  %24 = load i16, i16* %col.i32, align 2
+  %25 = load i8*, i8** %base.addr.i29, align 8
+  %26 = load i64, i64* %stride.addr.i30, align 8
+  store i16 %22, i16* %m.addr.i58, align 2
+  store i16 %24, i16* %n.addr.i59, align 2
+  store i8* %25, i8** %base.addr.i60, align 8
+  store i64 %26, i64* %stride.addr.i61, align 8
+  %27 = load i16, i16* %m.addr.i58, align 2
+  %28 = load i16, i16* %n.addr.i59, align 2
+  %29 = load i8*, i8** %base.addr.i60, align 8
+  %30 = load i64, i64* %stride.addr.i61, align 8
+  %31 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %27, i16 %28, i8* %29, i64 %30) #2
+  %32 = bitcast x86_amx %31 to <256 x i32>
+  %33 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i28, align 8
+  %tile.i34 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %33, i32 0, i32 3
+  store <256 x i32> %32, <256 x i32>* %tile.i34, align 64
+  store %struct.__tile1024i_str* %c, %struct.__tile1024i_str** %dst.addr.i21, align 8
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i8** %base.addr.i22, align 8
+  store i64 32, i64* %stride.addr.i23, align 8
+  %34 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i21, align 8
+  %row.i24 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %34, i32 0, i32 0
+  %35 = load i16, i16* %row.i24, align 64
+  %36 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i21, align 8
+  %col.i25 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %36, i32 0, i32 1
+  %37 = load i16, i16* %col.i25, align 2
+  %38 = load i8*, i8** %base.addr.i22, align 8
+  %39 = load i64, i64* %stride.addr.i23, align 8
+  store i16 %35, i16* %m.addr.i62, align 2
+  store i16 %37, i16* %n.addr.i63, align 2
+  store i8* %38, i8** %base.addr.i64, align 8
+  store i64 %39, i64* %stride.addr.i65, align 8
+  %40 = load i16, i16* %m.addr.i62, align 2
+  %41 = load i16, i16* %n.addr.i63, align 2
+  %42 = load i8*, i8** %base.addr.i64, align 8
+  %43 = load i64, i64* %stride.addr.i65, align 8
+  %44 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %40, i16 %41, i8* %42, i64 %43) #2
+  %45 = bitcast x86_amx %44 to <256 x i32>
+  %46 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i21, align 8
+  %tile.i27 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %46, i32 0, i32 3
+  store <256 x i32> %45, <256 x i32>* %tile.i27, align 64
+  br label %if.end
+
+if.else:                                          ; preds = %entry
+  store %struct.__tile1024i_str* %a, %struct.__tile1024i_str** %dst.addr.i14, align 8
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i8** %base.addr.i15, align 8
+  store i64 32, i64* %stride.addr.i16, align 8
+  %47 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i14, align 8
+  %row.i17 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %47, i32 0, i32 0
+  %48 = load i16, i16* %row.i17, align 64
+  %49 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i14, align 8
+  %col.i18 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %49, i32 0, i32 1
+  %50 = load i16, i16* %col.i18, align 2
+  %51 = load i8*, i8** %base.addr.i15, align 8
+  %52 = load i64, i64* %stride.addr.i16, align 8
+  store i16 %48, i16* %m.addr.i66, align 2
+  store i16 %50, i16* %n.addr.i67, align 2
+  store i8* %51, i8** %base.addr.i68, align 8
+  store i64 %52, i64* %stride.addr.i69, align 8
+  %53 = load i16, i16* %m.addr.i66, align 2
+  %54 = load i16, i16* %n.addr.i67, align 2
+  %55 = load i8*, i8** %base.addr.i68, align 8
+  %56 = load i64, i64* %stride.addr.i69, align 8
+  %57 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %53, i16 %54, i8* %55, i64 %56) #2
+  %58 = bitcast x86_amx %57 to <256 x i32>
+  %59 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i14, align 8
+  %tile.i20 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %59, i32 0, i32 3
+  store <256 x i32> %58, <256 x i32>* %tile.i20, align 64
+  store %struct.__tile1024i_str* %b, %struct.__tile1024i_str** %dst.addr.i7, align 8
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i8** %base.addr.i8, align 8
+  store i64 32, i64* %stride.addr.i9, align 8
+  %60 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i7, align 8
+  %row.i10 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %60, i32 0, i32 0
+  %61 = load i16, i16* %row.i10, align 64
+  %62 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i7, align 8
+  %col.i11 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %62, i32 0, i32 1
+  %63 = load i16, i16* %col.i11, align 2
+  %64 = load i8*, i8** %base.addr.i8, align 8
+  %65 = load i64, i64* %stride.addr.i9, align 8
+  store i16 %61, i16* %m.addr.i70, align 2
+  store i16 %63, i16* %n.addr.i71, align 2
+  store i8* %64, i8** %base.addr.i72, align 8
+  store i64 %65, i64* %stride.addr.i73, align 8
+  %66 = load i16, i16* %m.addr.i70, align 2
+  %67 = load i16, i16* %n.addr.i71, align 2
+  %68 = load i8*, i8** %base.addr.i72, align 8
+  %69 = load i64, i64* %stride.addr.i73, align 8
+  %70 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %66, i16 %67, i8* %68, i64 %69) #2
+  %71 = bitcast x86_amx %70 to <256 x i32>
+  %72 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i7, align 8
+  %tile.i13 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %72, i32 0, i32 3
+  store <256 x i32> %71, <256 x i32>* %tile.i13, align 64
+  store %struct.__tile1024i_str* %c, %struct.__tile1024i_str** %dst.addr.i, align 8
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i8** %base.addr.i, align 8
+  store i64 32, i64* %stride.addr.i, align 8
+  %73 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i, align 8
+  %row.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %73, i32 0, i32 0
+  %74 = load i16, i16* %row.i, align 64
+  %75 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i, align 8
+  %col.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %75, i32 0, i32 1
+  %76 = load i16, i16* %col.i, align 2
+  %77 = load i8*, i8** %base.addr.i, align 8
+  %78 = load i64, i64* %stride.addr.i, align 8
+  store i16 %74, i16* %m.addr.i74, align 2
+  store i16 %76, i16* %n.addr.i75, align 2
+  store i8* %77, i8** %base.addr.i76, align 8
+  store i64 %78, i64* %stride.addr.i77, align 8
+  %79 = load i16, i16* %m.addr.i74, align 2
+  %80 = load i16, i16* %n.addr.i75, align 2
+  %81 = load i8*, i8** %base.addr.i76, align 8
+  %82 = load i64, i64* %stride.addr.i77, align 8
+  %83 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %79, i16 %80, i8* %81, i64 %82) #2
+  %84 = bitcast x86_amx %83 to <256 x i32>
+  %85 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i, align 8
+  %tile.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %85, i32 0, i32 3
+  store <256 x i32> %84, <256 x i32>* %tile.i, align 64
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %if.then
+  %86 = bitcast %struct.__tile1024i_str* %b43 to i8*
+  %87 = bitcast %struct.__tile1024i_str* %b to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %86, i8* align 1 %87, i64 1088, i1 false) #2
+  %88 = bitcast %struct.__tile1024i_str* %a42 to i8*
+  %89 = bitcast %struct.__tile1024i_str* %a to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %88, i8* align 1 %89, i64 1088, i1 false) #2
+  store %struct.__tile1024i_str* %c, %struct.__tile1024i_str** %dst.addr.i44, align 8
+  %row.i45 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %a42, i32 0, i32 0
+  %90 = load i16, i16* %row.i45, align 64
+  %col.i46 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %b43, i32 0, i32 1
+  %91 = load i16, i16* %col.i46, align 2
+  %col1.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %a42, i32 0, i32 1
+  %92 = load i16, i16* %col1.i, align 2
+  %93 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i44, align 8
+  %tile.i47 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %93, i32 0, i32 3
+  %94 = load <256 x i32>, <256 x i32>* %tile.i47, align 64
+  %tile2.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %a42, i32 0, i32 3
+  %95 = load <256 x i32>, <256 x i32>* %tile2.i, align 64
+  %tile3.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %b43, i32 0, i32 3
+  %96 = load <256 x i32>, <256 x i32>* %tile3.i, align 64
+  store <256 x i32> %94, <256 x i32>* %indirect-arg-temp.i, align 1024
+  store <256 x i32> %95, <256 x i32>* %indirect-arg-temp4.i, align 1024
+  store <256 x i32> %96, <256 x i32>* %indirect-arg-temp5.i, align 1024
+  %97 = bitcast <256 x i32>* %indirect-arg-temp5.i80 to i8*
+  %98 = bitcast <256 x i32>* %indirect-arg-temp5.i to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %97, i8* align 1 %98, i64 1024, i1 false) #2
+  %99 = bitcast <256 x i32>* %indirect-arg-temp4.i79 to i8*
+  %100 = bitcast <256 x i32>* %indirect-arg-temp4.i to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %99, i8* align 1 %100, i64 1024, i1 false) #2
+  %101 = bitcast <256 x i32>* %indirect-arg-temp.i78 to i8*
+  %102 = bitcast <256 x i32>* %indirect-arg-temp.i to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %101, i8* align 1 %102, i64 1024, i1 false) #2
+  %dst.i = load <256 x i32>, <256 x i32>* %indirect-arg-temp.i78, align 1024
+  %src1.i = load <256 x i32>, <256 x i32>* %indirect-arg-temp4.i79, align 1024
+  %src2.i = load <256 x i32>, <256 x i32>* %indirect-arg-temp5.i80, align 1024
+  store i16 %90, i16* %m.addr.i81, align 2
+  store i16 %91, i16* %n.addr.i82, align 2
+  store i16 %92, i16* %k.addr.i, align 2
+  store <256 x i32> %dst.i, <256 x i32>* %dst.addr.i83, align 64
+  store <256 x i32> %src1.i, <256 x i32>* %src1.addr.i, align 64
+  store <256 x i32> %src2.i, <256 x i32>* %src2.addr.i, align 64
+  %103 = load i16, i16* %m.addr.i81, align 2
+  %104 = load i16, i16* %n.addr.i82, align 2
+  %105 = load i16, i16* %k.addr.i, align 2
+  %106 = load <256 x i32>, <256 x i32>* %dst.addr.i83, align 64
+  %107 = bitcast <256 x i32> %106 to x86_amx
+  %108 = load <256 x i32>, <256 x i32>* %src1.addr.i, align 64
+  %109 = bitcast <256 x i32> %108 to x86_amx
+  %110 = load <256 x i32>, <256 x i32>* %src2.addr.i, align 64
+  %111 = bitcast <256 x i32> %110 to x86_amx
+  %112 = call x86_amx @llvm.x86.tdpbssd.internal(i16 %103, i16 %104, i16 %105, x86_amx %107, x86_amx %109, x86_amx %111) #2
+  %113 = bitcast x86_amx %112 to <256 x i32>
+  %114 = load %struct.__tile1024i_str*, %struct.__tile1024i_str** %dst.addr.i44, align 8
+  %tile6.i = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %114, i32 0, i32 3
+  store <256 x i32> %113, <256 x i32>* %tile6.i, align 64
+  %115 = bitcast %struct.__tile1024i_str* %c49 to i8*
+  %116 = bitcast %struct.__tile1024i_str* %c to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %115, i8* align 1 %116, i64 1088, i1 false) #2
+  store i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i8** %base.addr.i50, align 8
+  store i64 32, i64* %stride.addr.i51, align 8
+  %row.i53 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %c49, i32 0, i32 0
+  %117 = load i16, i16* %row.i53, align 64
+  %col.i54 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %c49, i32 0, i32 1
+  %118 = load i16, i16* %col.i54, align 2
+  %119 = load i8*, i8** %base.addr.i50, align 8
+  %120 = load i64, i64* %stride.addr.i51, align 8
+  %tile.i55 = getelementptr inbounds %struct.__tile1024i_str, %struct.__tile1024i_str* %c49, i32 0, i32 3
+  %121 = load <256 x i32>, <256 x i32>* %tile.i55, align 64
+  store <256 x i32> %121, <256 x i32>* %indirect-arg-temp.i52, align 1024
+  %122 = bitcast <256 x i32>* %indirect-arg-temp.i5284 to i8*
+  %123 = bitcast <256 x i32>* %indirect-arg-temp.i52 to i8*
+  call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 1 %122, i8* align 1 %123, i64 1024, i1 false) #2
+  %tile.i89 = load <256 x i32>, <256 x i32>* %indirect-arg-temp.i5284, align 1024
+  store i16 %117, i16* %m.addr.i85, align 2
+  store i16 %118, i16* %n.addr.i86, align 2
+  store i8* %119, i8** %base.addr.i87, align 8
+  store i64 %120, i64* %stride.addr.i88, align 8
+  store <256 x i32> %tile.i89, <256 x i32>* %tile.addr.i, align 64
+  %124 = load i16, i16* %m.addr.i85, align 2
+  %125 = load i16, i16* %n.addr.i86, align 2
+  %126 = load i8*, i8** %base.addr.i87, align 8
+  %127 = load i64, i64* %stride.addr.i88, align 8
+  %128 = load <256 x i32>, <256 x i32>* %tile.addr.i, align 64
+  %129 = bitcast <256 x i32> %128 to x86_amx
+  call void @llvm.x86.tilestored64.internal(i16 %124, i16 %125, i8* %126, i64 %127, x86_amx %129) #2
+  ret void
+}
+
+; Function Attrs: argmemonly nofree nosync nounwind willreturn writeonly
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg) #1
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, i8*, i64) #2
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tdpbssd.internal(i16, i16, i16, x86_amx, x86_amx, x86_amx) #2
+
+; Function Attrs: nounwind
+declare void @llvm.x86.tilestored64.internal(i16, i16, i8*, i64, x86_amx) #2
+
+; Function Attrs: argmemonly nofree nosync nounwind willreturn
+declare void @llvm.memcpy.p0i8.p0i8.i64(i8* noalias nocapture writeonly, i8* noalias nocapture readonly, i64, i1 immarg) #3
+
+attributes #0 = { noinline nounwind optnone uwtable }
+attributes #1 = { argmemonly nofree nosync nounwind willreturn writeonly }
+attributes #2 = { nounwind }
+attributes #3 = { argmemonly nofree nosync nounwind willreturn }

llvm/test/CodeGen/X86/AMX/amx-configO2toO0-lower.ll

+; RUN: opt < %s -mtriple=x86_64-unknown-unknown -mattr=+amx-int8 -mattr=+avx512f -lower-amx-type -S | FileCheck %s --check-prefixes=LOWER_AMX
+
+@buf = dso_local global [1024 x i8] zeroinitializer, align 16
+@buf2 = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; Function Attrs: nounwind uwtable
+define dso_local void @test_api(i32 %cond, i16 signext %row, i16 signext %col) local_unnamed_addr {
+
+; LOWER_AMX-LABEL: entry:
+; LOWER_AMX:        %0 = alloca <256 x i32>, align 1024
+; LOWER_AMX-NEXT:   %1 = alloca <256 x i32>, align 1024
+; LOWER_AMX-NEXT:   %2 = alloca <256 x i32>, align 1024
+; LOWER_AMX-NEXT:   %3 = alloca <256 x i32>, align 1024
+; LOWER_AMX-NEXT:   %tobool.not = icmp eq i32 %cond, 0
+; LOWER_AMX-NEXT:   %4 = bitcast <256 x i32>* %2 to i8*
+; LOWER_AMX-NEXT:   %5 = bitcast <256 x i32>* %1 to i8*
+; LOWER_AMX-NEXT:   %6 = bitcast <256 x i32>* %0 to i8*
+; LOWER_AMX-NEXT:   br i1 %tobool.not, label %if.else, label %if.then
+; LOWER_AMX:      if.then:
+; LOWER_AMX-NEXT:   %7 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %4, i64 64, x86_amx %7)
+; LOWER_AMX-NEXT:   %8 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %5, i64 64, x86_amx %8)
+; LOWER_AMX-NEXT:   %9 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %6, i64 64, x86_amx %9)
+; LOWER_AMX-NEXT:   br label %if.end
+; LOWER_AMX:      if.else:
+; LOWER_AMX-NEXT:   %10 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %4, i64 64, x86_amx %10)
+; LOWER_AMX-NEXT:   %11 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %5, i64 64, x86_amx %11)
+; LOWER_AMX-NEXT:   %12 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %6, i64 64, x86_amx %12)
+; LOWER_AMX-NEXT:   br label %if.end
+; LOWER_AMX:      if.end:
+; LOWER_AMX-NEXT:   %13 = bitcast <256 x i32>* %3 to i8*
+; LOWER_AMX-NEXT:   %14 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* %4, i64 64)
+; LOWER_AMX-NEXT:   %15 = call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* %5, i64 64)
+; LOWER_AMX-NEXT:   %16 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %6, i64 64)
+; LOWER_AMX-NEXT:   %17 = tail call x86_amx @llvm.x86.tdpbssd.internal(i16 %row, i16 %col, i16 8, x86_amx %16, x86_amx %14, x86_amx %15)
+; LOWER_AMX-NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %13, i64 64, x86_amx %17)
+; LOWER_AMX-NEXT:   %18 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %13, i64 64)
+; LOWER_AMX-NEXT:   tail call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32, x86_amx %18)
+; LOWER_AMX-NEXT:   ret void
+
+entry:
+  %tobool.not = icmp eq i32 %cond, 0
+  br i1 %tobool.not, label %if.else, label %if.then
+
+if.then:                                          ; preds = %entry
+  %0 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  %1 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  %2 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  br label %if.end
+
+if.else:                                          ; preds = %entry
+  %3 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  %4 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  %5 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %if.then
+  %a.sroa.1094.0.in = phi x86_amx [ %3, %if.else ], [ %0, %if.then ]
+  %b.sroa.1069.0.in = phi x86_amx [ %4, %if.else ], [ %1, %if.then ]
+  %c.sroa.1044.0.in = phi x86_amx [ %5, %if.else ], [ %2, %if.then ]
+  %6 = tail call x86_amx @llvm.x86.tdpbssd.internal(i16 %row, i16 %col, i16 8, x86_amx %c.sroa.1044.0.in, x86_amx %a.sroa.1094.0.in, x86_amx %b.sroa.1069.0.in)
+  tail call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32, x86_amx %6)
+  ret void
+}
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, i8*, i64)
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tdpbssd.internal(i16, i16, i16, x86_amx, x86_amx, x86_amx)
+
+; Function Attrs: nounwind
+declare void @llvm.x86.tilestored64.internal(i16, i16, i8*, i64, x86_amx)

llvm/test/CodeGen/X86/AMX/amx-configO2toO0-precfg.ll

+; RUN: opt < %s -mtriple=x86_64-unknown-unknown -mattr=+amx-int8 -mattr=+avx512f -pre-amx-config -S | FileCheck %s --check-prefixes=PRECFG_AMX
+
+@buf = dso_local global [1024 x i8] zeroinitializer, align 16
+@buf2 = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; Function Attrs: nounwind uwtable
+define dso_local void @test_api(i32 %cond, i16 signext %row, i16 signext %col) local_unnamed_addr {
+; PRECFG_AMX_LABEL:  entry:
+; PRECFG_AMX:        %0 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %1 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %2 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %3 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %4 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %5 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %6 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %7 = alloca <16 x i32>, align 4
+; PRECFG_AMX_NEXT:   %8 = alloca <256 x i32>, align 1024
+; PRECFG_AMX_NEXT:   %9 = alloca <256 x i32>, align 1024
+; PRECFG_AMX_NEXT:   %10 = alloca <256 x i32>, align 1024
+; PRECFG_AMX_NEXT:   %11 = alloca <256 x i32>, align 1024
+; PRECFG_AMX_NEXT:   %tobool.not = icmp eq i32 %cond, 0
+; PRECFG_AMX_NEXT:   %12 = bitcast <256 x i32>* %10 to i8*
+; PRECFG_AMX_NEXT:   %13 = bitcast <256 x i32>* %9 to i8*
+; PRECFG_AMX_NEXT:   %14 = bitcast <256 x i32>* %8 to i8*
+; PRECFG_AMX_NEXT:   br i1 %tobool.not, label %if.else, label %if.then
+
+; PRECFG_AMX:      if.then:
+; PRECFG_AMX_NEXT:   %15 = bitcast <16 x i32>* %1 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %1, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row11 = getelementptr i8, i8* %15, i64 48
+; PRECFG_AMX_NEXT:   %16 = getelementptr i8, i8* %15, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col12 = bitcast i8* %16 to i16*
+; PRECFG_AMX_NEXT:   %17 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %17, i8* %amx.tmm.0.shape.row11, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 8, i16* %amx.tmm.0.shape.col12, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %15)
+; PRECFG_AMX_NEXT:   %18 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %12, i64 64, x86_amx %18)
+; PRECFG_AMX_NEXT:   %19 = bitcast <16 x i32>* %7 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %7, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row = getelementptr i8, i8* %19, i64 48
+; PRECFG_AMX_NEXT:   %20 = getelementptr i8, i8* %19, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col = bitcast i8* %20 to i16*
+; PRECFG_AMX_NEXT:   %21 = trunc i16 8 to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %21, i8* %amx.tmm.0.shape.row, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.0.shape.col, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %19)
+; PRECFG_AMX_NEXT:   %22 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %13, i64 64, x86_amx %22)
+; PRECFG_AMX_NEXT:   %23 = bitcast <16 x i32>* %0 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %0, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row13 = getelementptr i8, i8* %23, i64 48
+; PRECFG_AMX_NEXT:   %24 = getelementptr i8, i8* %23, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col14 = bitcast i8* %24 to i16*
+; PRECFG_AMX_NEXT:   %25 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %25, i8* %amx.tmm.0.shape.row13, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.0.shape.col14, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %23)
+; PRECFG_AMX_NEXT:   %26 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %14, i64 64, x86_amx %26)
+; PRECFG_AMX_NEXT:   br label %if.end
+
+; PRECFG_AMX:      if.else:
+; PRECFG_AMX_NEXT:   %27 = bitcast <16 x i32>* %2 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %2, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row9 = getelementptr i8, i8* %27, i64 48
+; PRECFG_AMX_NEXT:   %28 = getelementptr i8, i8* %27, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col10 = bitcast i8* %28 to i16*
+; PRECFG_AMX_NEXT:   %29 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %29, i8* %amx.tmm.0.shape.row9, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 8, i16* %amx.tmm.0.shape.col10, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %27)
+; PRECFG_AMX_NEXT:   %30 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %12, i64 64, x86_amx %30)
+; PRECFG_AMX_NEXT:   %31 = bitcast <16 x i32>* %3 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %3, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row7 = getelementptr i8, i8* %31, i64 48
+; PRECFG_AMX_NEXT:   %32 = getelementptr i8, i8* %31, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col8 = bitcast i8* %32 to i16*
+; PRECFG_AMX_NEXT:   %33 = trunc i16 8 to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %33, i8* %amx.tmm.0.shape.row7, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.0.shape.col8, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %31)
+; PRECFG_AMX_NEXT:   %34 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %13, i64 64, x86_amx %34)
+; PRECFG_AMX_NEXT:   %35 = bitcast <16 x i32>* %5 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %5, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row3 = getelementptr i8, i8* %35, i64 48
+; PRECFG_AMX_NEXT:   %36 = getelementptr i8, i8* %35, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col4 = bitcast i8* %36 to i16*
+; PRECFG_AMX_NEXT:   %37 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %37, i8* %amx.tmm.0.shape.row3, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.0.shape.col4, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %35)
+; PRECFG_AMX_NEXT:   %38 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %14, i64 64, x86_amx %38)
+; PRECFG_AMX_NEXT:   br label %if.end
+
+; PRECFG_AMX_NEXT: if.end:
+; PRECFG_AMX_NEXT:   %39 = bitcast <256 x i32>* %11 to i8*
+; PRECFG_AMX_NEXT:   %40 = bitcast <16 x i32>* %4 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %4, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row5 = getelementptr i8, i8* %40, i64 48
+; PRECFG_AMX_NEXT:   %41 = getelementptr i8, i8* %40, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col6 = bitcast i8* %41 to i16*
+; PRECFG_AMX_NEXT:   %42 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %42, i8* %amx.tmm.0.shape.row5, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 8, i16* %amx.tmm.0.shape.col6, align 2
+; PRECFG_AMX_NEXT:   %amx.tmm.1.shape.row = getelementptr i8, i8* %40, i64 49
+; PRECFG_AMX_NEXT:   %43 = getelementptr i8, i8* %40, i64 18
+; PRECFG_AMX_NEXT:   %amx.tmm.1.shape.col = bitcast i8* %43 to i16*
+; PRECFG_AMX_NEXT:   %44 = trunc i16 8 to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %44, i8* %amx.tmm.1.shape.row, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.1.shape.col, align 2
+; PRECFG_AMX_NEXT:   %amx.tmm.2.shape.row = getelementptr i8, i8* %40, i64 50
+; PRECFG_AMX_NEXT:   %45 = getelementptr i8, i8* %40, i64 20
+; PRECFG_AMX_NEXT:   %amx.tmm.2.shape.col = bitcast i8* %45 to i16*
+; PRECFG_AMX_NEXT:   %46 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %46, i8* %amx.tmm.2.shape.row, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.2.shape.col, align 2
+; PRECFG_AMX_NEXT:   %amx.tmm.3.shape.row = getelementptr i8, i8* %40, i64 51
+; PRECFG_AMX_NEXT:   %47 = getelementptr i8, i8* %40, i64 22
+; PRECFG_AMX_NEXT:   %amx.tmm.3.shape.col = bitcast i8* %47 to i16*
+; PRECFG_AMX_NEXT:   %48 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %48, i8* %amx.tmm.3.shape.row, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.3.shape.col, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %40)
+; PRECFG_AMX_NEXT:   %49 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* %12, i64 64)
+; PRECFG_AMX_NEXT:   %50 = call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* %13, i64 64)
+; PRECFG_AMX_NEXT:   %51 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %14, i64 64)
+; PRECFG_AMX_NEXT:   %52 = tail call x86_amx @llvm.x86.tdpbssd.internal(i16 %row, i16 %col, i16 8, x86_amx %51, x86_amx %49, x86_amx %50)
+; PRECFG_AMX_NEXT:   call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %39, i64 64, x86_amx %52)
+; PRECFG_AMX_NEXT:   %53 = bitcast <16 x i32>* %6 to i8*
+; PRECFG_AMX_NEXT:   store <16 x i32> zeroinitializer, <16 x i32>* %6, align 64
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.row1 = getelementptr i8, i8* %53, i64 48
+; PRECFG_AMX_NEXT:   %54 = getelementptr i8, i8* %53, i64 16
+; PRECFG_AMX_NEXT:   %amx.tmm.0.shape.col2 = bitcast i8* %54 to i16*
+; PRECFG_AMX_NEXT:   %55 = trunc i16 %row to i8
+; PRECFG_AMX_NEXT:   store volatile i8 %55, i8* %amx.tmm.0.shape.row1, align 1
+; PRECFG_AMX_NEXT:   store volatile i16 %col, i16* %amx.tmm.0.shape.col2, align 2
+; PRECFG_AMX_NEXT:   call void @llvm.x86.ldtilecfg.internal(i8* %53)
+; PRECFG_AMX_NEXT:   %56 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %39, i64 64)
+; PRECFG_AMX_NEXT:   tail call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32, x86_amx %56)
+; PRECFG_AMX_NEXT:   ret void
+entry:
+  %0 = alloca <256 x i32>, align 1024
+  %1 = alloca <256 x i32>, align 1024
+  %2 = alloca <256 x i32>, align 1024
+  %3 = alloca <256 x i32>, align 1024
+  %tobool.not = icmp eq i32 %cond, 0
+  %4 = bitcast <256 x i32>* %2 to i8*
+  %5 = bitcast <256 x i32>* %1 to i8*
+  %6 = bitcast <256 x i32>* %0 to i8*
+  br i1 %tobool.not, label %if.else, label %if.then
+if.then:
+  %7 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %4, i64 64, x86_amx %7)
+  %8 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %5, i64 64, x86_amx %8)
+  %9 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %6, i64 64, x86_amx %9)
+  br label %if.end
+if.else:
+  %10 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %4, i64 64, x86_amx %10)
+  %11 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %5, i64 64, x86_amx %11)
+  %12 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %6, i64 64, x86_amx %12)
+  br label %if.end
+if.end:
+  %13 = bitcast <256 x i32>* %3 to i8*
+  %14 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* %4, i64 64)
+  %15 = call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* %5, i64 64)
+  %16 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %6, i64 64)
+  %17 = tail call x86_amx @llvm.x86.tdpbssd.internal(i16 %row, i16 %col, i16 8, x86_amx %16, x86_amx %14, x86_amx %15)
+  call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %13, i64 64, x86_amx %17)
+  %18 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %13, i64 64)
+  tail call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32, x86_amx %18)
+  ret void
+}
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, i8*, i64)
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tdpbssd.internal(i16, i16, i16, x86_amx, x86_amx, x86_amx)
+
+; Function Attrs: nounwind
+declare void @llvm.x86.tilestored64.internal(i16, i16, i8*, i64, x86_amx)

llvm/test/CodeGen/X86/AMX/amx-configO2toO0.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -O0 -mtriple=x86_64-unknown-unknown -mattr=+amx-int8 -mattr=+avx512f | FileCheck %s --check-prefix=AMX_O0
+
+@buf = dso_local global [1024 x i8] zeroinitializer, align 16
+@buf2 = dso_local global [1024 x i8] zeroinitializer, align 16
+
+; Function Attrs: nounwind uwtable
+define dso_local void @test_api(i32 %cond, i16 signext %row, i16 signext %col) local_unnamed_addr {
+; AMX_O0-LABEL: test_api:
+; AMX_O0:       # %bb.0: # %entry
+; AMX_O0-NEXT:    pushq %rbp
+; AMX_O0-NEXT:    .cfi_def_cfa_offset 16
+; AMX_O0-NEXT:    .cfi_offset %rbp, -16
+; AMX_O0-NEXT:    movq %rsp, %rbp
+; AMX_O0-NEXT:    .cfi_def_cfa_register %rbp
+; AMX_O0-NEXT:    andq $-1024, %rsp # imm = 0xFC00
+; AMX_O0-NEXT:    subq $6144, %rsp # imm = 0x1800
+; AMX_O0-NEXT:    movw %dx, %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    movw %si, %ax
+; AMX_O0-NEXT:    movw %ax, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; AMX_O0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; AMX_O0-NEXT:    cmpl $0, %edi
+; AMX_O0-NEXT:    je .LBB0_2
+; AMX_O0-NEXT:  # %bb.1: # %if.then
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %ax # 2-byte Reload
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %cx # 2-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r11 # 8-byte Reload
+; AMX_O0-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %sil
+; AMX_O0-NEXT:    movb %sil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    ldtilecfg (%rsi)
+; AMX_O0-NEXT:    movl $buf, %r9d
+; AMX_O0-NEXT:    movl $32, %r10d
+; AMX_O0-NEXT:    movw $8, %si
+; AMX_O0-NEXT:    tileloadd (%r9,%r10), %tmm0
+; AMX_O0-NEXT:    movl $64, %r8d
+; AMX_O0-NEXT:    tilestored %tmm0, (%r11,%r8)
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r11
+; AMX_O0-NEXT:    ldtilecfg (%r11)
+; AMX_O0-NEXT:    tileloadd (%r9,%r10), %tmm0
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdi,%r8)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %dil
+; AMX_O0-NEXT:    movb %dil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    ldtilecfg (%rsi)
+; AMX_O0-NEXT:    movl $buf, %esi
+; AMX_O0-NEXT:    movl $32, %edi
+; AMX_O0-NEXT:    tileloadd (%rsi,%rdi), %tmm0
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    jmp .LBB0_3
+; AMX_O0-NEXT:  .LBB0_2: # %if.else
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %ax # 2-byte Reload
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %cx # 2-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r11 # 8-byte Reload
+; AMX_O0-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %sil
+; AMX_O0-NEXT:    movb %sil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    ldtilecfg (%rsi)
+; AMX_O0-NEXT:    movl $buf2, %r9d
+; AMX_O0-NEXT:    movl $32, %r10d
+; AMX_O0-NEXT:    movw $8, %si
+; AMX_O0-NEXT:    tileloadd (%r9,%r10), %tmm0
+; AMX_O0-NEXT:    movl $64, %r8d
+; AMX_O0-NEXT:    tilestored %tmm0, (%r11,%r8)
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %r11
+; AMX_O0-NEXT:    ldtilecfg (%r11)
+; AMX_O0-NEXT:    tileloadd (%r9,%r10), %tmm0
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdi,%r8)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %dil
+; AMX_O0-NEXT:    movb %dil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    ldtilecfg (%rsi)
+; AMX_O0-NEXT:    movl $buf2, %esi
+; AMX_O0-NEXT:    movl $32, %edi
+; AMX_O0-NEXT:    tileloadd (%rsi,%rdi), %tmm0
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:  .LBB0_3: # %if.end
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %ax # 2-byte Reload
+; AMX_O0-NEXT:    movw {{[-0-9]+}}(%r{{[sb]}}p), %cx # 2-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; AMX_O0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r10 # 8-byte Reload
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rdx
+; AMX_O0-NEXT:    vxorps %xmm0, %xmm0, %xmm0
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %sil
+; AMX_O0-NEXT:    movb %sil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb $8, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %sil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %sil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    ldtilecfg (%rsi)
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    movw $8, %di
+; AMX_O0-NEXT:    tileloadd (%r10,%rsi), %tmm1
+; AMX_O0-NEXT:    tileloadd (%r9,%rsi), %tmm2
+; AMX_O0-NEXT:    tileloadd (%r8,%rsi), %tmm0
+; AMX_O0-NEXT:    tdpbssd %tmm2, %tmm1, %tmm0
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    leaq {{[0-9]+}}(%rsp), %rsi
+; AMX_O0-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movb %al, %dil
+; AMX_O0-NEXT:    movb %dil, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    movw %cx, {{[0-9]+}}(%rsp)
+; AMX_O0-NEXT:    ldtilecfg (%rsi)
+; AMX_O0-NEXT:    movl $64, %esi
+; AMX_O0-NEXT:    tileloadd (%rdx,%rsi), %tmm0
+; AMX_O0-NEXT:    movl $buf, %edx
+; AMX_O0-NEXT:    movl $32, %esi
+; AMX_O0-NEXT:    tilestored %tmm0, (%rdx,%rsi)
+; AMX_O0-NEXT:    movq %rbp, %rsp
+; AMX_O0-NEXT:    popq %rbp
+; AMX_O0-NEXT:    .cfi_def_cfa %rsp, 8
+; AMX_O0-NEXT:    tilerelease
+; AMX_O0-NEXT:    vzeroupper
+; AMX_O0-NEXT:    retq
+entry:
+  %tobool.not = icmp eq i32 %cond, 0
+  br i1 %tobool.not, label %if.else, label %if.then
+
+if.then:                                          ; preds = %entry
+  %0 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  %1 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  %2 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+  br label %if.end
+
+if.else:                                          ; preds = %entry
+  %3 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  %4 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  %5 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %if.then
+  %a.sroa.1094.0.in = phi x86_amx [ %3, %if.else ], [ %0, %if.then ]
+  %b.sroa.1069.0.in = phi x86_amx [ %4, %if.else ], [ %1, %if.then ]
+  %c.sroa.1044.0.in = phi x86_amx [ %5, %if.else ], [ %2, %if.then ]
+  %6 = tail call x86_amx @llvm.x86.tdpbssd.internal(i16 %row, i16 %col, i16 8, x86_amx %c.sroa.1044.0.in, x86_amx %a.sroa.1094.0.in, x86_amx %b.sroa.1069.0.in)
+  tail call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32, x86_amx %6)
+  ret void
+}
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, i8*, i64)
+
+; Function Attrs: nounwind
+declare x86_amx @llvm.x86.tdpbssd.internal(i16, i16, i16, x86_amx, x86_amx, x86_amx)
+
+; Function Attrs: nounwind
+declare void @llvm.x86.tilestored64.internal(i16, i16, i8*, i64, x86_amx)

llvm/test/CodeGen/X86/AMX/amx-fast-tile-config.mir

+# RUN: llc -o - -mtriple=x86_64-unknown-unknown -mattr=+amx-int8 -mattr=+avx512f -run-pass=fasttileconfig %s | FileCheck %s
+
+--- |
+  @buf = dso_local global [1024 x i8] zeroinitializer, align 16
+  @buf2 = dso_local global [1024 x i8] zeroinitializer, align 16
+
+  define dso_local void @test_api(i32 %cond, i16 signext %row, i16 signext %col) local_unnamed_addr #0 {
+  entry:
+    %0 = alloca <16 x i32>, align 4
+    %1 = alloca <16 x i32>, align 4
+    %2 = alloca <16 x i32>, align 4
+    %3 = alloca <16 x i32>, align 4
+    %4 = alloca <16 x i32>, align 4
+    %5 = alloca <16 x i32>, align 4
+    %6 = alloca <16 x i32>, align 4
+    %7 = alloca <16 x i32>, align 4
+    %8 = alloca <256 x i32>, align 1024
+    %9 = alloca <256 x i32>, align 1024
+    %10 = alloca <256 x i32>, align 1024
+    %11 = alloca <256 x i32>, align 1024
+    %tobool.not = icmp eq i32 %cond, 0
+    %12 = bitcast <256 x i32>* %10 to i8*
+    %13 = bitcast <256 x i32>* %9 to i8*
+    %14 = bitcast <256 x i32>* %8 to i8*
+    br i1 %tobool.not, label %if.else, label %if.then
+
+  if.then:                                          ; preds = %entry
+    %15 = bitcast <16 x i32>* %2 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %2, align 64
+    %amx.tmm.0.shape.row9 = getelementptr i8, i8* %15, i64 48
+    %16 = getelementptr i8, i8* %15, i64 16
+    %amx.tmm.0.shape.col10 = bitcast i8* %16 to i16*
+    %17 = trunc i16 %row to i8
+    store volatile i8 %17, i8* %amx.tmm.0.shape.row9, align 1
+    store volatile i16 8, i16* %amx.tmm.0.shape.col10, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %15)
+    %18 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+    call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %12, i64 64, x86_amx %18)
+    %19 = bitcast <16 x i32>* %6 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %6, align 64
+    %amx.tmm.0.shape.row1 = getelementptr i8, i8* %19, i64 48
+    %20 = getelementptr i8, i8* %19, i64 16
+    %amx.tmm.0.shape.col2 = bitcast i8* %20 to i16*
+    %21 = trunc i16 8 to i8
+    store volatile i8 %21, i8* %amx.tmm.0.shape.row1, align 1
+    store volatile i16 %col, i16* %amx.tmm.0.shape.col2, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %19)
+    %22 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+    call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %13, i64 64, x86_amx %22)
+    %23 = bitcast <16 x i32>* %0 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %0, align 64
+    %amx.tmm.0.shape.row13 = getelementptr i8, i8* %23, i64 48
+    %24 = getelementptr i8, i8* %23, i64 16
+    %amx.tmm.0.shape.col14 = bitcast i8* %24 to i16*
+    %25 = trunc i16 %row to i8
+    store volatile i8 %25, i8* %amx.tmm.0.shape.row13, align 1
+    store volatile i16 %col, i16* %amx.tmm.0.shape.col14, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %23)
+    %26 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32)
+    call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %14, i64 64, x86_amx %26)
+    br label %if.end
+
+  if.else:                                          ; preds = %entry
+    %27 = bitcast <16 x i32>* %1 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %1, align 64
+    %amx.tmm.0.shape.row11 = getelementptr i8, i8* %27, i64 48
+    %28 = getelementptr i8, i8* %27, i64 16
+    %amx.tmm.0.shape.col12 = bitcast i8* %28 to i16*
+    %29 = trunc i16 %row to i8
+    store volatile i8 %29, i8* %amx.tmm.0.shape.row11, align 1
+    store volatile i16 8, i16* %amx.tmm.0.shape.col12, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %27)
+    %30 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+    call void @llvm.x86.tilestored64.internal(i16 %row, i16 8, i8* %12, i64 64, x86_amx %30)
+    %31 = bitcast <16 x i32>* %7 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %7, align 64
+    %amx.tmm.0.shape.row = getelementptr i8, i8* %31, i64 48
+    %32 = getelementptr i8, i8* %31, i64 16
+    %amx.tmm.0.shape.col = bitcast i8* %32 to i16*
+    %33 = trunc i16 8 to i8
+    store volatile i8 %33, i8* %amx.tmm.0.shape.row, align 1
+    store volatile i16 %col, i16* %amx.tmm.0.shape.col, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %31)
+    %34 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+    call void @llvm.x86.tilestored64.internal(i16 8, i16 %col, i8* %13, i64 64, x86_amx %34)
+    %35 = bitcast <16 x i32>* %4 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %4, align 64
+    %amx.tmm.0.shape.row5 = getelementptr i8, i8* %35, i64 48
+    %36 = getelementptr i8, i8* %35, i64 16
+    %amx.tmm.0.shape.col6 = bitcast i8* %36 to i16*
+    %37 = trunc i16 %row to i8
+    store volatile i8 %37, i8* %amx.tmm.0.shape.row5, align 1
+    store volatile i16 %col, i16* %amx.tmm.0.shape.col6, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %35)
+    %38 = tail call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf2, i64 0, i64 0), i64 32)
+    call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %14, i64 64, x86_amx %38)
+    br label %if.end
+
+  if.end:                                           ; preds = %if.else, %if.then
+    %39 = bitcast <256 x i32>* %11 to i8*
+    %40 = bitcast <16 x i32>* %5 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %5, align 64
+    %amx.tmm.0.shape.row3 = getelementptr i8, i8* %40, i64 48
+    %41 = getelementptr i8, i8* %40, i64 16
+    %amx.tmm.0.shape.col4 = bitcast i8* %41 to i16*
+    %42 = trunc i16 %row to i8
+    store volatile i8 %42, i8* %amx.tmm.0.shape.row3, align 1
+    store volatile i16 8, i16* %amx.tmm.0.shape.col4, align 2
+    %amx.tmm.1.shape.row = getelementptr i8, i8* %40, i64 49
+    %43 = getelementptr i8, i8* %40, i64 18
+    %amx.tmm.1.shape.col = bitcast i8* %43 to i16*
+    %44 = trunc i16 8 to i8
+    store volatile i8 %44, i8* %amx.tmm.1.shape.row, align 1
+    store volatile i16 %col, i16* %amx.tmm.1.shape.col, align 2
+    %amx.tmm.2.shape.row = getelementptr i8, i8* %40, i64 50
+    %45 = getelementptr i8, i8* %40, i64 20
+    %amx.tmm.2.shape.col = bitcast i8* %45 to i16*
+    %46 = trunc i16 %row to i8
+    store volatile i8 %46, i8* %amx.tmm.2.shape.row, align 1
+    store volatile i16 %col, i16* %amx.tmm.2.shape.col, align 2
+    %amx.tmm.3.shape.row = getelementptr i8, i8* %40, i64 51
+    %47 = getelementptr i8, i8* %40, i64 22
+    %amx.tmm.3.shape.col = bitcast i8* %47 to i16*
+    %48 = trunc i16 %row to i8
+    store volatile i8 %48, i8* %amx.tmm.3.shape.row, align 1
+    store volatile i16 %col, i16* %amx.tmm.3.shape.col, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %40)
+    %49 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 8, i8* %12, i64 64)
+    %50 = call x86_amx @llvm.x86.tileloadd64.internal(i16 8, i16 %col, i8* %13, i64 64)
+    %51 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %14, i64 64)
+    %52 = tail call x86_amx @llvm.x86.tdpbssd.internal(i16 %row, i16 %col, i16 8, x86_amx %51, x86_amx %49, x86_amx %50)
+    call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* %39, i64 64, x86_amx %52)
+    %53 = bitcast <16 x i32>* %3 to i8*
+    store <16 x i32> zeroinitializer, <16 x i32>* %3, align 64
+    %amx.tmm.0.shape.row7 = getelementptr i8, i8* %53, i64 48
+    %54 = getelementptr i8, i8* %53, i64 16
+    %amx.tmm.0.shape.col8 = bitcast i8* %54 to i16*
+    %55 = trunc i16 %row to i8
+    store volatile i8 %55, i8* %amx.tmm.0.shape.row7, align 1
+    store volatile i16 %col, i16* %amx.tmm.0.shape.col8, align 2
+    call void @llvm.x86.ldtilecfg.internal(i8* %53)
+    %56 = call x86_amx @llvm.x86.tileloadd64.internal(i16 %row, i16 %col, i8* %39, i64 64)
+    tail call void @llvm.x86.tilestored64.internal(i16 %row, i16 %col, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @buf, i64 0, i64 0), i64 32, x86_amx %56)
+    ret void
+  }
+
+  ; Function Attrs: nounwind
+  declare x86_amx @llvm.x86.tileloadd64.internal(i16, i16, i8*, i64) #1
+
+  ; Function Attrs: nounwind
+  declare x86_amx @llvm.x86.tdpbssd.internal(i16, i16, i16, x86_amx, x86_amx, x86_amx) #1
+
+  ; Function Attrs: nounwind
+  declare void @llvm.x86.tilestored64.internal(i16, i16, i8*, i64, x86_amx) #1
+
+  ; Function Attrs: nounwind
+  declare void @llvm.x86.ldtilecfg.internal(i8*) #2
+
+  attributes #0 = { "target-features"="+amx-int8,+avx512f" }
+  attributes #1 = { nounwind "target-features"="+amx-int8,+avx512f" }
+  attributes #2 = { nounwind }
+
+...
+---
+name:            test_api
+alignment:       16
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$edi', virtual-reg: '' }
+  - { reg: '$esi', virtual-reg: '' }
+  - { reg: '$edx', virtual-reg: '' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1024
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 4294967295
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 2, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 3, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 4, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 5, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 6, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 7, name: '', type: default, offset: 0, size: 64, alignment: 16,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 8, name: '', type: default, offset: 0, size: 1024, alignment: 1024,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 9, name: '', type: default, offset: 0, size: 1024, alignment: 1024,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 10, name: '', type: default, offset: 0, size: 1024, alignment: 1024,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 11, name: '', type: default, offset: 0, size: 1024, alignment: 1024,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 12, name: '', type: spill-slot, offset: 0, size: 8, alignment: 8,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 13, name: '', type: spill-slot, offset: 0, size: 8, alignment: 8,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 14, name: '', type: spill-slot, offset: 0, size: 8, alignment: 8,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 15, name: '', type: spill-slot, offset: 0, size: 2, alignment: 2,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 16, name: '', type: spill-slot, offset: 0, size: 2, alignment: 2,
+      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.2(0x40000000), %bb.1(0x40000000)
+    liveins: $edi, $esi, $edx
+
+    renamable $ax = COPY renamable $dx, implicit killed $edx
+    MOV16mr %stack.16, 1, $noreg, 0, $noreg, killed $ax :: (store 2 into %stack.16)
+    renamable $ax = COPY renamable $si, implicit killed $esi
+    MOV16mr %stack.15, 1, $noreg, 0, $noreg, killed $ax :: (store 2 into %stack.15)
+    renamable $rax = LEA64r %stack.10, 1, $noreg, 0, $noreg
+    MOV64mr %stack.14, 1, $noreg, 0, $noreg, killed $rax :: (store 8 into %stack.14)
+    renamable $rax = LEA64r %stack.9, 1, $noreg, 0, $noreg
+    MOV64mr %stack.13, 1, $noreg, 0, $noreg, killed $rax :: (store 8 into %stack.13)
+    renamable $rax = LEA64r %stack.8, 1, $noreg, 0, $noreg
+    MOV64mr %stack.12, 1, $noreg, 0, $noreg, killed $rax :: (store 8 into %stack.12)
+    CMP32ri8 killed renamable $edi, 0, implicit-def $eflags
+    JCC_1 %bb.2, 4, implicit killed $eflags
+
+  bb.1.if.then:
+    successors: %bb.3(0x80000000)
+     ; CHECK-LABEL: bb.1.if.then
+     ; tmm1 --> row_offset = 49, col_offset = 18
+     ; CHECK: MOV8mr %stack.2, 1, $noreg, 49, $noreg, killed renamable $sil :: (volatile store 1 into %ir.amx.tmm.0.shape.row9)
+     ; CHECK: MOV16mi %stack.2, 1, $noreg, 18, $noreg, 8 :: (volatile store 2 into %ir.amx.tmm.0.shape.col10)
+     ; CHECK: PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $tmm1 = PTILELOADDV renamable $ax, renamable $si, renamable $r9, 1, renamable $r10, 0, $noreg
+     ; CHECK: PTILESTOREDV renamable $ax, renamable $si, renamable $r11, 1, renamable $r8, 0, $noreg, killed renamable $tmm1
+
+     ; tmm0 --> row_offset = 48, col_offset = 16
+     ; CHECK: MOV8mi %stack.6, 1, $noreg, 48, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.0.shape.row1)
+     ; CHECK: MOV16mr %stack.6, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col2)
+     ; CHECK: PLDTILECFGV killed renamable $r11, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $tmm0 = PTILELOADDV renamable $si, renamable $cx, killed renamable $r9, 1, killed renamable $r10, 0, $noreg
+     ; CHECK: PTILESTOREDV killed renamable $si, renamable $cx, renamable $rdi, 1, killed renamable $r8, 0, $noreg, killed renamable $tmm0
+
+     ; tmm2 --> row_offset = 50, col_offset = 20
+     ; CHECK: MOV8mr %stack.0, 1, $noreg, 50, $noreg, killed renamable $dil :: (volatile store 1 into %ir.amx.tmm.0.shape.row13)
+     ; CHECK: MOV16mr %stack.0, 1, $noreg, 20, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col14)
+     ; CHECK: PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $tmm2 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $rsi, 1, killed renamable $rdi, 0, $noreg
+     ; CHECK: PTILESTOREDV renamable $ax, renamable $cx, renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm2
+
+    $ax = MOV16rm %stack.15, 1, $noreg, 0, $noreg :: (load 2 from %stack.15)
+    $cx = MOV16rm %stack.16, 1, $noreg, 0, $noreg :: (load 2 from %stack.16)
+    $rdx = MOV64rm %stack.12, 1, $noreg, 0, $noreg :: (load 8 from %stack.12)
+    $rdi = MOV64rm %stack.13, 1, $noreg, 0, $noreg :: (load 8 from %stack.13)
+    $r11 = MOV64rm %stack.14, 1, $noreg, 0, $noreg :: (load 8 from %stack.14)
+    renamable $zmm0 = AVX512_512_SET0
+    VMOVDQA64Zmr %stack.2, 1, $noreg, 0, $noreg, renamable $zmm0 :: (store 64 into %ir.2)
+    renamable $sil = COPY renamable $al
+    MOV8mr %stack.2, 1, $noreg, 48, $noreg, killed renamable $sil :: (volatile store 1 into %ir.amx.tmm.0.shape.row9)
+    MOV16mi %stack.2, 1, $noreg, 16, $noreg, 8 :: (volatile store 2 into %ir.amx.tmm.0.shape.col10)
+    renamable $rsi = LEA64r %stack.2, 1, $noreg, 0, $noreg
+    PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+    renamable $r9 = MOV32ri64 @buf
+    renamable $r10 = MOV32ri64 32
+    renamable $si = MOV16ri 8
+    renamable $tmm1 = PTILELOADDV renamable $ax, renamable $si, renamable $r9, 1, renamable $r10, 0, $noreg
+    renamable $r8 = MOV32ri64 64
+    PTILESTOREDV renamable $ax, renamable $si, renamable $r11, 1, renamable $r8, 0, $noreg, killed renamable $tmm1
+    VMOVDQA64Zmr %stack.6, 1, $noreg, 0, $noreg, renamable $zmm0 :: (store 64 into %ir.6)
+    MOV8mi %stack.6, 1, $noreg, 48, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.0.shape.row1)
+    MOV16mr %stack.6, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col2)
+    renamable $r11 = LEA64r %stack.6, 1, $noreg, 0, $noreg
+    PLDTILECFGV killed renamable $r11, 1, $noreg, 0, $noreg
+    renamable $tmm0 = PTILELOADDV renamable $si, renamable $cx, killed renamable $r9, 1, killed renamable $r10, 0, $noreg
+    PTILESTOREDV killed renamable $si, renamable $cx, renamable $rdi, 1, killed renamable $r8, 0, $noreg, killed renamable $tmm0
+    renamable $rsi = LEA64r %stack.0, 1, $noreg, 0, $noreg
+    VMOVDQA64Zmr %stack.0, 1, $noreg, 0, $noreg, killed renamable $zmm0 :: (store 64 into %ir.0)
+    renamable $dil = COPY renamable $al
+    MOV8mr %stack.0, 1, $noreg, 48, $noreg, killed renamable $dil :: (volatile store 1 into %ir.amx.tmm.0.shape.row13)
+    MOV16mr %stack.0, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col14)
+    PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+    renamable $rsi = MOV32ri64 @buf
+    renamable $rdi = MOV32ri64 32
+    renamable $tmm2 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $rsi, 1, killed renamable $rdi, 0, $noreg
+    renamable $rsi = MOV32ri64 64
+    PTILESTOREDV renamable $ax, renamable $cx, renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm2
+    JMP_1 %bb.3
+
+  bb.2.if.else:
+    successors: %bb.3(0x80000000)
+     ; CHECK: bb.2.if.else:
+     ; tmm3 --> row_offset = 51, col_offset = 22
+     ; CHECK: MOV8mr %stack.1, 1, $noreg, 51, $noreg, killed renamable $sil :: (volatile store 1 into %ir.amx.tmm.0.shape.row11)
+     ; CHECK: MOV16mi %stack.1, 1, $noreg, 22, $noreg, 8 :: (volatile store 2 into %ir.amx.tmm.0.shape.col12)
+     ; CHECK: PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $tmm3 = PTILELOADDV renamable $ax, renamable $si, renamable $r9, 1, renamable $r10, 0, $noreg
+     ; CHECK: PTILESTOREDV renamable $ax, renamable $si, renamable $r11, 1, renamable $r8, 0, $noreg, killed renamable $tmm3
+
+     ; tmm3 --> row_offset = 52, col_offset = 24
+     ; CHECK: MOV8mi %stack.7, 1, $noreg, 52, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.0.shape.row)
+     ; CHECK: MOV16mr %stack.7, 1, $noreg, 24, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col)
+     ; CHECK: PLDTILECFGV killed renamable $r11, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $tmm4 = PTILELOADDV renamable $si, renamable $cx, killed renamable $r9, 1, killed renamable $r10, 0, $noreg
+     ; CHECK: PTILESTOREDV killed renamable $si, renamable $cx, renamable $rdi, 1, killed renamable $r8, 0, $noreg, killed renamable $tmm4
+
+     ; tmm3 --> row_offset = 53, col_offset = 26
+     ; CHECK: MOV8mr %stack.4, 1, $noreg, 53, $noreg, killed renamable $dil :: (volatile store 1 into %ir.amx.tmm.0.shape.row5)
+     ; CHECK: MOV16mr %stack.4, 1, $noreg, 26, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col6)
+     ; CHECK: PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $tmm5 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $rsi, 1, killed renamable $rdi, 0, $noreg
+     ; CHECK: PTILESTOREDV renamable $ax, renamable $cx, renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm5
+
+    $ax = MOV16rm %stack.15, 1, $noreg, 0, $noreg :: (load 2 from %stack.15)
+    $cx = MOV16rm %stack.16, 1, $noreg, 0, $noreg :: (load 2 from %stack.16)
+    $rdx = MOV64rm %stack.12, 1, $noreg, 0, $noreg :: (load 8 from %stack.12)
+    $rdi = MOV64rm %stack.13, 1, $noreg, 0, $noreg :: (load 8 from %stack.13)
+    $r11 = MOV64rm %stack.14, 1, $noreg, 0, $noreg :: (load 8 from %stack.14)
+    renamable $zmm0 = AVX512_512_SET0
+    VMOVDQA64Zmr %stack.1, 1, $noreg, 0, $noreg, renamable $zmm0 :: (store 64 into %ir.1)
+    renamable $sil = COPY renamable $al
+    MOV8mr %stack.1, 1, $noreg, 48, $noreg, killed renamable $sil :: (volatile store 1 into %ir.amx.tmm.0.shape.row11)
+    MOV16mi %stack.1, 1, $noreg, 16, $noreg, 8 :: (volatile store 2 into %ir.amx.tmm.0.shape.col12)
+    renamable $rsi = LEA64r %stack.1, 1, $noreg, 0, $noreg
+    PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+    renamable $r9 = MOV32ri64 @buf2
+    renamable $r10 = MOV32ri64 32
+    renamable $si = MOV16ri 8
+    renamable $tmm3 = PTILELOADDV renamable $ax, renamable $si, renamable $r9, 1, renamable $r10, 0, $noreg
+    renamable $r8 = MOV32ri64 64
+    PTILESTOREDV renamable $ax, renamable $si, renamable $r11, 1, renamable $r8, 0, $noreg, killed renamable $tmm3
+    VMOVDQA64Zmr %stack.7, 1, $noreg, 0, $noreg, renamable $zmm0 :: (store 64 into %ir.7)
+    MOV8mi %stack.7, 1, $noreg, 48, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.0.shape.row)
+    MOV16mr %stack.7, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col)
+    renamable $r11 = LEA64r %stack.7, 1, $noreg, 0, $noreg
+    PLDTILECFGV killed renamable $r11, 1, $noreg, 0, $noreg
+    renamable $tmm4 = PTILELOADDV renamable $si, renamable $cx, killed renamable $r9, 1, killed renamable $r10, 0, $noreg
+    PTILESTOREDV killed renamable $si, renamable $cx, renamable $rdi, 1, killed renamable $r8, 0, $noreg, killed renamable $tmm4
+    renamable $rsi = LEA64r %stack.4, 1, $noreg, 0, $noreg
+    VMOVDQA64Zmr %stack.4, 1, $noreg, 0, $noreg, killed renamable $zmm0 :: (store 64 into %ir.4)
+    renamable $dil = COPY renamable $al
+    MOV8mr %stack.4, 1, $noreg, 48, $noreg, killed renamable $dil :: (volatile store 1 into %ir.amx.tmm.0.shape.row5)
+    MOV16mr %stack.4, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col6)
+    PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+    renamable $rsi = MOV32ri64 @buf2
+    renamable $rdi = MOV32ri64 32
+    renamable $tmm5 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $rsi, 1, killed renamable $rdi, 0, $noreg
+    renamable $rsi = MOV32ri64 64
+    PTILESTOREDV renamable $ax, renamable $cx, renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm5
+
+  bb.3.if.end:
+     ; CHECK: bb.3.if.end:
+     ; tmm0 --> row_offset = 48, col_offset = 16
+     ; tmm1 --> row_offset = 49, col_offset = 18
+     ; tmm2 --> row_offset = 50, col_offset = 20
+
+     ; CHECK: MOV8mr %stack.5, 1, $noreg, 48, $noreg, renamable $sil :: (volatile store 1 into %ir.amx.tmm.0.shape.row3)
+     ; CHECK: MOV16mi %stack.5, 1, $noreg, 16, $noreg, 8 :: (volatile store 2 into %ir.amx.tmm.0.shape.col4)
+     ; CHECK: MOV8mi %stack.5, 1, $noreg, 49, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.1.shape.row)
+     ; CHECK: MOV16mr %stack.5, 1, $noreg, 18, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.1.shape.col)
+     ; CHECK: MOV8mr %stack.5, 1, $noreg, 50, $noreg, renamable $sil :: (volatile store 1 into %ir.amx.tmm.2.shape.row)
+     ; CHECK: MOV16mr %stack.5, 1, $noreg, 20, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.2.shape.col)
+     ; CHECK: MOV8mr %stack.5, 1, $noreg, 48, $noreg, killed renamable $sil :: (volatile store 1 into %ir.amx.tmm.3.shape.row)
+     ; CHECK: MOV16mr %stack.5, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.3.shape.col)
+     ; CHECK: renamable $rsi = LEA64r %stack.5, 1, $noreg, 0, $noreg
+     ; CHECK: PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+     ; CHECK: renamable $rsi = MOV32ri64 64
+     ; CHECK: renamable $di = MOV16ri 8
+     ; CHECK: renamable $tmm1 = PTILELOADDV renamable $ax, renamable $di, killed renamable $r10, 1, renamable $rsi, 0, $noreg
+     ; CHECK: renamable $tmm2 = PTILELOADDV renamable $di, renamable $cx, killed renamable $r9, 1, renamable $rsi, 0, $noreg
+     ; CHECK: renamable $tmm0 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $r8, 1, renamable $rsi, 0, $noreg
+     ; CHECK: renamable $tmm0 = PTDPBSSDV renamable $ax, renamable $cx, killed renamable $di, renamable $tmm0, killed renamable $tmm1, killed renamable $tmm2
+     ; CHECK: PTILESTOREDV renamable $ax, renamable $cx, renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm0
+
+    $ax = MOV16rm %stack.15, 1, $noreg, 0, $noreg :: (load 2 from %stack.15)
+    $cx = MOV16rm %stack.16, 1, $noreg, 0, $noreg :: (load 2 from %stack.16)
+    $r8 = MOV64rm %stack.12, 1, $noreg, 0, $noreg :: (load 8 from %stack.12)
+    $r9 = MOV64rm %stack.13, 1, $noreg, 0, $noreg :: (load 8 from %stack.13)
+    $r10 = MOV64rm %stack.14, 1, $noreg, 0, $noreg :: (load 8 from %stack.14)
+    renamable $rdx = LEA64r %stack.11, 1, $noreg, 0, $noreg
+    renamable $zmm0 = AVX512_512_SET0
+    VMOVDQA64Zmr %stack.5, 1, $noreg, 0, $noreg, renamable $zmm0 :: (store 64 into %ir.5)
+    renamable $sil = COPY renamable $al
+    MOV8mr %stack.5, 1, $noreg, 48, $noreg, renamable $sil :: (volatile store 1 into %ir.amx.tmm.0.shape.row3)
+    MOV16mi %stack.5, 1, $noreg, 16, $noreg, 8 :: (volatile store 2 into %ir.amx.tmm.0.shape.col4)
+    MOV8mi %stack.5, 1, $noreg, 49, $noreg, 8 :: (volatile store 1 into %ir.amx.tmm.1.shape.row)
+    MOV16mr %stack.5, 1, $noreg, 18, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.1.shape.col)
+    MOV8mr %stack.5, 1, $noreg, 50, $noreg, renamable $sil :: (volatile store 1 into %ir.amx.tmm.2.shape.row)
+    MOV16mr %stack.5, 1, $noreg, 20, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.2.shape.col)
+    MOV8mr %stack.5, 1, $noreg, 51, $noreg, killed renamable $sil :: (volatile store 1 into %ir.amx.tmm.3.shape.row)
+    MOV16mr %stack.5, 1, $noreg, 22, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.3.shape.col)
+    renamable $rsi = LEA64r %stack.5, 1, $noreg, 0, $noreg
+    PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+    renamable $rsi = MOV32ri64 64
+    renamable $di = MOV16ri 8
+    renamable $tmm1 = PTILELOADDV renamable $ax, renamable $di, killed renamable $r10, 1, renamable $rsi, 0, $noreg
+    renamable $tmm2 = PTILELOADDV renamable $di, renamable $cx, killed renamable $r9, 1, renamable $rsi, 0, $noreg
+    renamable $tmm0 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $r8, 1, renamable $rsi, 0, $noreg
+    renamable $tmm0 = PTDPBSSDV renamable $ax, renamable $cx, killed renamable $di, renamable $tmm0, killed renamable $tmm1, killed renamable $tmm2
+    PTILESTOREDV renamable $ax, renamable $cx, renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm0
+    renamable $rsi = LEA64r %stack.3, 1, $noreg, 0, $noreg
+    VMOVDQA64Zmr %stack.3, 1, $noreg, 0, $noreg, killed renamable $zmm0 :: (store 64 into %ir.3)
+    renamable $dil = COPY renamable $al
+    MOV8mr %stack.3, 1, $noreg, 48, $noreg, killed renamable $dil :: (volatile store 1 into %ir.amx.tmm.0.shape.row7)
+    MOV16mr %stack.3, 1, $noreg, 16, $noreg, renamable $cx :: (volatile store 2 into %ir.amx.tmm.0.shape.col8)
+    PLDTILECFGV killed renamable $rsi, 1, $noreg, 0, $noreg
+    renamable $rsi = MOV32ri64 64
+    renamable $tmm0 = PTILELOADDV renamable $ax, renamable $cx, killed renamable $rdx, 1, killed renamable $rsi, 0, $noreg
+    renamable $rdx = MOV32ri64 @buf
+    renamable $rsi = MOV32ri64 32
+    PTILESTOREDV killed renamable $ax, killed renamable $cx, killed renamable $rdx, 1, killed renamable $rsi, 0, $noreg, killed renamable $tmm0
+    RETQ
+
+...

llvm/test/CodeGen/X86/AMX/amx-low-intrinsics-no-amx-bitcast.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -mtriple=x86_64 -lower-amx-intrinsics %s -S | FileCheck %s
+; RUN: opt -mtriple=x86_64 -lower-amx-intrinsics -enable-x86-scalar-amx=true %s -S | FileCheck %s
 
 define dso_local void @test_no_bitcast(i32* %A_mem, i32* %B_mem, i32* %C_mem) local_unnamed_addr #0 {
 ; CHECK-LABEL: @test_no_bitcast(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[C_MEM:%.*]] to i8*
 ; CHECK-NEXT:    br label [[TILELOAD_SCALARIZE_ROWS_HEADER:%.*]]
 ; CHECK:       tileload.scalarize.rows.header:
 ; CHECK-NEXT:    [[TILELOAD_SCALARIZE_ROWS_IV:%.*]] = phi i16 [ 0, [[ENTRY:%.*]] ], [ [[TILELOAD_SCALARIZE_ROWS_STEP:%.*]], [[TILELOAD_SCALARIZE_ROWS_LATCH:%.*]] ]

llvm/test/CodeGen/X86/AMX/amx-low-intrinsics.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -mtriple=x86_64 -lower-amx-intrinsics %s -S | FileCheck %s
+; RUN: opt -mtriple=x86_64 -lower-amx-intrinsics -enable-x86-scalar-amx=true %s -S | FileCheck %s
 
 define dso_local void @test_amx_load_non_O0(i16 signext %row, i16 signext %col, i8 *%ptr, i64 %stride, <256 x i32>* %vptr) {
 ; CHECK-LABEL: @test_amx_load_non_O0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = lshr i16 [[COL:%.*]], 2
 ; CHECK-NEXT:    [[TMP1:%.*]] = lshr i64 [[STRIDE:%.*]], 2
 ; CHECK-NEXT:    br label [[TILELOAD_SCALARIZE_ROWS_HEADER:%.*]]
 ; CHECK:       tileload.scalarize.rows.header:

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT: Assumption Cache Tracker
 ; CHECK-NEXT: Profile summary info
 ; CHECK-NEXT: Machine Branch Probability Analysis
 ; CHECK-NEXT:   ModulePass Manager
 ; CHECK-NEXT:     Pre-ISel Intrinsic Lowering
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Lower AMX intrinsics
+; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Lower AMX type for load/store
+; CHECK-NEXT:       Pre AMX Tile Config
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Lower Garbage Collection Instructions
 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Expand indirectbr instructions

 ; CHECK-NEXT:       Local Stack Slot Allocation
 ; CHECK-NEXT:       X86 speculative load hardening
 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       X86 EFLAGS copy lowering
 ; CHECK-NEXT:       X86 WinAlloca Expander
 ; CHECK-NEXT:       Eliminate PHI nodes for register allocation
 ; CHECK-NEXT:       Two-Address instruction pass
 ; CHECK-NEXT:       Fast Register Allocator
+; CHECK-NEXT:       Fast Tile Register Configure
 ; CHECK-NEXT:       X86 Lower Tile Copy
 ; CHECK-NEXT:       Bundle Machine CFG Edges
 ; CHECK-NEXT:       X86 FP Stackifier
 ; CHECK-NEXT:       Fixup Statepoint Caller Saved
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Optimization Remark Emitter
 ; CHECK-NEXT:       Prologue/Epilogue Insertion & Frame Finalization
 ; CHECK-NEXT:       Post-RA pseudo instruction expansion pass

llvm/test/CodeGen/X86/opt-pipeline.ll

 ; CHECK-NEXT: Profile summary info
 ; CHECK-NEXT: Create Garbage Collector Module Metadata
 ; CHECK-NEXT: Machine Branch Probability Analysis
 ; CHECK-NEXT:   ModulePass Manager
 ; CHECK-NEXT:     Pre-ISel Intrinsic Lowering
 ; CHECK-NEXT:     FunctionPass Manager
 ; CHECK-NEXT:       Expand Atomic instructions
 ; CHECK-NEXT:       Lower AMX intrinsics
+; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Lower AMX type for load/store
 ; CHECK-NEXT:       Module Verifier
-; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Canonicalize natural loops
 ; CHECK-NEXT:       Scalar Evolution Analysis
 ; CHECK-NEXT:       Loop Pass Manager
 ; CHECK-NEXT:         Canonicalize Freeze Instructions in Loops
 ; CHECK-NEXT:         Induction Variable Users
 ; CHECK-NEXT:         Loop Strength Reduction

llvm/tools/opt/opt.cpp

static bool shouldPinPassToLegacyPM(StringRef Pass) {
   if (llvm::is_contained(PassNameExactToIgnore, Pass))
     return false;
 
   std::vector<StringRef> PassNamePrefix = {
       "x86-",  "xcore-", "wasm-",    "systemz-", "ppc-",    "nvvm-",   "nvptx-",
       "mips-", "lanai-", "hexagon-", "bpf-",     "avr-",    "thumb2-", "arm-",
       "si-",   "gcn-",   "amdgpu-",  "aarch64-", "amdgcn-", "polly-"};
   std::vector<StringRef> PassNameContain = {"ehprepare"};
   std::vector<StringRef> PassNameExact = {
       "safe-stack",           "cost-model",
       "codegenprepare",       "interleaved-load-combine",
       "unreachableblockelim", "verify-safepoint-ir",
       "atomic-expand",
       "hardware-loops",       "type-promotion",
       "mve-tail-predication", "interleaved-access",
       "global-merge",         "pre-isel-intrinsic-lowering",
       "expand-reductions",    "indirectbr-expand",
       "generic-to-nvvm",      "expandmemcmp",
       "loop-reduce",          "lower-amx-type",
-      "lower-amx-intrinsics", "polyhedral-info",
-      "replace-with-veclib"};
+      "pre-amx-config",       "lower-amx-intrinsics",
+      "polyhedral-info",      "replace-with-veclib"};
   for (const auto &P : PassNamePrefix)
     if (Pass.startswith(P))
       return true;
   for (const auto &P : PassNameContain)
     if (Pass.contains(P))
       return true;
   return llvm::is_contained(PassNameExact, Pass);
 }

llvm/utils/gn/secondary/llvm/lib/Target/X86/BUILD.gn

sources = [
     "X86InstrInfo.cpp",
     "X86InstructionSelector.cpp",
     "X86InterleavedAccess.cpp",
     "X86LegalizerInfo.cpp",
     "X86LoadValueInjectionLoadHardening.cpp",
     "X86LoadValueInjectionRetHardening.cpp",
     "X86LowerAMXIntrinsics.cpp",
     "X86LowerAMXType.cpp",
+    "X86PreAMXConfig.cpp",
     "X86LowerTileCopy.cpp",
     "X86MCInstLower.cpp",
     "X86MachineFunctionInfo.cpp",
     "X86MacroFusion.cpp",
     "X86OptimizeLEAs.cpp",
     "X86PadShortFunction.cpp",
     "X86PartialReduction.cpp",
     "X86PreTileConfig.cpp",