Clzero intrinsic and its addition under znver1

[GGanesh]

Summary

This patch does the following.

1. Adds an Intrinsic int_x86_clzero which works with __builtin_ia32_clzero
2. Identifies clzero feature using cpuid info. (Function:8000_0008, Checks if EBX[0]=1)
3. Adds the clzero feature under znver1 architecture.
4. The custom inserter is added in Lowering.
5. A testcase is added to check the intrinsic.
6. The clzero instruction is added to assembler test.

[RKSimon]

Copyright notices in the test files are rare - do you need this?

test/CodeGen/X86/clzero.ll

36 ↗

(On Diff #86597)

Please check on i686 target triples as well as x86_64

test/MC/X86/x86-64.s

1508 ↗

(On Diff #86597)

Should we test for the clzero %rax / clzero %eax aliases as well?

[GGanesh]

Updated for the review comments

[RKSimon]

One minor but no other comments from me. @craig.topper ?

test/MC/X86/x86-32.s

450 ↗

(On Diff #87385)

Add clzero only test as well

// CHECK:       clzero
// CHECK:  encoding: [0x0f,0x01,0xfc]
                clzero

[GGanesh]

Updated the test file "x86-32.s" for clzero only test!

[RKSimon]

lib/Target/X86/X86InstrInfo.td

2461 ↗

(On Diff #87438)

Given that clzero writes to memory shouldn't it have a mayStore = 1 attribute? I'm not sure if WriteSystem alone creates the necessary barrier.

This needs testing - check if separate load+instruction and instruction+store don't get folded across the clzero?

[andreadb]

I have a (probably dumb) question: should CLZERO be treated as a memory read/write barrier for the purpose of scheduling? Is it okay to reoder CLFLUSH based on the EAX/RAX register dependency only?

I am asking this because it looks to me that we don't model the 'flush/zero' behavior of cache line operations on x86. For what I can see (please correct me if I am wrong), nothing in the code suggests that CLZERO (or even CLFLUSH) is treated as a memory barrier for scheduling purpose. Is that behavior intended (i.e. do we care about it)? Am I reading those tablegen definitions incorrectly?

[andreadb]

Ah, I see that Simon asked a similar question :-)

[craig.topper]

The instruction will default to having MCID::UnmodeledSideEffects set. Is that sufficient to protect the scheduling?

[RKSimon]

In D29385#670375, @craig.topper wrote:

The instruction will default to having MCID::UnmodeledSideEffects set. Is that sufficient to protect the scheduling?

In which case we just need a test to check for it

[GGanesh]

I think it is okay even if we don't set the mayStore attribute.
I wrote a simple test to check the following

1. Schedules based on the instruction attribute
2. Side-effect handling

As Craig Topper says it falls back the UnmodeledSideEffects case by default.

Test:

#include "xmmintrin.h"
int a[2];
int b[2];
int c,d, *p;
int main ()
{
  b[0] += a[0];
  __builtin_ia32_clzero (p) ;
  d += c;
  b[1] += a[0];
  return 0;
}

We can notice that the schedule without the intrinsic has two mov scheduled for the first cycle. Without the intrinsic they go back to safe-order.
Without clzero intrinsic: $>clang -O3 ClzeroFoldTest.c -S -march=znver1

# BB#0:                                 # %entry
        movl    a(%rip), %eax
        movl    c(%rip), %ecx
        addl    %eax, b(%rip)
        addl    %eax, b+4(%rip)
        addl    %ecx, d(%rip)
        xorl    %eax, %eax
        retq

With clzero intrinsic: $>clang -O3 ClzeroFoldTest.c -S -march=znver1

# BB#0:
        movl    a(%rip), %eax
        addl    %eax, b(%rip)
        movq    p(%rip), %rax
        leaq    (%rax), %rax
        clzero
        movl    c(%rip), %eax
        addl    %eax, d(%rip)
        movl    a(%rip), %eax
        addl    %eax, b+4(%rip)
        xorl    %eax, %eax
        retq

[andreadb]

In D29385#670375, @craig.topper wrote:

The instruction will default to having MCID::UnmodeledSideEffects set. Is that sufficient to protect the scheduling?

Yes, that would work.
An instruction with unmodeled side effects is basically treated as a scheduling barrier. So we are safe.

@craig.topper, is it fair to say that flag UnmodeledSideEffects is conservatively implicitly set because the tablegen definition for CLZEROr does not provide a primary instruction pattern, and flag hasSideEffects_Unset is true (i.e. hasSideEffects is not explicitly mentioned)?

[GGanesh]

@craig.topper If you are okay, can you please commit the changes on my behalf?

[craig.topper]

LGTM

[craig.topper]

I'll commit shortly.

llvm/trunk/include/llvm/IR/IntrinsicsX86.td

 let TargetPrefix = "x86" in {
   def int_x86_monitorx
       : GCCBuiltin<"__builtin_ia32_monitorx">,
         Intrinsic<[], [ llvm_ptr_ty, llvm_i32_ty, llvm_i32_ty ], []>;
   def int_x86_mwaitx
       : GCCBuiltin<"__builtin_ia32_mwaitx">,
         Intrinsic<[], [ llvm_i32_ty, llvm_i32_ty, llvm_i32_ty ], []>;
 }
+
+//===----------------------------------------------------------------------===//
+// Cache-line zero
+let TargetPrefix = "x86" in {
+  def int_x86_clzero : GCCBuiltin<"__builtin_ia32_clzero">,
+      Intrinsic<[], [llvm_ptr_ty], []>;
+}

llvm/trunk/lib/Support/Host.cpp

bool sys::getHostCPUFeatures(StringMap<bool> &Features) {
   Features["lzcnt"] = HasExtLeaf1 && ((ECX >> 5) & 1);
   Features["sse4a"] = HasExtLeaf1 && ((ECX >> 6) & 1);
   Features["prfchw"] = HasExtLeaf1 && ((ECX >> 8) & 1);
   Features["xop"] = HasExtLeaf1 && ((ECX >> 11) & 1) && HasAVXSave;
   Features["fma4"] = HasExtLeaf1 && ((ECX >> 16) & 1) && HasAVXSave;
   Features["tbm"] = HasExtLeaf1 && ((ECX >> 21) & 1);
   Features["mwaitx"] = HasExtLeaf1 && ((ECX >> 29) & 1);
 
+  bool HasExtLeaf8 = MaxExtLevel >= 0x80000008 &&
+                     !getX86CpuIDAndInfoEx(0x80000008,0x0, &EAX, &EBX, &ECX, &EDX);
+  Features["clzero"] = HasExtLeaf8 && ((EBX >> 0) & 1);
+
   bool HasLeaf7 =
       MaxLevel >= 7 && !getX86CpuIDAndInfoEx(0x7, 0x0, &EAX, &EBX, &ECX, &EDX);
 
   // AVX2 is only supported if we have the OS save support from AVX.
   Features["avx2"] = HasAVXSave && HasLeaf7 && ((EBX >> 5) & 1);
 
   Features["fsgsbase"] = HasLeaf7 && ((EBX >> 0) & 1);
   Features["sgx"] = HasLeaf7 && ((EBX >> 2) & 1);

llvm/trunk/lib/Target/X86/X86.td

 def FeaturePRFCHW  : SubtargetFeature<"prfchw", "HasPRFCHW", "true",
                                       "Support PRFCHW instructions">;
 def FeatureRDSEED  : SubtargetFeature<"rdseed", "HasRDSEED", "true",
                                       "Support RDSEED instruction">;
 def FeatureLAHFSAHF : SubtargetFeature<"sahf", "HasLAHFSAHF", "true",
                                        "Support LAHF and SAHF instructions">;
 def FeatureMWAITX  : SubtargetFeature<"mwaitx", "HasMWAITX", "true",
                                       "Enable MONITORX/MWAITX timer functionality">;
+def FeatureCLZERO  : SubtargetFeature<"clzero", "HasCLZERO", "true",
+                                      "Enable Cache Line Zero">;
 def FeatureMPX     : SubtargetFeature<"mpx", "HasMPX", "true",
                                       "Support MPX instructions">;
 def FeatureLEAForSP : SubtargetFeature<"lea-sp", "UseLeaForSP", "true",
                                      "Use LEA for adjusting the stack pointer">;
 def FeatureSlowDivide32 : SubtargetFeature<"idivl-to-divb",
                                      "HasSlowDivide32", "true",
                                      "Use 8-bit divide for positive values less than 256">;
 def FeatureSlowDivide64 : SubtargetFeature<"idivq-to-divl",

 // Zen
 def: ProcessorModel<"znver1", BtVer2Model, [
   FeatureADX,
   FeatureAES,
   FeatureAVX2,
   FeatureBMI,
   FeatureBMI2,
   FeatureCLFLUSHOPT,
+  FeatureCLZERO,
   FeatureCMPXCHG16B,
   FeatureF16C,
   FeatureFMA,
   FeatureFSGSBase,
   FeatureFXSR,
   FeatureFastLZCNT,
   FeatureLAHFSAHF,
   FeatureLZCNT,

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

static MachineBasicBlock *emitMonitor(MachineInstr &MI, MachineBasicBlock *BB,
 
   // The instruction doesn't actually take any operands though.
   BuildMI(*BB, MI, dl, TII->get(Opc));
 
   MI.eraseFromParent(); // The pseudo is gone now.
   return BB;
 }
 
+static MachineBasicBlock *emitClzero(MachineInstr *MI, MachineBasicBlock *BB,
+                                      const X86Subtarget &Subtarget) {
+  DebugLoc dl = MI->getDebugLoc();
+  const TargetInstrInfo *TII = Subtarget.getInstrInfo();
+  // Address into RAX/EAX
+  unsigned MemOpc = Subtarget.is64Bit() ? X86::LEA64r : X86::LEA32r;
+  unsigned MemReg = Subtarget.is64Bit() ? X86::RAX : X86::EAX;
+  MachineInstrBuilder MIB = BuildMI(*BB, MI, dl, TII->get(MemOpc), MemReg);
+  for (int i = 0; i < X86::AddrNumOperands; ++i)
+    MIB.add(MI->getOperand(i));
+
+  // The instruction doesn't actually take any operands though.
+  BuildMI(*BB, MI, dl, TII->get(X86::CLZEROr));
+
+  MI->eraseFromParent(); // The pseudo is gone now.
+  return BB;
+}
+
+
+
 MachineBasicBlock *
 X86TargetLowering::EmitVAARG64WithCustomInserter(MachineInstr &MI,
                                                  MachineBasicBlock *MBB) const {
   // Emit va_arg instruction on X86-64.
 
   // Operands to this pseudo-instruction:
   // 0  ) Output        : destination address (reg)
   // 1-5) Input         : va_list address (addr, i64mem)
assert(Subtarget.hasSSE42() &&
            "Target must have SSE4.2 or AVX features enabled");
     return emitPCMPSTRI(MI, BB, Subtarget.getInstrInfo());
 
   // Thread synchronization.
   case X86::MONITOR:
     return emitMonitor(MI, BB, Subtarget, X86::MONITORrrr);
   case X86::MONITORX:
     return emitMonitor(MI, BB, Subtarget, X86::MONITORXrrr);
+
+  // Cache line zero
+  case X86::CLZERO:
+    return emitClzero(&MI, BB, Subtarget);
+
   // PKU feature
   case X86::WRPKRU:
     return emitWRPKRU(MI, BB, Subtarget);
   case X86::RDPKRU:
     return emitRDPKRU(MI, BB, Subtarget);
   // xbegin
   case X86::XBEGIN:
     return emitXBegin(MI, BB, Subtarget.getInstrInfo());

llvm/trunk/lib/Target/X86/X86InstrInfo.td

 def HasTSX       : Predicate<"Subtarget->hasRTM() || Subtarget->hasHLE()">;
 def HasADX       : Predicate<"Subtarget->hasADX()">;
 def HasSHA       : Predicate<"Subtarget->hasSHA()">;
 def HasPRFCHW    : Predicate<"Subtarget->hasPRFCHW()">;
 def HasRDSEED    : Predicate<"Subtarget->hasRDSEED()">;
 def HasPrefetchW : Predicate<"Subtarget->hasPRFCHW()">;
 def HasLAHFSAHF  : Predicate<"Subtarget->hasLAHFSAHF()">;
 def HasMWAITX    : Predicate<"Subtarget->hasMWAITX()">;
+def HasCLZERO    : Predicate<"Subtarget->hasCLZERO()">;
 def FPStackf32   : Predicate<"!Subtarget->hasSSE1()">;
 def FPStackf64   : Predicate<"!Subtarget->hasSSE2()">;
 def HasMPX       : Predicate<"Subtarget->hasMPX()">;
 def HasCLFLUSHOPT : Predicate<"Subtarget->hasCLFLUSHOPT()">;
 def HasCmpxchg16b: Predicate<"Subtarget->hasCmpxchg16b()">;
 def Not64BitMode : Predicate<"!Subtarget->is64Bit()">,
                              AssemblerPredicate<"!Mode64Bit", "Not 64-bit mode">;
 def In64BitMode  : Predicate<"Subtarget->is64Bit()">,

 def : InstAlias<"monitorx\t{%eax, %ecx, %edx|edx, ecx, eax}", (MONITORXrrr)>,
       Requires<[ Not64BitMode ]>;
 def : InstAlias<"monitorx\t{%rax, %rcx, %rdx|rdx, rcx, rax}", (MONITORXrrr)>,
       Requires<[ In64BitMode ]>;
 
 //===----------------------------------------------------------------------===//
 // CLZERO Instruction
 //
+let SchedRW = [WriteSystem] in {
   let Uses = [EAX] in
-def CLZEROr : I<0x01, MRM_FC, (outs), (ins), "clzero", []>, TB;
+  def CLZEROr : I<0x01, MRM_FC, (outs), (ins), "clzero", [], IIC_SSE_CLZERO>,
+                TB, Requires<[HasCLZERO]>;
+
+  let usesCustomInserter = 1 in {
+  def CLZERO : PseudoI<(outs), (ins i32mem:$src1),
+                       [(int_x86_clzero addr:$src1)]>, Requires<[HasCLZERO]>;
+  }
+} // SchedRW
+
+def : InstAlias<"clzero\t{%eax|eax}", (CLZEROr)>, Requires<[Not64BitMode]>;
+def : InstAlias<"clzero\t{%rax|rax}", (CLZEROr)>, Requires<[In64BitMode]>;
 
 //===----------------------------------------------------------------------===//
 // Pattern fragments to auto generate TBM instructions.
 //===----------------------------------------------------------------------===//
 
 let Predicates = [HasTBM] in {
   def : Pat<(X86bextr GR32:$src1, (i32 imm:$src2)),
             (BEXTRI32ri GR32:$src1, imm:$src2)>;

llvm/trunk/lib/Target/X86/X86Schedule.td

 def IIC_SSE_PMADD : InstrItinClass;
 def IIC_SSE_PMULHRSW : InstrItinClass;
 def IIC_SSE_PALIGNRR : InstrItinClass;
 def IIC_SSE_PALIGNRM : InstrItinClass;
 def IIC_SSE_MWAIT : InstrItinClass;
 def IIC_SSE_MONITOR : InstrItinClass;
 def IIC_SSE_MWAITX : InstrItinClass;
 def IIC_SSE_MONITORX : InstrItinClass;
+def IIC_SSE_CLZERO : InstrItinClass;
 
 def IIC_SSE_PREFETCH : InstrItinClass;
 def IIC_SSE_PAUSE : InstrItinClass;
 def IIC_SSE_LFENCE : InstrItinClass;
 def IIC_SSE_MFENCE : InstrItinClass;
 def IIC_SSE_SFENCE : InstrItinClass;
 def IIC_SSE_LDMXCSR : InstrItinClass;
 def IIC_SSE_STMXCSR : InstrItinClass;

llvm/trunk/lib/Target/X86/X86Subtarget.h

protected:
   bool HasRDSEED;
 
   /// Processor has LAHF/SAHF instructions.
   bool HasLAHFSAHF;
 
   /// Processor has MONITORX/MWAITX instructions.
   bool HasMWAITX;
 
+  /// Processor has Cache Line Zero instruction
+  bool HasCLZERO;
+
   /// Processor has Prefetch with intent to Write instruction
   bool HasPFPREFETCHWT1;
 
   /// True if BT (bit test) of memory instructions are slow.
   bool IsBTMemSlow;
 
   /// True if SHLD instructions are slow.
   bool IsSHLDSlow;
public:
   bool hasRTM() const { return HasRTM; }
   bool hasHLE() const { return HasHLE; }
   bool hasADX() const { return HasADX; }
   bool hasSHA() const { return HasSHA; }
   bool hasPRFCHW() const { return HasPRFCHW; }
   bool hasRDSEED() const { return HasRDSEED; }
   bool hasLAHFSAHF() const { return HasLAHFSAHF; }
   bool hasMWAITX() const { return HasMWAITX; }
+  bool hasCLZERO() const { return HasCLZERO; }
   bool isBTMemSlow() const { return IsBTMemSlow; }
   bool isSHLDSlow() const { return IsSHLDSlow; }
   bool isPMULLDSlow() const { return IsPMULLDSlow; }
   bool isUnalignedMem16Slow() const { return IsUAMem16Slow; }
   bool isUnalignedMem32Slow() const { return IsUAMem32Slow; }
   bool hasSSEUnalignedMem() const { return HasSSEUnalignedMem; }
   bool hasCmpxchg16b() const { return HasCmpxchg16b; }
   bool useLeaForSP() const { return UseLeaForSP; }

llvm/trunk/lib/Target/X86/X86Subtarget.cpp

void X86Subtarget::initializeEnvironment() {
   HasVLX = false;
   HasADX = false;
   HasPKU = false;
   HasSHA = false;
   HasPRFCHW = false;
   HasRDSEED = false;
   HasLAHFSAHF = false;
   HasMWAITX = false;
+  HasCLZERO = false;
   HasMPX = false;
   IsBTMemSlow = false;
   IsPMULLDSlow = false;
   IsSHLDSlow = false;
   IsUAMem16Slow = false;
   IsUAMem32Slow = false;
   HasSSEUnalignedMem = false;
   HasCmpxchg16b = false;

llvm/trunk/test/CodeGen/X86/clzero.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-linux -mattr=+clzero | FileCheck %s --check-prefix=X64
+; RUN: llc < %s -mtriple=i386-pc-linux -mattr=+clzero | FileCheck %s --check-prefix=X32
+
+define void @foo(i8* %p) #0 {
+; X64-LABEL: foo:
+; X64:       # BB#0: # %entry
+; X64-NEXT:    leaq (%rdi), %rax
+; X64-NEXT:    clzero
+; X64-NEXT:    retq
+;
+; X32-LABEL: foo:
+; X32:       # BB#0: # %entry
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X32-NEXT:    leal (%eax), %eax
+; X32-NEXT:    clzero
+; X32-NEXT:    retl
+entry:
+  tail call void @llvm.x86.clzero(i8* %p) #1
+  ret void
+}
+
+declare void @llvm.x86.clzero(i8*) #1

llvm/trunk/test/MC/Disassembler/X86/x86-32.txt

 0x0f 0x01 0xdd
 
 # CHECK: skinit
 0x0f 0x01 0xde
 
 # CHECK: invlpga
 0x0f 0x01 0xdf
 
+# CHECK: clzero
+0x0f,0x01,0xfc
+
 # CHECK: movl $0, -4(%ebp)
 0xc7 0x45 0xfc 0x00 0x00 0x00 0x00
 
 # CHECK: movl %cr0, %ecx
 0x0f 0x20 0xc1
 
 # CHECK: leal 4(%esp), %ecx
 0x8d 0x4c 0x24 0x04 

llvm/trunk/test/MC/X86/x86-32.s

 // CHECK: movl  %dr6, %eax
 // CHECK:  encoding: [0x0f,0x21,0xf0]
         movl %dr6,%eax
 
 // CHECK: movl  %dr7, %eax
 // CHECK:  encoding: [0x0f,0x21,0xf8]
         movl %dr7,%eax
 
+// CHECK:       clzero
+// CHECK:  encoding: [0x0f,0x01,0xfc]
+                clzero
+
+// CHECK:       clzero
+// CHECK:  encoding: [0x0f,0x01,0xfc]
+                clzero %eax
+
 // radr://8017522
 // CHECK: wait
 // CHECK:  encoding: [0x9b]
   fwait
 
 // rdar://7873482
 // CHECK: [0x65,0xa1,0x7c,0x00,0x00,0x00]
         movl  %gs:124, %eax

llvm/trunk/test/MC/X86/x86-64.s

 // CHECK:   mwaitx
 // CHECK:  encoding: [0x0f,0x01,0xfb]
           mwaitx
 
 // CHECK:   mwaitx
 // CHECK:  encoding: [0x0f,0x01,0xfb]
           mwaitx %rax, %rcx, %rbx
 
+// CHECK:       clzero
+// CHECK:  encoding: [0x0f,0x01,0xfc]
+                clzero
+
+// CHECK:       clzero
+// CHECK:  encoding: [0x0f,0x01,0xfc]
+                clzero %rax
+
 // CHECK:   movl %r15d, (%r15,%r15)
 // CHECK:  encoding: [0x47,0x89,0x3c,0x3f]
 movl %r15d, (%r15,%r15)