[X86] Recognize constant arrays with special values and replace loads from it with subtract and shift instructions, which then will be replaced by X86 BZHI machine instruction.

[aymanmus]

Summary

Recognize constant arrays with the following values:

0x0, 0x1, 0x3, 0x7, 0xF, 0x1F, .... , 2^(size - 1) -1

where size is the size of the array.

the result of a load with index idx from this array is equivalent to the result of the following:

(0xFFFFFFFF >> (sub 32, idx))             (assuming the array of type 32-bit integer).

And the result of an 'AND' operation on the returned value of such a load and another input, is exactly equivalent to the X86 BZHI instruction behavior.

See test cases in the LIT test for better understanding.

[aymanmus]

Apply clang-format on changes

[delena]

I suggest to put this transformation in a separate pass. The pass should be added to X86 IR passes ,see X86PassConfig::addIRPasses().
Other targets, interested in this pass, may add it as well.

[aymanmus]

Moving the transformation to an independent pass, which currently runs only on X86, as suggested by @delena.

[RKSimon]

While I don't want to end up forcing you to over-engineer this, it seems this pass is currently very specific and should be generalised to make it straightforward to work for constants arrays of other patterns that are easily re-materializable (upper/lower masks, single bit masks, etc.) - "SubstituteLoadWithRematerializationPass", possibly driven by the TTI cost models?

lib/CodeGen/SubstituteLoadWithSubShr.cpp

70

(style) Don't nest these - do an early-out if they are null

146

(style) Remove all the extra braces.

[delena]

In D34141#786514, @RKSimon wrote:

While I don't want to end up forcing you to over-engineer this, it seems this pass is currently very specific and should be generalised to make it straightforward to work for constants arrays of other patterns that are easily re-materializable (upper/lower masks, single bit masks, etc.) - "SubstituteLoadWithRematerializationPass", possibly driven by the TTI cost models?

I agree that the name of the pass should be more generic.
Do you mean renaming by "generalization" ? Or you propose to add this transformation for all targets and ask TTI about profitability of this transformation?

[RKSimon]

In D34141#786536, @delena wrote:

In D34141#786514, @RKSimon wrote:

While I don't want to end up forcing you to over-engineer this, it seems this pass is currently very specific and should be generalised to make it straightforward to work for constants arrays of other patterns that are easily re-materializable (upper/lower masks, single bit masks, etc.) - "SubstituteLoadWithRematerializationPass", possibly driven by the TTI cost models?

I agree that the name of the pass should be more generic.
Do you mean renaming by "generalization" ? Or you propose to add this transformation for all targets and ask TTI about profitability of this transformation?

By generalization I mean setup the pass so that other patterns can be easily added in the future (e.g. 0xFF, 0xFE, ..., 0x80, 0x00 high 'sign' masks are common too).

I think its OK to keep this as X86 only initially, but it shouldn't be BMI2 only - checking the cost of a sub + shift compared to a constant pool load should work alright (even better if PR31274 ever gets addressed).

include/llvm/CodeGen/Passes.h

423

Comment describing the pass.

lib/CodeGen/SubstituteLoadWithSubShr.cpp

86

dyn_cast<> can fail - add a check for null:

if (!Elem || Elem->getZExtValue() != (((uint64_t)1 << i) - 1))

102

Duplicate code - pull out

unsigned EltSizeInBits = ElemTy->getScalarSizeInBits();
APInt ElemSize = APInt(EltSizeInBits, EltSizeInBits);

test/CodeGen/X86/replace-load-with-sub-shr.ll

2

Test without bmi2 as well - the pass currently runs on that too.

[aymanmus]

Total redesign to the previous solution.
moved the transformation to a more "natural" area.

[aymanmus]

Ping.

[aymanmus]

Kind ping #2

[aymanmus]

Ping #3!

[delena]

Please submit the same test cases before optimization.

[m_zuckerman]

lib/Target/X86/X86ISelLowering.cpp

32498 ↗

(On Diff #119825)

align the line

32521 ↗

(On Diff #119825)

Can you explain why the element must to be a global variable?

32526 ↗

(On Diff #119825)

1. please clarify this if (add comments)
2. Why you need also the isa<ConstantDataArray>(Init) ,is it not the same as GV->isConstant()?

32548 ↗

(On Diff #119825)

What about 64 bit?

32552 ↗

(On Diff #119825)

You can write
SDValue Inp = Node->getOperand(1-i);

[aymanmus]

lib/Target/X86/X86ISelLowering.cpp

32521 ↗

(On Diff #119825)

Clang will always hoist a local constant array to be global in LLVM IR.

32526 ↗

(On Diff #119825)

The second condition makes sure that it is a constant data *array*. GV->isConstant does not.

32548 ↗

(On Diff #119825)

It's only an example for the 32-bit case, the same happens for 64 as well.

[m_zuckerman]

LGTM with no one has nothing to add !!!

include/llvm/CodeGen/Passes.h

/// MachineDominanaceFrontier - This pass is a machine dominators analysis pass.
   /// This pass performs outlining on machine instructions directly before
   /// printing assembly.
   ModulePass *createMachineOutlinerPass();
 
   /// This pass expands the experimental reduction intrinsics into sequences of
   /// shuffles.
   FunctionPass *createExpandReductionsPass();
 
+  FunctionPass *createSubstituteLoadWithSubShrPass();
+
 } // End llvm namespace
 
 #endif

include/llvm/InitializePasses.h

 void initializeStripDeadDebugInfoPass(PassRegistry&);
 void initializeStripDeadPrototypesLegacyPassPass(PassRegistry&);
 void initializeStripDebugDeclarePass(PassRegistry&);
 void initializeStripGCRelocatesPass(PassRegistry&);
 void initializeStripNonDebugSymbolsPass(PassRegistry&);
 void initializeStripNonLineTableDebugInfoPass(PassRegistry&);
 void initializeStripSymbolsPass(PassRegistry&);
 void initializeStructurizeCFGPass(PassRegistry&);
+void initializeSubstituteLoadWithSubShrPass(PassRegistry &);
 void initializeTailCallElimPass(PassRegistry&);
 void initializeTailDuplicatePassPass(PassRegistry&);
 void initializeTargetLibraryInfoWrapperPassPass(PassRegistry&);
 void initializeTargetPassConfigPass(PassRegistry&);
 void initializeTargetTransformInfoWrapperPassPass(PassRegistry&);
 void initializeThreadSanitizerPass(PassRegistry&);
 void initializeTwoAddressInstructionPassPass(PassRegistry&);
 void initializeTypeBasedAAWrapperPassPass(PassRegistry&);

lib/CodeGen/CMakeLists.txt

add_llvm_library(LLVMCodeGen
   SlotIndexes.cpp
   SpillPlacement.cpp
   SplitKit.cpp
   StackColoring.cpp
   StackMapLivenessAnalysis.cpp
   StackMaps.cpp
   StackProtector.cpp
   StackSlotColoring.cpp
+  SubstituteLoadWithSubShr.cpp
   TailDuplication.cpp
   TailDuplicator.cpp
   TargetFrameLoweringImpl.cpp
   TargetInstrInfo.cpp
   TargetLoweringBase.cpp
   TargetLoweringObjectFileImpl.cpp
   TargetOptionsImpl.cpp
   TargetPassConfig.cpp

lib/CodeGen/SubstituteLoadWithSubShr.cpp

+//===-------------------- SubstituteLoadWithSubShr.cpp --------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass recognizes cases where a load can be replaced by subtract and
+// shift right instructions.
+//
+// In case of loading integer value from an array of constants which is defined
+// as follows:
+//
+//   int array[SIZE] = {0x0, 0x1, 0x3, 0x7, 0xF ..., 2^(SIZE-1) - 1}
+//
+// array[idx] can be replaced with (0xFFFFFFFF >> (sub 32, idx))
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/CodeGen/Passes.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/Transforms/Utils/Local.h"
+#include <iostream>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "substitute-load-with-sub-shr"
+
+namespace {
+
+class SubstituteLoadWithSubShr : public FunctionPass {
+
+public:
+  static char ID;
+  SubstituteLoadWithSubShr() : FunctionPass(ID) {
+    initializeSubstituteLoadWithSubShrPass(*PassRegistry::getPassRegistry());
+  }
+
+  StringRef getPassName() const override {
+    return "Substitute Load With Sub Shr";
+  }
+
+  bool runOnFunction(Function &F) override;
+
+private:
+  bool
+  ReplaceLoadWithSubShr(LoadInst *LI, SmallVector<Instruction *, 32> &DeadInsts,
+                        SmallVector<GlobalVariable *, 32> &PossibleDeadArrays);
+};
+} // end anonymous namespace.
+
+char SubstituteLoadWithSubShr::ID = 0;
+INITIALIZE_PASS_BEGIN(SubstituteLoadWithSubShr, DEBUG_TYPE,
+                      "SubstituteLoadWithSubShr", false, false)
+INITIALIZE_PASS_END(SubstituteLoadWithSubShr, DEBUG_TYPE,
+                    "SubstituteLoadWithSubShr", false, false)
+
+FunctionPass *llvm::createSubstituteLoadWithSubShrPass() {
+  return new SubstituteLoadWithSubShr();
+}
+
+bool SubstituteLoadWithSubShr::ReplaceLoadWithSubShr(
+    LoadInst *LI, SmallVector<Instruction *, 32> &DeadInsts,
+    SmallVector<GlobalVariable *, 32> &PossibleDeadArrays) {
+
+  if (GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(LI->getOperand(0))) {
+    if (GlobalVariable *GV = dyn_cast<GlobalVariable>(GEP->getOperand(0))) {
+      if (GV->isConstant() && GV->hasDefinitiveInitializer()) {
+
+        Constant *Init = GV->getInitializer();
+        Type *Ty = Init->getType();
+        if ((!isa<ConstantArray>(Init) && !isa<ConstantDataArray>(Init)) ||
+            !Ty->getArrayElementType()->isIntegerTy() ||
+            Ty->getArrayNumElements() >
+                Ty->getArrayElementType()->getScalarSizeInBits())
+          return nullptr;
+
+        // Check if the array's constant elements are suitable to our case.
+        uint64_t ArrayElementCount = Init->getType()->getArrayNumElements();
+        for (uint64_t i = 0; i < ArrayElementCount; i++) {
+          ConstantInt *Elem =
+              dyn_cast<ConstantInt>(Init->getAggregateElement(i));
+          if (Elem->getZExtValue() != (((uint64_t)1 << i) - 1))
+            return nullptr;
+        }
+
+        IRBuilder<> Builder(LI->getContext());
+        Builder.SetInsertPoint(LI);
+
+        // Do the transformation (assuming 32-bit elements):
+        // -> elemPtr = getelementptr @array, 0, idx
+        //    elem = load elemPtr
+        // <- hiBit = 32 - idx
+        //    elem = 0xFFFFFFFF >> hiBit
+        Type *ElemTy = Ty->getArrayElementType();
+        Value *LoadIdx = Builder.CreateZExtOrTrunc(GEP->getOperand(2), ElemTy);
+
+        APInt ElemSize =
+            APInt(ElemTy->getScalarSizeInBits(), ElemTy->getScalarSizeInBits());
+        Constant *ElemSizeConst = Constant::getIntegerValue(ElemTy, ElemSize);
+        Value *Sub = Builder.CreateSub(ElemSizeConst, LoadIdx);
+
+        Constant *AllOnes = Constant::getAllOnesValue(ElemTy);
+        Value *LShr = Builder.CreateLShr(AllOnes, Sub);
+
+        LI->replaceAllUsesWith(LShr);
+        DeadInsts.push_back(LI);
+
+        // Insert the array to mark it as potentially dead.
+        if (find(PossibleDeadArrays, GV) == PossibleDeadArrays.end())
+          PossibleDeadArrays.push_back(GV);
+
+        return true;
+      }
+    }
+  }
+  return false;
+}
+
+bool SubstituteLoadWithSubShr::runOnFunction(Function &F) {
+  SmallVector<Instruction *, 32> DeadInsts;
+  SmallVector<GlobalVariable *, 32> PossibleDeadArrays;
+  bool Changed = false;
+
+  // Perform the transformation if possible while keep tracking of dead
+  // instructions and potential dead arrays.
+  for (auto &I : instructions(F)) {
+    if (LoadInst *LI = dyn_cast<LoadInst>(&I))
+      Changed |= ReplaceLoadWithSubShr(LI, DeadInsts, PossibleDeadArrays);
+  }
+
+  if (Changed) {
+    for (auto &I : DeadInsts) {
+      RecursivelyDeleteTriviallyDeadInstructions(I);
+    }
+
+    for (auto &GV : PossibleDeadArrays) {
+      if (GV->use_empty() && GV->isDiscardableIfUnused())
+        GV->eraseFromParent();
+    }
+  }
+
+  return Changed;
+}

lib/Target/X86/X86TargetMachine.cpp

TargetPassConfig *X86TargetMachine::createPassConfig(PassManagerBase &PM) {
   return new X86PassConfig(*this, PM);
 }
 
 void X86PassConfig::addIRPasses() {
   addPass(createAtomicExpandPass());
 
   TargetPassConfig::addIRPasses();
 
-  if (TM->getOptLevel() != CodeGenOpt::None)
+  if (TM->getOptLevel() != CodeGenOpt::None) {
     addPass(createInterleavedAccessPass());
+    addPass(createSubstituteLoadWithSubShrPass());
+  }
 }
 
 bool X86PassConfig::addInstSelector() {
   // Install an instruction selector.
   addPass(createX86ISelDag(getX86TargetMachine(), getOptLevel()));
 
   // For ELF, cleanup any local-dynamic TLS accesses.
   if (TM->getTargetTriple().isOSBinFormatELF() &&

test/CodeGen/X86/replace-load-with-sub-shr.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+bmi2 | FileCheck %s
+
+@fill_table32 = internal unnamed_addr constant [32 x i32] [i32 0, i32 1, i32 3, i32 7, i32 15, i32 31, i32 63, i32 127, i32 255, i32 511, i32 1023, i32 2047, i32 4095, i32 8191, i32 16383, i32 32767, i32 65535, i32 131071, i32 262143, i32 524287, i32 1048575, i32 2097151, i32 4194303, i32 8388607, i32 16777215, i32 33554431, i32 67108863, i32 134217727, i32 268435455, i32 536870911, i32 1073741823, i32 2147483647], align 16
+@fill_table64 = internal unnamed_addr constant [64 x i64] [i64 0, i64 1, i64 3, i64 7, i64 15, i64 31, i64 63, i64 127, i64 255, i64 511, i64 1023, i64 2047, i64 4095, i64 8191, i64 16383, i64 32767, i64 65535, i64 131071, i64 262143, i64 524287, i64 1048575, i64 2097151, i64 4194303, i64 8388607, i64 16777215, i64 33554431, i64 67108863, i64 134217727, i64 268435455, i64 536870911, i64 1073741823, i64 2147483647, i64 4294967295, i64 8589934591, i64 17179869183, i64 34359738367, i64 68719476735, i64 137438953471, i64 274877906943, i64 549755813887, i64 1099511627775, i64 2199023255551, i64 4398046511103, i64 8796093022207, i64 17592186044415, i64 35184372088831, i64 70368744177663, i64 140737488355327, i64 281474976710655, i64 562949953421311, i64 1125899906842623, i64 2251799813685247, i64 4503599627370495, i64 9007199254740991, i64 18014398509481983, i64 36028797018963967, i64 72057594037927935, i64 144115188075855871, i64 288230376151711743, i64 576460752303423487, i64 1152921504606846975, i64 2305843009213693951, i64 4611686018427387903, i64 9223372036854775807], align 16
+
+define i32 @f32(i32 %y) local_unnamed_addr {
+; CHECK-LABEL: f32:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    movl $32, %eax
+; CHECK-NEXT:    subl %edi, %eax
+; CHECK-NEXT:    movl $-1, %ecx
+; CHECK-NEXT:    shrxl %eax, %ecx, %eax
+; CHECK-NEXT:    retq
+entry:
+  %idxprom = sext i32 %y to i64
+  %arrayidx = getelementptr inbounds [32 x i32], [32 x i32]* @fill_table32, i64 0, i64 %idxprom
+  %0 = load i32, i32* %arrayidx, align 4
+  ret i32 %0
+}
+
+define i64 @f64(i64 %y) local_unnamed_addr {
+; CHECK-LABEL: f64:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    movl $64, %eax
+; CHECK-NEXT:    subl %edi, %eax
+; CHECK-NEXT:    movq $-1, %rcx
+; CHECK-NEXT:    shrxq %rax, %rcx, %rax
+; CHECK-NEXT:    retq
+entry:
+  %arrayidx = getelementptr inbounds [64 x i64], [64 x i64]* @fill_table64, i64 0, i64 %y
+  %0 = load i64, i64* %arrayidx, align 8
+  ret i64 %0
+}
+
+define i32 @f32_bzhi(i32 %x, i32 %y) local_unnamed_addr {
+; CHECK-LABEL: f32_bzhi:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    bzhil %esi, %edi, %eax
+; CHECK-NEXT:    retq
+entry:
+  %idxprom = sext i32 %y to i64
+  %arrayidx = getelementptr inbounds [32 x i32], [32 x i32]* @fill_table32, i64 0, i64 %idxprom
+  %0 = load i32, i32* %arrayidx, align 4
+  %and = and i32 %0, %x
+  ret i32 %and
+}
+
+define i64 @f64_bzhi(i64 %x, i64 %y) local_unnamed_addr {
+; CHECK-LABEL: f64_bzhi:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    bzhiq %rsi, %rdi, %rax
+; CHECK-NEXT:    retq
+entry:
+  %arrayidx = getelementptr inbounds [64 x i64], [64 x i64]* @fill_table64, i64 0, i64 %y
+  %0 = load i64, i64* %arrayidx, align 8
+  %and = and i64 %0, %x
+  ret i64 %and
+}
+
+; CHECK-NOT: fill_table32:
+; CHECK-NOT: fill_table64:
+