[X86] Use bit test instructions to optimize some logic atomic operations

[pengfei]

Summary

This is to match GCC's optimizations: https://gcc.godbolt.org/z/3odh9e7WE

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

30332

Probably want to make sure I is in the same BB given the SelectionDAG restrictions?

30341

Is this the same as ~C1->getValue() == C2->getValue()?

31115

Why does it matter if the AND has more than one use? Or were you trying to check that the ATOMIC_LOAD has only one use of the Result 0?

31151

Use the getSETCC function from this file.

31153

Don't you need getZExtOrTrunc here? You aren't guaranteed to shift out the extended bits so you need to make sure they are 0 like the and would have done.

llvm/test/CodeGen/X86/atomic-bit-test.ll

292

Was -1 supposed to be -2?

332

Was -2 supposed to be -3?

[pengfei]

Great review! Thanks Craig.

llvm/lib/Target/X86/X86ISelLowering.cpp

30332

Yes, good catch!

30341

Good idea!

31115

Yes, good catch!

31151

Good idea!

31153

Yes. I misunderstood of AnyExt. Good catch!

llvm/test/CodeGen/X86/atomic-bit-test.ll

292

Good catch!

332

Good catch!

[pengfei]

ping~

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

31078

Verify that operand 1 is a constant too.

31079

My big concern is that we rely on the checks done in IR holding until SelectionDAG. If that fails and a user hits this fatal error, how would they know how to fix their code?

Did you give any thought about converting the patterns into a target specific intrinsic in IR? That would probably need to be a new hook in the AtomicExpandPass or a separate X86 pass.

@RKSimon or @spatel what are your thoughts?

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

31078

This is not used to check constantency but to check the code is entered once only (C2 hasn't been assigned to a constant value).
We check the constantency through cast<ConstantSDNode>

31079

Did you give any thought about converting the patterns into a target specific intrinsic in IR? That would probably need to be a new hook in the AtomicExpandPass or a separate X86 pass.

I'm not sure if we can simply use target specific intrinsic here. Atomic instruction has attributes like ordering that need special handling.

My big concern is that we rely on the checks done in IR holding until SelectionDAG. If that fails and a user hits this fatal error, how would they know how to fix their code?

We mainly check 3 aspects here:

1. One use, constantency and consistency of the constants.
   • I don't think they will be changed given there are no room to optimize them.
2. AND operation.
   • It won't be changed either. The reason is the same. But we might have chance to roll back the AND if it changed.
3. AND operation been splitted to a different BB.
   • There may be chance to happen in reality. We can always hoise it by a separate pass before SelectionDAG. I'd like to leave it till it happens.

What do you think?

[craig.topper]

This test triggers the fatal error

#include <atomic>

std::atomic<int> x;

void bar();
void baz();

void foo(int a, int b) {
  int y = std::atomic_fetch_or(&x, 8);
  if (y & 8 && a)
    bar();
  if ((y & 8) && b)
    baz();
}

[pengfei]

Break AND + CMP 0 fold for atomic RMW logic instructions. This fixes the problem in Craig's example. Thanks Craig!

[craig.topper]

This test fails

%"struct.std::atomic" = type { %"struct.std::__atomic_base" }                    
%"struct.std::__atomic_base" = type { i32 }                                      
                                                                                 
@x = global %"struct.std::atomic" zeroinitializer, align 4                       
                                                                                 
define i32 @_Z3fooii(i32 %0, i32 %1) {                                           
  %3 = atomicrmw or i32* getelementptr inbounds (%"struct.std::atomic", %"struct.std::atomic"* @x, i64 0, i32 0, i32 0), i32 8 seq_cst, align 4
  %4 = tail call i32 @llvm.ctlz.i32(i32 %0, i1 false)                            
  %5 = and i32 %3, 8                                                             
  %6 = icmp ne i32 %5, 0                                                         
  %7 = icmp ne i32 %1, 0                                                         
  %8 = select i1 %6, i1 %7, i1 false                                             
  br i1 %8, label %9, label %10                                                  
                                                                                 
9:                                                ; preds = %2                   
  tail call void @_Z3bazi(i32 %4)                                                
  br label %10                                                                   
                                                                                 
10:                                               ; preds = %9, %2               
  ret i32 %4                                                                     
}                                                                                
                                                                                 
declare i32 @llvm.ctlz.i32(i32, i1 immarg)                                       
                                                                                 
declare void @_Z3bazi(i32)

CodeGenPrepare splits the basic block to despeculate the ctlz.

[pengfei]

Add a new pass to hosit AND instructions that been split by other passes.
This fixes the failed example. Thanks a lot for Craig's helps!

[craig.topper]

llvm/lib/Target/X86/X86AtomicANDHoist.cpp

62

It might not be safe to move the And if the other operand isn't a constant. There's no guarantee that the atomic instruction occurs after the source of the other operand.

[pengfei]

Add constant check for AND operand.

llvm/lib/Target/X86/X86AtomicANDHoist.cpp

62

Makes sense. Thank you!

[pengfei]

ping~

[RKSimon]

pre-commit the tests and rebase so this patch shows the codegen diff?

llvm/lib/Target/X86/X86ISelLowering.h

1647

Log?

[RKSimon]

reverse ping?

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.h

1647

I assumed it meant Logic

[pengfei]

Rename Log to Logic;
Pre-commit the tests and rebased on it;

[pengfei]

Ping

[craig.topper]

This breaks -O0. Fast isel consumes the And, then we fall back to SelectionDAG, but the And is gone so it triggers the fatal error.

I'm uncomfortable with the cross pass requirements this approach depends on. It seems very fragile.

[pengfei]

In D116375#3318318, @craig.topper wrote:

This breaks -O0. Fast isel consumes the And, then we fall back to SelectionDAG, but the And is gone so it triggers the fatal error.

I'm uncomfortable with the cross pass requirements this approach depends on. It seems very fragile.

Thanks Craig! An alternative implementation D120199.

llvm/lib/Target/X86/CMakeLists.txt

 if (X86_GEN_FOLD_TABLES)
   tablegen(LLVM X86GenFoldTables.inc -gen-x86-fold-tables)
 endif()
 
 add_public_tablegen_target(X86CommonTableGen)
 
 set(sources
   X86AsmPrinter.cpp
+  X86AtomicANDHoist.cpp
   X86AvoidTrailingCall.cpp
   X86CallFrameOptimization.cpp
   X86CallingConv.cpp
   X86CallLowering.cpp
   X86CmovConversion.cpp
   X86DomainReassignment.cpp
   X86DiscriminateMemOps.cpp
   X86LowerTileCopy.cpp

llvm/lib/Target/X86/X86.h

 /// fp exceptions when strict-fp enabled.
 FunctionPass *createX86InsertX87waitPass();
 
 /// This pass optimizes arithmetic based on knowledge that is only used by
 /// a reduction sequence and is therefore safe to reassociate in interesting
 /// ways.
 FunctionPass *createX86PartialReductionPass();
 
+/// This pass hoists AND instruction that uses a atomic logic instruction to
+/// the same basic block so that ISel has the ability to optimize them.
+FunctionPass *createX86AtomicANDHoistPass();
+
 InstructionSelector *createX86InstructionSelector(const X86TargetMachine &TM,
                                                   X86Subtarget &,
                                                   X86RegisterBankInfo &);
 
 FunctionPass *createX86LoadValueInjectionLoadHardeningPass();
 FunctionPass *createX86LoadValueInjectionRetHardeningPass();
 FunctionPass *createX86SpeculativeLoadHardeningPass();
 FunctionPass *createX86SpeculativeExecutionSideEffectSuppression();
 
 void initializeEvexToVexInstPassPass(PassRegistry &);
 void initializeFixupBWInstPassPass(PassRegistry &);
 void initializeFixupLEAPassPass(PassRegistry &);
 void initializeFPSPass(PassRegistry &);
 void initializeWinEHStatePassPass(PassRegistry &);
+void initializeX86AtomicANDHoistPass(PassRegistry &);
 void initializeX86AvoidSFBPassPass(PassRegistry &);
 void initializeX86AvoidTrailingCallPassPass(PassRegistry &);
 void initializeX86CallFrameOptimizationPass(PassRegistry &);
 void initializeX86CmovConverterPassPass(PassRegistry &);
 void initializeX86DomainReassignmentPass(PassRegistry &);
 void initializeX86ExecutionDomainFixPass(PassRegistry &);
 void initializeX86ExpandPseudoPass(PassRegistry &);
 void initializeX86FixupSetCCPassPass(PassRegistry &);

llvm/lib/Target/X86/X86AtomicANDHoist.cpp

+//===- X86AtomicANDHoist.ll - Hoist AND instrution uses atomic logic one --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "X86.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/Pass.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "x86-atomic-and-hoist"
+
+namespace {
+
+class X86AtomicANDHoist : public FunctionPass {
+public:
+  static char ID; // Pass identification, replacement for typeid.
+
+  X86AtomicANDHoist() : FunctionPass(ID) {}
+
+  bool runOnFunction(Function &F) override;
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+  }
+
+  StringRef getPassName() const override { return "X86 Atomic AND Hoist"; }
+};
+} // namespace
+
+FunctionPass *llvm::createX86AtomicANDHoistPass() {
+  return new X86AtomicANDHoist();
+}
+
+char X86AtomicANDHoist::ID = 0;
+
+INITIALIZE_PASS(X86AtomicANDHoist, DEBUG_TYPE, "X86 Atomic AND Hoist", false,
+                false)
+
+bool X86AtomicANDHoist::runOnFunction(Function &F) {
+  bool Changed = false;
+
+  for (auto &BB : F) {
+    for (auto &I : BB) {
+      auto *AI = dyn_cast<AtomicRMWInst>(&I);
+      if (!AI || !AI->hasOneUse())
+        continue;
+      Instruction *And = AI->user_back();
+      if (And->getOpcode() != Instruction::And || And->getParent() == &BB ||
+          !isa<ConstantInt>(And->getOperand(1)))
+        continue;
+      AtomicRMWInst::BinOp Op = AI->getOperation();
+      if (Op != AtomicRMWInst::Or && Op != AtomicRMWInst::And &&
+          Op != AtomicRMWInst::Xor)
+        continue;
+      Changed = true;
+      And->moveAfter(AI);
+    }
+  }
+
+  return Changed;
+}

llvm/lib/Target/X86/X86ISelLowering.h

enum NodeType : unsigned {
 
     /// LOCK-prefixed arithmetic read-modify-write instructions.
     /// EFLAGS, OUTCHAIN = LADD(INCHAIN, PTR, RHS)
     LADD,
     LSUB,
     LOR,
     LXOR,
     LAND,
+    LBTS,
+    LBTC,
+    LBTR,
 
     // Load, scalar_to_vector, and zero extend.
     VZEXT_LOAD,
 
     // extract_vector_elt, store.
     VEXTRACT_STORE,
 
     // scalar broadcast from memory.
private:
 
     const MCPhysReg *getScratchRegisters(CallingConv::ID CC) const override;
 
     TargetLoweringBase::AtomicExpansionKind
     shouldExpandAtomicLoadInIR(LoadInst *LI) const override;
     bool shouldExpandAtomicStoreInIR(StoreInst *SI) const override;
     TargetLoweringBase::AtomicExpansionKind
     shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const override;
+    TargetLoweringBase::AtomicExpansionKind
+    shouldExpandAtomicLogicRMWInIR(AtomicRMWInst *AI) const;
 
     LoadInst *
     lowerIdempotentRMWIntoFencedLoad(AtomicRMWInst *AI) const override;
 
     bool lowerAtomicStoreAsStoreSDNode(const StoreInst &SI) const override;
     bool lowerAtomicLoadAsLoadSDNode(const LoadInst &LI) const override;
 
     bool needsCmpXchgNb(Type *MemType) const;

llvm/lib/Target/X86/X86ISelLowering.cpp

 }
 
 bool X86TargetLowering::isCtlzFast() const {
   return Subtarget.hasFastLZCNT();
 }
 
 bool X86TargetLowering::isMaskAndCmp0FoldingBeneficial(
     const Instruction &AndI) const {
+  auto *AI = dyn_cast<AtomicRMWInst>(AndI.getOperand(0));
+  if (!AI)
     return true;
+  AtomicRMWInst::BinOp Op = AI->getOperation();
+  return Op != AtomicRMWInst::Or && Op != AtomicRMWInst::And &&
+         Op != AtomicRMWInst::Xor;
 }
 
 bool X86TargetLowering::hasAndNotCompare(SDValue Y) const {
   EVT VT = Y.getValueType();
 
   if (VT.isVector())
     return false;
 
if (MemType->getPrimitiveSizeInBits() == 64 && !Subtarget.is64Bit() &&
       (Subtarget.hasSSE1() || Subtarget.hasX87()))
     return AtomicExpansionKind::None;
 
   return needsCmpXchgNb(MemType) ? AtomicExpansionKind::CmpXChg
                                  : AtomicExpansionKind::None;
 }
 
 TargetLowering::AtomicExpansionKind
+X86TargetLowering::shouldExpandAtomicLogicRMWInIR(AtomicRMWInst *AI) const {
+  // If the atomicrmw's result isn't actually used, we can just add a "lock"
+  // prefix to a normal instruction for these operations.
+  if (AI->use_empty())
+    return AtomicExpansionKind::None;
+
+  // If the atomicrmw's result is used by a single bit AND, we may use
+  // bts/btr/btc instruction for these operations.
+  auto *C1 = dyn_cast<ConstantInt>(AI->getValOperand());
+  Instruction *I = AI->user_back();
+  if (!C1 || !AI->hasOneUse() || I->getOpcode() != Instruction::And ||
+      AI->getParent() != I->getParent())
+    return AtomicExpansionKind::CmpXChg;
+  // The following instruction must be a AND single bit.
+  auto *C2 = dyn_cast<ConstantInt>(I->getOperand(1));
+  unsigned Bits = AI->getType()->getPrimitiveSizeInBits();
+  if (!C2 || Bits == 8 || !isPowerOf2_64(C2->getZExtValue()))
+    return AtomicExpansionKind::CmpXChg;
+
+  if (AI->getOperation() == AtomicRMWInst::And)
+    return ~C1->getValue() == C2->getValue() ? AtomicExpansionKind::None
+                                             : AtomicExpansionKind::CmpXChg;
+
+  return C1 == C2 ? AtomicExpansionKind::None : AtomicExpansionKind::CmpXChg;
+}
+
+TargetLowering::AtomicExpansionKind
 X86TargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
   unsigned NativeWidth = Subtarget.is64Bit() ? 64 : 32;
   Type *MemType = AI->getType();
 
   // If the operand is too big, we must see if cmpxchg8/16b is available
   // and default to library calls otherwise.
   if (MemType->getPrimitiveSizeInBits() > NativeWidth) {
     return needsCmpXchgNb(MemType) ? AtomicExpansionKind::CmpXChg
                                    : AtomicExpansionKind::None;
   }
 
   AtomicRMWInst::BinOp Op = AI->getOperation();
   switch (Op) {
   default:
     llvm_unreachable("Unknown atomic operation");
   case AtomicRMWInst::Xchg:
   case AtomicRMWInst::Add:
   case AtomicRMWInst::Sub:
     // It's better to use xadd, xsub or xchg for these in all cases.
     return AtomicExpansionKind::None;
   case AtomicRMWInst::Or:
   case AtomicRMWInst::And:
   case AtomicRMWInst::Xor:
-    // If the atomicrmw's result isn't actually used, we can just add a "lock"
-    // prefix to a normal instruction for these operations.
-    return !AI->use_empty() ? AtomicExpansionKind::CmpXChg
-                            : AtomicExpansionKind::None;
+    return shouldExpandAtomicLogicRMWInIR(AI);
   case AtomicRMWInst::Nand:
   case AtomicRMWInst::Max:
   case AtomicRMWInst::Min:
   case AtomicRMWInst::UMax:
   case AtomicRMWInst::UMin:
   case AtomicRMWInst::FAdd:
   case AtomicRMWInst::FSub:
     // These always require a non-trivial set of data operations on x86. We must
static SDValue lowerAtomicArith(SDValue N, SelectionDAG &DAG,
   if (N->hasAnyUseOfValue(0)) {
     // Handle (atomic_load_sub p, v) as (atomic_load_add p, -v), to be able to
     // select LXADD if LOCK_SUB can't be selected.
     if (Opc == ISD::ATOMIC_LOAD_SUB) {
       RHS = DAG.getNode(ISD::SUB, DL, VT, DAG.getConstant(0, DL, VT), RHS);
       return DAG.getAtomic(ISD::ATOMIC_LOAD_ADD, DL, VT, Chain, LHS,
                            RHS, AN->getMemOperand());
     }
+    if (Opc == ISD::ATOMIC_LOAD_OR || Opc == ISD::ATOMIC_LOAD_XOR ||
+        Opc == ISD::ATOMIC_LOAD_AND) {
+      ConstantSDNode *C1 = cast<ConstantSDNode>(RHS);
+      ConstantSDNode *C2 = nullptr;
+      for (auto UI = N->use_begin(), UE = N->use_end(); UI != UE; ++UI) {
+        // Skip uses of the chain value. Result 0 of the node is the load value.
+        if (UI.getUse().getResNo() != 0)
+          continue;
+        if (C2 || UI->getOpcode() != ISD::AND)
+          report_fatal_error("Atomic result must be used by one AND");
+        C2 = cast<ConstantSDNode>(UI->getOperand(1));
+        assert(isPowerOf2_64(C2->getZExtValue()) && "Must be power of 2 value");
+      }
+      if (Opc == ISD::ATOMIC_LOAD_AND) {
+        assert(~C1->getAPIntValue() == C2->getAPIntValue() &&
+               "Cannot lower to BTR");
+        Opc = X86ISD::LBTR;
+      } else {
+        assert(C1 == C2 && "Cannot lower to BTS/BTC");
+        Opc = Opc == ISD::ATOMIC_LOAD_OR ? X86ISD::LBTS : X86ISD::LBTC;
+      }
+
+      unsigned Imm = countTrailingZeros(C2->getZExtValue());
+      MachineMemOperand *MMO = cast<MemSDNode>(N)->getMemOperand();
+      SDValue Res = DAG.getMemIntrinsicNode(
+          Opc, DL, DAG.getVTList(VT, MVT::Other),
+          {Chain, LHS, DAG.getConstant(Imm, DL, VT)}, VT, MMO);
+      Chain = Res.getValue(1);
+      Res = DAG.getZExtOrTrunc(getSETCC(X86::COND_B, Res, DL, DAG), DL, VT);
+      if (Imm)
+        Res = DAG.getNode(ISD::SHL, DL, VT, Res,
+                          DAG.getShiftAmountConstant(Imm, VT, DL));
+      return DAG.getNode(ISD::MERGE_VALUES, DL, N->getVTList(), Res, Chain);
+    }
     assert(Opc == ISD::ATOMIC_LOAD_ADD &&
            "Used AtomicRMW ops other than Add should have been expanded!");
     return N;
   }
 
   // Specialized lowering for the canonical form of an idemptotent atomicrmw.
   // The core idea here is that since the memory location isn't actually
   // changing, all we need is a lowering for the *ordering* impacts of the
   // atomicrmw.  As such, we can chose a different operation and memory
   // location to minimize impact on other code.
   if (Opc == ISD::ATOMIC_LOAD_OR && isNullConstant(RHS)) {
     // On X86, the only ordering which actually requires an instruction is
     // seq_cst which isn't SingleThread, everything just needs to be preserved
     // during codegen and then dropped. Note that we expect (but don't assume),
     // that orderings other than seq_cst and acq_rel have been canonicalized to
     // a store or load.
     if (AN->getSuccessOrdering() == AtomicOrdering::SequentiallyConsistent &&
         AN->getSyncScopeID() == SyncScope::System) {
       // Prefer a locked operation against a stack location to minimize cache
       // traffic.  This assumes that stack locations are very likely to be
static SDValue lowerAtomicArith(SDValue N, SelectionDAG &DAG,
   return DAG.getNode(ISD::MERGE_VALUES, DL, N->getVTList(),
                      DAG.getUNDEF(VT), LockOp.getValue(1));
 }
 
 static SDValue LowerATOMIC_STORE(SDValue Op, SelectionDAG &DAG,
                                  const X86Subtarget &Subtarget) {
   auto *Node = cast<AtomicSDNode>(Op.getNode());
   SDLoc dl(Node);
   EVT VT = Node->getMemoryVT();
 
   bool IsSeqCst =
       Node->getSuccessOrdering() == AtomicOrdering::SequentiallyConsistent;
   bool IsTypeLegal = DAG.getTargetLoweringInfo().isTypeLegal(VT);
 
   // If this store is not sequentially consistent and the type is legal
   // we can just keep it.
   if (!IsSeqCst && IsTypeLegal)
     return Op;
 
#define NODE_NAME_CASE(NODE) case X86ISD::NODE: return "X86ISD::" #NODE;
   NODE_NAME_CASE(LCMPXCHG8_DAG)
   NODE_NAME_CASE(LCMPXCHG16_DAG)
   NODE_NAME_CASE(LCMPXCHG16_SAVE_RBX_DAG)
   NODE_NAME_CASE(LADD)
   NODE_NAME_CASE(LSUB)
   NODE_NAME_CASE(LOR)
   NODE_NAME_CASE(LXOR)
   NODE_NAME_CASE(LAND)
+  NODE_NAME_CASE(LBTS)
+  NODE_NAME_CASE(LBTC)
+  NODE_NAME_CASE(LBTR)
   NODE_NAME_CASE(VZEXT_MOVL)
   NODE_NAME_CASE(VZEXT_LOAD)
   NODE_NAME_CASE(VEXTRACT_STORE)
   NODE_NAME_CASE(VTRUNC)
   NODE_NAME_CASE(VTRUNCS)
   NODE_NAME_CASE(VTRUNCUS)
   NODE_NAME_CASE(VMTRUNC)
   NODE_NAME_CASE(VMTRUNCS)

llvm/lib/Target/X86/X86InstrCompiler.td

let Predicates = [UseIncDec] in {
   def : Pat<(X86lock_add addr:$dst, (i32 -1)), (LOCK_DEC32m addr:$dst)>;
   def : Pat<(X86lock_add addr:$dst, (i64 -1)), (LOCK_DEC64m addr:$dst)>;
   def : Pat<(X86lock_sub addr:$dst, (i8  -1)), (LOCK_INC8m  addr:$dst)>;
   def : Pat<(X86lock_sub addr:$dst, (i16 -1)), (LOCK_INC16m addr:$dst)>;
   def : Pat<(X86lock_sub addr:$dst, (i32 -1)), (LOCK_INC32m addr:$dst)>;
   def : Pat<(X86lock_sub addr:$dst, (i64 -1)), (LOCK_INC64m addr:$dst)>;
 }
 
+multiclass ATOMIC_LOG<Format Form, string mnemonic, SDPatternOperator frag> {
+  let Defs = [EFLAGS], mayLoad = 1, mayStore = 1, isCodeGenOnly = 1,
+      SchedRW = [WriteBitTestSetRegRMW]  in {
+    def 16m : Ii8<0xBA, Form, (outs), (ins i16mem:$src1, i16imm:$src2),
+                  !strconcat(mnemonic, "{w}\t{$src2, $src1|$src1, $src2}"),
+                  [(set EFLAGS, (frag addr:$src1, (i16 imm:$src2)))]>,
+              OpSize16, TB, LOCK;
+    def 32m : Ii8<0xBA, Form, (outs), (ins i32mem:$src1, i32imm:$src2),
+                  !strconcat(mnemonic, "{l}\t{$src2, $src1|$src1, $src2}"),
+                  [(set EFLAGS, (frag addr:$src1, (i32 imm:$src2)))]>,
+              OpSize32, TB, LOCK;
+    def 64m : RIi8<0xBA, Form, (outs), (ins i64mem:$src1, i64imm:$src2),
+                   !strconcat(mnemonic, "{q}\t{$src2, $src1|$src1, $src2}"),
+                   [(set EFLAGS, (frag addr:$src1, (i64 imm:$src2)))]>,
+              TB, LOCK;
+  }
+}
+
+defm LOCK_BTS : ATOMIC_LOG<MRM5m, "bts", X86lock_bts>;
+defm LOCK_BTR : ATOMIC_LOG<MRM6m, "btr", X86lock_btr>;
+defm LOCK_BTC : ATOMIC_LOG<MRM7m, "btc", X86lock_btc>;
+
 // Atomic compare and swap.
 multiclass LCMPXCHG_BinOp<bits<8> Opc8, bits<8> Opc, Format Form,
                           string mnemonic, SDPatternOperator frag> {
 let isCodeGenOnly = 1, SchedRW = [WriteCMPXCHGRMW] in {
   let Defs = [AL, EFLAGS], Uses = [AL] in
   def NAME#8  : I<Opc8, Form, (outs), (ins i8mem:$ptr, GR8:$swap),
                   !strconcat(mnemonic, "{b}\t{$swap, $ptr|$ptr, $swap}"),
                   [(frag addr:$ptr, GR8:$swap, 1)]>, TB, LOCK;

llvm/lib/Target/X86/X86InstrInfo.td

def X86lock_or  : SDNode<"X86ISD::LOR",  SDTLockBinaryArithWithFlags,
                          [SDNPHasChain, SDNPMayStore, SDNPMayLoad,
                           SDNPMemOperand]>;
 def X86lock_xor  : SDNode<"X86ISD::LXOR",  SDTLockBinaryArithWithFlags,
                           [SDNPHasChain, SDNPMayStore, SDNPMayLoad,
                            SDNPMemOperand]>;
 def X86lock_and  : SDNode<"X86ISD::LAND",  SDTLockBinaryArithWithFlags,
                           [SDNPHasChain, SDNPMayStore, SDNPMayLoad,
                            SDNPMemOperand]>;
+def X86lock_bts  : SDNode<"X86ISD::LBTS",  SDTLockBinaryArithWithFlags,
+                          [SDNPHasChain, SDNPMayStore, SDNPMayLoad,
+                           SDNPMemOperand]>;
+def X86lock_btc  : SDNode<"X86ISD::LBTC",  SDTLockBinaryArithWithFlags,
+                          [SDNPHasChain, SDNPMayStore, SDNPMayLoad,
+                           SDNPMemOperand]>;
+def X86lock_btr  : SDNode<"X86ISD::LBTR",  SDTLockBinaryArithWithFlags,
+                          [SDNPHasChain, SDNPMayStore, SDNPMayLoad,
+                           SDNPMemOperand]>;
 
 def X86bextr  : SDNode<"X86ISD::BEXTR",  SDTIntBinOp>;
 def X86bextri : SDNode<"X86ISD::BEXTRI", SDTIntBinOp>;
 
 def X86bzhi   : SDNode<"X86ISD::BZHI",   SDTIntBinOp>;
 
 def X86pdep   : SDNode<"X86ISD::PDEP",   SDTIntBinOp>;
 def X86pext   : SDNode<"X86ISD::PEXT",   SDTIntBinOp>;

llvm/lib/Target/X86/X86TargetMachine.cpp

bool X86PassConfig::addILPOpts() {
   addPass(&EarlyIfConverterID);
   if (EnableMachineCombinerPass)
     addPass(&MachineCombinerID);
   addPass(createX86CmovConverterPass());
   return true;
 }
 
 bool X86PassConfig::addPreISel() {
+  addPass(createX86AtomicANDHoistPass());
   // Only add this pass for 32-bit x86 Windows.
   const Triple &TT = TM->getTargetTriple();
   if (TT.isOSWindows() && TT.getArch() == Triple::x86)
     addPass(createX86WinEHStatePass());
   return true;
 }
 
 void X86PassConfig::addPreRegAlloc() {

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:       Shadow Stack GC Lowering
 ; CHECK-NEXT:       Lower constant intrinsics
 ; CHECK-NEXT:       Remove unreachable blocks from the CFG
 ; CHECK-NEXT:       Expand vector predication intrinsics
 ; CHECK-NEXT:       Scalarize Masked Memory Intrinsics
 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Expand indirectbr instructions
 ; CHECK-NEXT:       Exception handling preparation
+; CHECK-NEXT:       X86 Atomic AND Hoist
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       X86 DAG->DAG Instruction Selection
 ; CHECK-NEXT:       X86 PIC Global Base Reg Initialization
 ; CHECK-NEXT:       Finalize ISel and expand pseudo-instructions
 ; CHECK-NEXT:       Local Stack Slot Allocation
 ; CHECK-NEXT:       X86 speculative load hardening

llvm/test/CodeGen/X86/atomic-bit-test.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown | FileCheck %s --check-prefix=X86
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefix=X64
 
 @v16 = dso_local global i16 0, align 2
 @v32 = dso_local global i32 0, align 4
 @v64 = dso_local global i64 0, align 8
 
 define i16 @bts1() nounwind {
 ; X86-LABEL: bts1:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB0_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $1, %ecx
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB0_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btsw $0, v16
+; X86-NEXT:    setb %al
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: bts1:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB0_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $1, %ecx
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB0_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $1, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsw $0, v16(%rip)
+; X64-NEXT:    setb %al
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw or i16* @v16, i16 1 monotonic, align 2
   %and = and i16 %0, 1
   ret i16 %and
 }
 
 define i16 @bts2() nounwind {
 ; X86-LABEL: bts2:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB1_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $2, %ecx
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB1_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $2, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btsw $1, v16
+; X86-NEXT:    setb %al
+; X86-NEXT:    addl %eax, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: bts2:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB1_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $2, %ecx
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB1_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $2, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsw $1, v16(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    addl %eax, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw or i16* @v16, i16 2 monotonic, align 2
   %and = and i16 %0, 2
   ret i16 %and
 }
 
 define i16 @bts15() nounwind {
 ; X86-LABEL: bts15:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB2_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $32768, %ecx # imm = 0x8000
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB2_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $32768, %eax # imm = 0x8000
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btsw $15, v16
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $15, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: bts15:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB2_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $32768, %ecx # imm = 0x8000
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB2_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $32768, %eax # imm = 0x8000
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsw $15, v16(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $15, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw or i16* @v16, i16 32768 monotonic, align 2
   %and = and i16 %0, 32768
   ret i16 %and
 }
 
 define i32 @bts31() nounwind {
 ; X86-LABEL: bts31:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movl v32, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB3_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $-2147483648, %ecx # imm = 0x80000000
-; X86-NEXT:    lock cmpxchgl %ecx, v32
-; X86-NEXT:    jne .LBB3_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $-2147483648, %eax # imm = 0x80000000
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btsl $31, v32
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $31, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: bts31:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movl v32(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB3_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $-2147483648, %ecx # imm = 0x80000000
-; X64-NEXT:    lock cmpxchgl %ecx, v32(%rip)
-; X64-NEXT:    jne .LBB3_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $-2147483648, %eax # imm = 0x80000000
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsl $31, v32(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $31, %eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw or i32* @v32, i32 2147483648 monotonic, align 4
   %and = and i32 %0, 2147483648
   ret i32 %and
 }
 
 define i64 @bts63() nounwind {

 ; X86-NEXT:    andl %esi, %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: bts63:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movabsq $-9223372036854775808, %rcx # imm = 0x8000000000000000
-; X64-NEXT:    movq v64(%rip), %rax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB4_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movq %rax, %rdx
-; X64-NEXT:    orq %rcx, %rdx
-; X64-NEXT:    lock cmpxchgq %rdx, v64(%rip)
-; X64-NEXT:    jne .LBB4_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andq %rcx, %rax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsq $63, v64(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shlq $63, %rax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw or i64* @v64, i64 -9223372036854775808 monotonic, align 8
   %and = and i64 %0, -9223372036854775808
   ret i64 %and
 }
 
 define i16 @btc1() nounwind {
 ; X86-LABEL: btc1:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB5_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    xorl $1, %ecx
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB5_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btcw $0, v16
+; X86-NEXT:    setb %al
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btc1:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB5_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    xorl $1, %ecx
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB5_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $1, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btcw $0, v16(%rip)
+; X64-NEXT:    setb %al
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw xor i16* @v16, i16 1 monotonic, align 2
   %and = and i16 %0, 1
   ret i16 %and
 }
 
 define i16 @btc2() nounwind {
 ; X86-LABEL: btc2:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB6_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    xorl $2, %ecx
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB6_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $2, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btcw $1, v16
+; X86-NEXT:    setb %al
+; X86-NEXT:    addl %eax, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btc2:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB6_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    xorl $2, %ecx
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB6_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $2, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btcw $1, v16(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    addl %eax, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw xor i16* @v16, i16 2 monotonic, align 2
   %and = and i16 %0, 2
   ret i16 %and
 }
 
 define i16 @btc15() nounwind {
 ; X86-LABEL: btc15:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB7_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    xorl $32768, %ecx # imm = 0x8000
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB7_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $32768, %eax # imm = 0x8000
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btcw $15, v16
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $15, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btc15:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB7_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    xorl $32768, %ecx # imm = 0x8000
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB7_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $32768, %eax # imm = 0x8000
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btcw $15, v16(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $15, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw xor i16* @v16, i16 32768 monotonic, align 2
   %and = and i16 %0, 32768
   ret i16 %and
 }
 
 define i32 @btc31() nounwind {
 ; X86-LABEL: btc31:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movl v32, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB8_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    xorl $-2147483648, %ecx # imm = 0x80000000
-; X86-NEXT:    lock cmpxchgl %ecx, v32
-; X86-NEXT:    jne .LBB8_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $-2147483648, %eax # imm = 0x80000000
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btcl $31, v32
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $31, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btc31:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movl v32(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB8_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    xorl $-2147483648, %ecx # imm = 0x80000000
-; X64-NEXT:    lock cmpxchgl %ecx, v32(%rip)
-; X64-NEXT:    jne .LBB8_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $-2147483648, %eax # imm = 0x80000000
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btcl $31, v32(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $31, %eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw xor i32* @v32, i32 2147483648 monotonic, align 4
   %and = and i32 %0, 2147483648
   ret i32 %and
 }
 
 define i64 @btc63() nounwind {

 ; X86-NEXT:    andl %esi, %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btc63:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movabsq $-9223372036854775808, %rcx # imm = 0x8000000000000000
-; X64-NEXT:    movq v64(%rip), %rax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB9_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movq %rax, %rdx
-; X64-NEXT:    xorq %rcx, %rdx
-; X64-NEXT:    lock cmpxchgq %rdx, v64(%rip)
-; X64-NEXT:    jne .LBB9_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andq %rcx, %rax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btcq $63, v64(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shlq $63, %rax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw xor i64* @v64, i64 -9223372036854775808 monotonic, align 8
   %and = and i64 %0, -9223372036854775808
   ret i64 %and
 }
 
 define i16 @btr1() nounwind {
 ; X86-LABEL: btr1:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB10_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    andl $65534, %ecx # imm = 0xFFFE
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB10_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btrw $0, v16
+; X86-NEXT:    setb %al
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btr1:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB10_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    andl $65534, %ecx # imm = 0xFFFE
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB10_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $1, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btrw $0, v16(%rip)
+; X64-NEXT:    setb %al
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw and i16* @v16, i16 -2 monotonic, align 2
   %and = and i16 %0, 1
   ret i16 %and
 }
 
 define i16 @btr2() nounwind {
 ; X86-LABEL: btr2:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB11_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    andl $65533, %ecx # imm = 0xFFFD
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB11_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $2, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btrw $1, v16
+; X86-NEXT:    setb %al
+; X86-NEXT:    addl %eax, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btr2:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB11_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    andl $65533, %ecx # imm = 0xFFFD
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB11_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $2, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btrw $1, v16(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    addl %eax, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw and i16* @v16, i16 -3 monotonic, align 2
   %and = and i16 %0, 2
   ret i16 %and
 }
 
 define i16 @btr15() nounwind {
 ; X86-LABEL: btr15:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB12_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    andl $32767, %ecx # imm = 0x7FFF
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB12_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $32768, %eax # imm = 0x8000
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btrw $15, v16
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $15, %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btr15:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB12_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    andl $32767, %ecx # imm = 0x7FFF
-; X64-NEXT:    # kill: def $ax killed $ax killed $eax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $eax
-; X64-NEXT:    jne .LBB12_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $32768, %eax # imm = 0x8000
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btrw $15, v16(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $15, %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw and i16* @v16, i16 32767 monotonic, align 2
   %and = and i16 %0, 32768
   ret i16 %and
 }
 
 define i32 @btr31() nounwind {
 ; X86-LABEL: btr31:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movl v32, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB13_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    andl $2147483647, %ecx # imm = 0x7FFFFFFF
-; X86-NEXT:    lock cmpxchgl %ecx, v32
-; X86-NEXT:    jne .LBB13_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $-2147483648, %eax # imm = 0x80000000
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btrl $31, v32
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $31, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btr31:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movl v32(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB13_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    andl $2147483647, %ecx # imm = 0x7FFFFFFF
-; X64-NEXT:    lock cmpxchgl %ecx, v32(%rip)
-; X64-NEXT:    jne .LBB13_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $-2147483648, %eax # imm = 0x80000000
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btrl $31, v32(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $31, %eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw and i32* @v32, i32 2147483647 monotonic, align 4
   %and = and i32 %0, 2147483648
   ret i32 %and
 }
 
 define i64 @btr63() nounwind {

 ; X86-NEXT:    andl %esi, %edx
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: btr63:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movabsq $9223372036854775807, %rcx # imm = 0x7FFFFFFFFFFFFFFF
-; X64-NEXT:    movq v64(%rip), %rax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB14_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movq %rax, %rdx
-; X64-NEXT:    andq %rcx, %rdx
-; X64-NEXT:    lock cmpxchgq %rdx, v64(%rip)
-; X64-NEXT:    jne .LBB14_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    incq %rcx
-; X64-NEXT:    andq %rcx, %rax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btrq $63, v64(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shlq $63, %rax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw and i64* @v64, i64 9223372036854775807 monotonic, align 8
   %and = and i64 %0, -9223372036854775808
   ret i64 %and
 }
 
 define i16 @multi_use1() nounwind {
entry:
   %2 = xor i16 %0, 2
   %3 = or i16 %1, %2
   ret i16 %3
 }
 
 define i16 @multi_use2() nounwind {
 ; X86-LABEL: multi_use2:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movzwl v16, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB16_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $1, %ecx
-; X86-NEXT:    # kill: def $ax killed $ax killed $eax
-; X86-NEXT:    lock cmpxchgw %cx, v16
-; X86-NEXT:    # kill: def $ax killed $ax def $eax
-; X86-NEXT:    jne .LBB16_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
-; X86-NEXT:    andl $1, %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btsw $0, v16
+; X86-NEXT:    setb %al
 ; X86-NEXT:    leal (%eax,%eax,2), %eax
 ; X86-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: multi_use2:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movzwl v16(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB16_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $1, %ecx
-; X64-NEXT:    # kill: def $ax killed $ax killed $rax
-; X64-NEXT:    lock cmpxchgw %cx, v16(%rip)
-; X64-NEXT:    # kill: def $ax killed $ax def $rax
-; X64-NEXT:    jne .LBB16_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
-; X64-NEXT:    andl $1, %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsw $0, v16(%rip)
+; X64-NEXT:    setb %al
 ; X64-NEXT:    leal (%rax,%rax,2), %eax
 ; X64-NEXT:    # kill: def $ax killed $ax killed $eax
 ; X64-NEXT:    retq
 entry:
   %0 = atomicrmw or i16* @v16, i16 1 monotonic, align 2
   %1 = and i16 %0, 1
   %2 = shl i16 %1, 1
   %3 = or i16 %1, %2
3:
   ret i16 %and
 }
 
 declare void @foo()
 
 define void @no_and_cmp0_fold() nounwind {
 ; X86-LABEL: no_and_cmp0_fold:
 ; X86:       # %bb.0: # %entry
-; X86-NEXT:    movl v32, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB18_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %ecx
-; X86-NEXT:    orl $8, %ecx
-; X86-NEXT:    lock cmpxchgl %ecx, v32
-; X86-NEXT:    jne .LBB18_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
+; X86-NEXT:    lock btsl $3, v32
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    testb %al, %al
-; X86-NEXT:    je .LBB18_3
-; X86-NEXT:  # %bb.4: # %if.end
+; X86-NEXT:    je .LBB18_1
+; X86-NEXT:  # %bb.2: # %if.end
 ; X86-NEXT:    retl
-; X86-NEXT:  .LBB18_3: # %if.then
+; X86-NEXT:  .LBB18_1: # %if.then
 ;
 ; X64-LABEL: no_and_cmp0_fold:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movl v32(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB18_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $8, %ecx
-; X64-NEXT:    lock cmpxchgl %ecx, v32(%rip)
-; X64-NEXT:    jne .LBB18_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
+; X64-NEXT:    lock btsl $3, v32(%rip)
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    testb %al, %al
-; X64-NEXT:    je .LBB18_3
-; X64-NEXT:  # %bb.4: # %if.end
+; X64-NEXT:    je .LBB18_1
+; X64-NEXT:  # %bb.2: # %if.end
 ; X64-NEXT:    retq
-; X64-NEXT:  .LBB18_3: # %if.then
+; X64-NEXT:  .LBB18_1: # %if.then
 entry:
   %0 = atomicrmw or i32* @v32, i32 8 monotonic, align 4
   %and = and i32 %0, 8
   %tobool = icmp ne i32 %and, 0
   br i1 undef, label %if.then, label %if.end
 
 if.then:                                          ; preds = %entry
   unreachable
 
 if.end:                                           ; preds = %entry
   %or.cond8 = select i1 %tobool, i1 undef, i1 false
   ret void
 }
 
 define i32 @split_hoist_and(i32 %0) nounwind {
 ; X86-LABEL: split_hoist_and:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    movl v32, %eax
-; X86-NEXT:    .p2align 4, 0x90
-; X86-NEXT:  .LBB19_1: # %atomicrmw.start
-; X86-NEXT:    # =>This Inner Loop Header: Depth=1
-; X86-NEXT:    movl %eax, %edx
-; X86-NEXT:    orl $8, %edx
-; X86-NEXT:    lock cmpxchgl %edx, v32
-; X86-NEXT:    jne .LBB19_1
-; X86-NEXT:  # %bb.2: # %atomicrmw.end
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    lock btsl $3, v32
+; X86-NEXT:    setb %al
+; X86-NEXT:    shll $3, %eax
 ; X86-NEXT:    testl %ecx, %ecx
-; X86-NEXT:    andl $8, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: split_hoist_and:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl v32(%rip), %eax
-; X64-NEXT:    .p2align 4, 0x90
-; X64-NEXT:  .LBB19_1: # %atomicrmw.start
-; X64-NEXT:    # =>This Inner Loop Header: Depth=1
-; X64-NEXT:    movl %eax, %ecx
-; X64-NEXT:    orl $8, %ecx
-; X64-NEXT:    lock cmpxchgl %ecx, v32(%rip)
-; X64-NEXT:    jne .LBB19_1
-; X64-NEXT:  # %bb.2: # %atomicrmw.end
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    lock btsl $3, v32(%rip)
+; X64-NEXT:    setb %al
+; X64-NEXT:    shll $3, %eax
 ; X64-NEXT:    testl %edi, %edi
-; X64-NEXT:    andl $8, %eax
 ; X64-NEXT:    retq
   %2 = atomicrmw or i32* @v32, i32 8 monotonic, align 4
   %3 = tail call i32 @llvm.ctlz.i32(i32 %0, i1 false)
   %4 = and i32 %2, 8
   ret i32 %4
 }
 
 declare i32 @llvm.ctlz.i32(i32, i1)

llvm/test/CodeGen/X86/opt-pipeline.ll

 ; CHECK-NEXT:       Expand reduction intrinsics
 ; CHECK-NEXT:       Interleaved Access Pass
 ; CHECK-NEXT:       X86 Partial Reduction
 ; CHECK-NEXT:       Expand indirectbr instructions
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       CodeGen Prepare
 ; CHECK-NEXT:       Dominator Tree Construction
 ; CHECK-NEXT:       Exception handling preparation
+; CHECK-NEXT:       X86 Atomic AND Hoist
 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:       Function Alias Analysis Results
 ; CHECK-NEXT:       Natural Loop Information
 ; CHECK-NEXT:       Post-Dominator Tree Construction
 ; CHECK-NEXT:       Branch Probability Analysis