[DAGCombine] Match a pattern where a wide type scalar value is stored by several narrow stores

[steven.zhang]

Summary

This opportunity is found from spec 2017 557.xz_r. And it is used by the sha encrypt/decrypt. See sha-2/sha512.c

static void store64(u64 x, unsigned char* y)
{
    for(int i = 0; i != 8; ++i)
        y[i] = (x >> ((7-i) * 8)) & 255;
}

static u64 load64(const unsigned char* y)
{
    u64 res = 0;
    for(int i = 0; i != 8; ++i)
        res |= (u64)(y[i]) << ((7-i) * 8);
    return res;
}

The load64 has been implemented by https://reviews.llvm.org/D26149
This patch is trying to implement the store pattern.

Match a pattern where a wide type scalar value is stored by several narrow
stores. Fold it into a single store or a BSWAP and a store if the targets
supports it.

Assuming little endian target:
i8 *p = ...
i32 val = ...
p[0] = (val >> 0) & 0xFF;
p[1] = (val >> 8) & 0xFF;
p[2] = (val >> 16) & 0xFF;
p[3] = (val >> 24) & 0xFF;

>

*((i32)p) = val;

i8 *p = ...
i32 val = ...
p[0] = (val >> 24) & 0xFF;
p[1] = (val >> 16) & 0xFF;
p[2] = (val >> 8) & 0xFF;
p[3] = (val >> 0) & 0xFF;

>

*((i32)p) = BSWAP(val);

[lebedev.ri]

This looks like a middle-end optimization problem to me.
https://godbolt.org/z/bwR-k1
Final i64 is certainly a legal type as per datalayout, and that IR certainly doesn't loop optimal to me.

[steven.zhang]

That was discussed widely when https://reviews.llvm.org/D26149 is reviewed. This is the commit log saying something about the delay.

This optimization was discussed on llvm-dev some time ago in "Load combine pass" thread. We came to the conclusion that we want to do
this transformation late in the pipeline because in presence of atomic loads load widening is irreversible transformation and it might hinder other optimizations.
    
Eventually we'd like to support folding patterns like this where the offset has a variable and a constant part:
i32 val = a[i] | (a[i + 1] << 8) | (a[i + 2] << 16) | (a[i + 3] << 24)

Matching the pattern above is easier at SelectionDAG level since address reassociation has already happened and the fact that the loads are adjacent is clear. Understanding that these loads are adjacent at IR level would have involved looking through geps/zexts/adds while looking at the addresses.

The general scheme is to match OR expressions by recursively calculating the origin of individual bytes which constitute the resulting OR value. If all the OR bytes come from memory verify that they are adjacent and match with little or big endian encoding of a wider value. If so and the load of the wider type (and bswap if needed) is allowed by the target generate a load and a bswap if needed.

[steven.zhang]

FYI. This is the thread of the discussion. http://lists.llvm.org/pipermail/llvm-dev/2016-September/105291.html

[steven.zhang]

Gentle ping ... Thank you!

[RKSimon]

llvm/test/CodeGen/SystemZ/codegenprepare-splitstore.ll

3 ↗

(On Diff #199199)

The change to the form of test needs committing beforehand, assuming the systemz guys are happy with it. Then this patch should just show any delta in the IR.

[steven.zhang]

llvm/test/CodeGen/SystemZ/codegenprepare-splitstore.ll

3 ↗

(On Diff #199199)

ok. I will commit another patch for this and let you know if they accept it.

[steven.zhang]

https://reviews.llvm.org/D62370 is created to update the test llvm/test/CodeGen/SystemZ/codegenprepare-splitstore.ll

[apilipenko]

Have you considered reusing calculateByteProvider mechanism from load combining instead of matching an exact pattern?

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6219–6233 ↗

(On Diff #199199)

Can you add a comment outlining the pattern you are looking for? It would be easier to grasp from a short comment than from the implementation.

6251 ↗

(On Diff #199199)

Typo "small" -> "smaller"

6318 ↗

(On Diff #199199)

What happens to the individual stores after? Do you rely on other DAG combine rules to remove them?

[steven.zhang]

Yes, I tried but decide not to reuse it. Because, the load pattern is trying to find out the load sequences that they are SHIFT and OR together, so, they have to walk the tree recursively to collect all the loads. But the store pattern is much easier. Because, We already know the stores from the chain. What we need to do is to check if all the store values are from some fixed pattern.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6219–6233 ↗

(On Diff #199199)

ok

6318 ↗

(On Diff #199199)

Exactly.

[apilipenko]

LGTM

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6318 ↗

(On Diff #199199)

Can you add a comment saying that?

[steven.zhang]

Address comments.

[steven.zhang]

I will hold this patch until the test case update for systemZ is accepted. D62370: [NFC] Check the endianness after the CodeGenPrepare.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

6318 ↗

(On Diff #199199)

ok

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue MatchBSwapHWordLow(SDNode *N, SDValue N0, SDValue N1,
                                bool DemandHighBits = true);
     SDValue MatchBSwapHWord(SDNode *N, SDValue N0, SDValue N1);
     SDNode *MatchRotatePosNeg(SDValue Shifted, SDValue Pos, SDValue Neg,
                               SDValue InnerPos, SDValue InnerNeg,
                               unsigned PosOpcode, unsigned NegOpcode,
                               const SDLoc &DL);
     SDNode *MatchRotate(SDValue LHS, SDValue RHS, const SDLoc &DL);
     SDValue MatchLoadCombine(SDNode *N);
+    SDValue MatchStoreCombine(StoreSDNode *N);
     SDValue ReduceLoadWidth(SDNode *N);
     SDValue ReduceLoadOpStoreWidth(SDNode *N);
     SDValue splitMergedValStore(StoreSDNode *ST);
     SDValue TransformFPLoadStorePair(SDNode *N);
     SDValue convertBuildVecZextToZext(SDNode *N);
     SDValue reduceBuildVecExtToExtBuildVec(SDNode *N);
     SDValue reduceBuildVecToShuffle(SDNode *N);
     SDValue createBuildVecShuffle(const SDLoc &DL, SDNode *N,
if (!BigEndian && !LittleEndian)
       return None;
   }
 
   assert((BigEndian != LittleEndian) && "It should be either big endian or"
                                         "little endian");
   return BigEndian;
 }
 
+static SDValue stripTruncAndExt(SDValue Value) {
+  switch (Value.getOpcode()) {
+  case ISD::TRUNCATE:
+  case ISD::ZERO_EXTEND:
+  case ISD::SIGN_EXTEND:
+  case ISD::ANY_EXTEND:
+    return stripTruncAndExt(Value.getOperand(0));
+  }
+  return Value;
+}
+
+/// Match a pattern where a wide type scalar value is stored by several narrow
+/// stores. Fold it into a single store or a BSWAP and a store if the targets
+/// supports it.
+///
+/// Assuming little endian target:
+///  i8 *p = ...
+///  i32 val = ...
+///  p[0] = (val >> 0) & 0xFF;
+///  p[1] = (val >> 8) & 0xFF;
+///  p[2] = (val >> 16) & 0xFF;
+///  p[3] = (val >> 24) & 0xFF;
+/// =>
+///  *((i32)p) = val;
+///
+///  i8 *p = ...
+///  i32 val = ...
+///  p[0] = (val >> 24) & 0xFF;
+///  p[1] = (val >> 16) & 0xFF;
+///  p[2] = (val >> 8) & 0xFF;
+///  p[3] = (val >> 0) & 0xFF;
+/// =>
+///  *((i32)p) = BSWAP(val);
+SDValue DAGCombiner::MatchStoreCombine(StoreSDNode *N) {
+  // Collect all the stores in the chain.
+  SDValue Chain;
+  SmallVector<StoreSDNode *, 8> Stores;
+  for (StoreSDNode *Store = N; Store; Store = dyn_cast<StoreSDNode>(Chain)) {
+    if (Store->getMemoryVT() != MVT::i8 ||
+        Store->isVolatile() || Store->isIndexed())
+      return SDValue();
+    Stores.push_back(Store);
+    Chain = Store->getChain();
+  }
+  // Handle the simple type only.
+  unsigned Width = Stores.size();
+  EVT VT = EVT::getIntegerVT(
+    *DAG.getContext(), Width * N->getMemoryVT().getSizeInBits());
+  if (VT != MVT::i16 && VT != MVT::i32 && VT != MVT::i64)
+    return SDValue();
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  if (LegalOperations && !TLI.isOperationLegal(ISD::STORE, VT))
+    return SDValue();
+
+  // Check if all the bytes of the combined value we are looking at are stored 
+  // to the same base address. Collect bytes offsets from Base address into 
+  // ByteOffsets. 
+  SDValue CombinedValue;
+  SmallVector<int64_t, 4> ByteOffsets(Width);
+  int64_t FirstOffset = INT64_MAX;
+  StoreSDNode *FirstStore = nullptr;
+  Optional<BaseIndexOffset> Base;
+  for (auto Store : Stores) {
+    // All the stores store different byte of the CombinedValue. A truncate is
+    // required to get that byte value.
+    SDValue Trunc = Store->getValue();
+    if (Trunc.getOpcode() != ISD::TRUNCATE)
+      return SDValue();
+    // A shift operation is required to get the right byte offset, except the
+    // first byte.
+    int64_t Offset = 0;
+    SDValue Value = Trunc.getOperand(0);
+    if (Value.getOpcode() == ISD::SRL ||
+        Value.getOpcode() == ISD::SRA) {
+      ConstantSDNode *ShiftOffset =
+        dyn_cast<ConstantSDNode>(Value.getOperand(1));
+      // Trying to match the following pattern. The shift offset must be 
+      // a constant and a multiple of 8. It is the byte offset in "y".
+      // 
+      // x = srl y, offset
+      // i8 z = trunc x 
+      // store z, ...
+      if (!ShiftOffset || (ShiftOffset->getSExtValue() % 8))
+        return SDValue();
+  
+     Offset = ShiftOffset->getSExtValue()/8;
+     Value = Value.getOperand(0);
+    }
+
+    // Stores must share the same combined value with different offsets.
+    if (!CombinedValue)
+      CombinedValue = Value;
+    else if (stripTruncAndExt(CombinedValue) != stripTruncAndExt(Value))
+      return SDValue();
+
+    // The trunc and all the extend operation should be stripped to get the
+    // real value we are stored.
+    else if (CombinedValue.getValueType() != VT) {
+      if (Value.getValueType() == VT ||
+          Value.getValueSizeInBits() > CombinedValue.getValueSizeInBits())
+        CombinedValue = Value;
+      // Give up if the combined value type is smaller than the store size.
+      if (CombinedValue.getValueSizeInBits() < VT.getSizeInBits())
+        return SDValue();
+    }
+
+    // Stores must share the same base address
+    BaseIndexOffset Ptr = BaseIndexOffset::match(Store, DAG);
+    int64_t ByteOffsetFromBase = 0;
+    if (!Base)
+      Base = Ptr;
+    else if (!Base->equalBaseIndex(Ptr, DAG, ByteOffsetFromBase))
+      return SDValue();
+
+    // Remember the first byte store
+    if (ByteOffsetFromBase < FirstOffset) {
+      FirstStore = Store;
+      FirstOffset = ByteOffsetFromBase;
+    }
+    // Map the offset in the store and the offset in the combined value.
+    if (Offset < 0 || Offset >= Width)
+      return SDValue();
+    ByteOffsets[Offset] = ByteOffsetFromBase;
+  }
+
+  assert(FirstOffset != INT64_MAX && "First byte offset must be set");
+  assert(FirstStore && "First store must be set");
+
+  // Check if the bytes of the combined value we are looking at match with 
+  // either big or little endian value store.
+  Optional<bool> IsBigEndian = isBigEndian(ByteOffsets, FirstOffset);
+  if (!IsBigEndian.hasValue())
+    return SDValue();
+
+  // The node we are looking at matches with the pattern, check if we can
+  // replace it with a single bswap if needed and store.
+
+  // If the store needs byte swap check if the target supports it
+  bool NeedsBswap = DAG.getDataLayout().isBigEndian() != *IsBigEndian;
+
+  // Before legalize we can introduce illegal bswaps which will be later
+  // converted to an explicit bswap sequence. This way we end up with a single
+  // store and byte shuffling instead of several stores and byte shuffling.
+  if (NeedsBswap && LegalOperations && !TLI.isOperationLegal(ISD::BSWAP, VT))
+    return SDValue();
+
+  // Check that a store of the wide type is both allowed and fast on the target
+  bool Fast = false;
+  bool Allowed = TLI.allowsMemoryAccess(*DAG.getContext(), DAG.getDataLayout(),
+                                        VT, FirstStore->getAddressSpace(),
+                                        FirstStore->getAlignment(), &Fast);
+  if (!Allowed || !Fast)
+    return SDValue();
+
+  if (VT != CombinedValue.getValueType()) {
+    assert(CombinedValue.getValueType().getSizeInBits() > VT.getSizeInBits() &&
+           "Get unexpected store value to combine");
+    CombinedValue = DAG.getNode(ISD::TRUNCATE, SDLoc(N), VT,
+                             CombinedValue);
+  }
+
+  if (NeedsBswap)
+    CombinedValue = DAG.getNode(ISD::BSWAP, SDLoc(N), VT, CombinedValue);
+
+  SDValue NewStore =
+    DAG.getStore(Chain, SDLoc(N),  CombinedValue, FirstStore->getBasePtr(),
+                 FirstStore->getPointerInfo(), FirstStore->getAlignment());
+
+  // Rely on other DAG combine rules to remove the other individual stores.
+  DAG.ReplaceAllUsesWith(N, NewStore.getNode());
+  return NewStore;
+}
+
 /// Match a pattern where a wide type scalar value is loaded by several narrow
 /// loads and combined by shifts and ors. Fold it into a single load or a load
 /// and a BSWAP if the targets supports it.
 ///
 /// Assuming little endian target:
 ///  i8 *a = ...
 ///  i32 val = a[0] | (a[1] << 8) | (a[2] << 16) | (a[3] << 24)
 /// =>
if (OptLevel != CodeGenOpt::None && ST->isUnindexed()) {
     }
   }
 
   // Try transforming a pair floating point load / store ops to integer
   // load / store ops.
   if (SDValue NewST = TransformFPLoadStorePair(N))
     return NewST;
 
+  // Try transforming several stores into STORE (BSWAP).
+  if (SDValue Store = MatchStoreCombine(ST))
+    return Store;
+
   if (ST->isUnindexed()) {
     // Walk up chain skipping non-aliasing memory nodes, on this store and any
     // adjacent stores.
     if (findBetterNeighborChains(ST)) {
       // replaceStoreChain uses CombineTo, which handled all of the worklist
       // manipulation. Return the original node to not do anything else.
       return SDValue(ST, 0);
     }

llvm/trunk/test/CodeGen/PowerPC/store-combine.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 -verify-machineinstrs < %s | FileCheck %s -check-prefix=CHECK-PPC64LE
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 -verify-machineinstrs < %s | FileCheck %s -check-prefix=CHECK-PPC64
 ; i8* p;
 ; i32 m;
 ; p[0] = (m >> 0) & 0xFF;
 ; p[1] = (m >> 8) & 0xFF;
 ; p[2] = (m >> 16) & 0xFF;
 ; p[3] = (m >> 24) & 0xFF;
 define void @store_i32_by_i8(i32 signext %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 3(4)
+; CHECK-PPC64LE-NEXT:    stw 3, 0(4)
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, 2(4)
-; CHECK-PPC64-NEXT:    stb 3, 3(4)
+; CHECK-PPC64-NEXT:    stwbrx 3, 0, 4 
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %conv = trunc i32 %m to i8
   store i8 %conv, i8* %p, align 1
   %0 = lshr i32 %m, 8
   %conv3 = trunc i32 %0 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 1
   store i8 %conv3, i8* %arrayidx4, align 1

 ; i32 m;
 ; p[0] = (m >> 24) & 0xFF;
 ; p[1] = (m >> 16) & 0xFF;
 ; p[2] = (m >> 8) & 0xFF;
 ; p[3] = (m >> 0) & 0xFF;
 define void @store_i32_by_i8_bswap(i32 signext %m, i8* %p)  {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 0(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 3(4)
+; CHECK-PPC64LE-NEXT:    stwbrx 3, 0, 4
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 24
-; CHECK-PPC64-NEXT:    srwi 6, 3, 16
-; CHECK-PPC64-NEXT:    stb 5, 0(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 6, 1(4)
-; CHECK-PPC64-NEXT:    stb 5, 2(4)
-; CHECK-PPC64-NEXT:    stb 3, 3(4)
+; CHECK-PPC64-NEXT:    stw 3, 0(4)
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %0 = lshr i32 %m, 24
   %conv = trunc i32 %0 to i8
   store i8 %conv, i8* %p, align 1
   %1 = lshr i32 %m, 16
   %conv3 = trunc i32 %1 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 1

 ; p[3] = (m >> 24) & 0xFF;
 ; p[4] = (m >> 32) & 0xFF;
 ; p[5] = (m >> 40) & 0xFF;
 ; p[6] = (m >> 48) & 0xFF;
 ; p[7] = (m >> 56) & 0xFF;
 define void @store_i64_by_i8(i64 %m, i8* %p)  {
 ; CHECK-PPC64LE-LABEL: store_i64_by_i8:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 56, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 48, 16
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 40, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 3(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 32, 32
-; CHECK-PPC64LE-NEXT:    stb 5, 4(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 24, 40
-; CHECK-PPC64LE-NEXT:    stb 5, 5(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 16, 48
-; CHECK-PPC64LE-NEXT:    rldicl 3, 3, 8, 56
-; CHECK-PPC64LE-NEXT:    stb 5, 6(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 7(4)
+; CHECK-PPC64LE-NEXT:    stdx 3, 0, 4
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i64_by_i8:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 56, 8
-; CHECK-PPC64-NEXT:    rldicl 6, 3, 48, 16
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 40, 24
-; CHECK-PPC64-NEXT:    stb 6, 2(4)
-; CHECK-PPC64-NEXT:    rldicl 6, 3, 32, 32
-; CHECK-PPC64-NEXT:    stb 5, 3(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 24, 40
-; CHECK-PPC64-NEXT:    stb 6, 4(4)
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
-; CHECK-PPC64-NEXT:    stb 5, 5(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 16, 48
-; CHECK-PPC64-NEXT:    rldicl 3, 3, 8, 56
-; CHECK-PPC64-NEXT:    stb 5, 6(4)
-; CHECK-PPC64-NEXT:    stb 3, 7(4)
+; CHECK-PPC64-NEXT:    stdbrx 3, 0, 4
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %conv = trunc i64 %m to i8
   store i8 %conv, i8* %p, align 1
   %0 = lshr i64 %m, 8
   %conv3 = trunc i64 %0 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 1
   store i8 %conv3, i8* %arrayidx4, align 1

 ; p[4] = (m >> 24) & 0xFF;
 ; p[3] = (m >> 32) & 0xFF;
 ; p[2] = (m >> 40) & 0xFF;
 ; p[1] = (m >> 48) & 0xFF;
 ; p[0] = (m >> 56) & 0xFF;
 define void @store_i64_by_i8_bswap(i64 %m, i8* %p)  {
 ; CHECK-PPC64LE-LABEL: store_i64_by_i8_bswap:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 56, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 7(4)
-; CHECK-PPC64LE-NEXT:    stb 5, 6(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 48, 16
-; CHECK-PPC64LE-NEXT:    stb 5, 5(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 40, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 4(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 32, 32
-; CHECK-PPC64LE-NEXT:    stb 5, 3(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 24, 40
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 16, 48
-; CHECK-PPC64LE-NEXT:    rldicl 3, 3, 8, 56
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
+; CHECK-PPC64LE-NEXT:    stdbrx 3, 0, 4
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i64_by_i8_bswap:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 56, 8
-; CHECK-PPC64-NEXT:    rldicl 6, 3, 48, 16
-; CHECK-PPC64-NEXT:    stb 5, 6(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 40, 24
-; CHECK-PPC64-NEXT:    stb 6, 5(4)
-; CHECK-PPC64-NEXT:    rldicl 6, 3, 32, 32
-; CHECK-PPC64-NEXT:    stb 5, 4(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 24, 40
-; CHECK-PPC64-NEXT:    stb 6, 3(4)
-; CHECK-PPC64-NEXT:    stb 3, 7(4)
-; CHECK-PPC64-NEXT:    stb 5, 2(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 16, 48
-; CHECK-PPC64-NEXT:    rldicl 3, 3, 8, 56
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
+; CHECK-PPC64-NEXT:    stdx 3, 0, 4
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %conv = trunc i64 %m to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 7
   store i8 %conv, i8* %arrayidx, align 1
   %0 = lshr i64 %m, 8
   %conv3 = trunc i64 %0 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 6

 ; p[4] = (m >> 24) & 0xFF;
 ; p[3] = (m >> 32) & 0xFF;
 ; p[2] = (m >> 40) & 0xFF;
 ; p[1] = (m >> 48) & 0xFF;
 ; p[0] = (m >> 56) & 0xFF;
 define void @store_i64_by_i8_bswap_uses(i32 signext %t, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i64_by_i8_bswap_uses:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    slwi 5, 3, 3
-; CHECK-PPC64LE-NEXT:    subf 3, 3, 5
-; CHECK-PPC64LE-NEXT:    extsw 3, 3
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 56, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 7(4)
-; CHECK-PPC64LE-NEXT:    stb 5, 6(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 48, 16
-; CHECK-PPC64LE-NEXT:    stb 5, 5(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 40, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 4(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 32, 32
-; CHECK-PPC64LE-NEXT:    stb 5, 3(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 24, 40
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    rldicl 5, 3, 16, 48
-; CHECK-PPC64LE-NEXT:    rldicl 3, 3, 8, 56
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
+; CHECK-PPC64LE-NEXT:    slwi [[REG:[0-9]+]], 3, 3
+; CHECK-PPC64LE-NEXT:    subf [[REG1:[0-9]+]], 3, [[REG]] 
+; CHECK-PPC64LE-NEXT:    extsw [[REG2:[0-9]+]], [[REG1]]
+; CHECK-PPC64LE-NEXT:    stdbrx [[REG2]], 0, 4
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i64_by_i8_bswap_uses:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    slwi 5, 3, 3
-; CHECK-PPC64-NEXT:    subf 3, 3, 5
-; CHECK-PPC64-NEXT:    extsw 3, 3
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 56, 8
-; CHECK-PPC64-NEXT:    rldicl 6, 3, 48, 16
-; CHECK-PPC64-NEXT:    stb 5, 6(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 40, 24
-; CHECK-PPC64-NEXT:    stb 6, 5(4)
-; CHECK-PPC64-NEXT:    rldicl 6, 3, 32, 32
-; CHECK-PPC64-NEXT:    stb 5, 4(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 24, 40
-; CHECK-PPC64-NEXT:    stb 6, 3(4)
-; CHECK-PPC64-NEXT:    stb 3, 7(4)
-; CHECK-PPC64-NEXT:    stb 5, 2(4)
-; CHECK-PPC64-NEXT:    rldicl 5, 3, 16, 48
-; CHECK-PPC64-NEXT:    rldicl 3, 3, 8, 56
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
+; CHECK-PPC64-NEXT:    slwi [[REG:[0-9]+]], 3, 3
+; CHECK-PPC64-NEXT:    subf [[REG1:[0-9]+]], 3, [[REG]]
+; CHECK-PPC64-NEXT:    extsw [[REG2:[0-9]+]], [[REG1]]
+; CHECK-PPC64-NEXT:    stdx [[REG2]], 0, 4
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %mul = mul nsw i32 %t, 7
   %conv = sext i32 %mul to i64
   %conv1 = trunc i32 %mul to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 7
   store i8 %conv1, i8* %arrayidx, align 1
   %0 = lshr i64 %conv, 8

 ; p0 = volatile *p;
 ; p[3] = (m >> 0) & 0xFF;
 ; p[2] = (m >> 8) & 0xFF;
 ; p[1] = (m >> 16) & 0xFF;
 ; *p0 = (m >> 24) & 0xFF;
 define void @store_i32_by_i8_bswap_volatile(i32 signext %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_volatile:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 3(4)
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
-; CHECK-PPC64LE-NEXT:    blr
+; CHECK-PPC64LE-NOT:   stwbrx 
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_volatile:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 3, 3(4)
-; CHECK-PPC64-NEXT:    stb 5, 2(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
-; CHECK-PPC64-NEXT:    blr
+; CHECK-PPC64-NOT:   stw 
 entry:
   %conv = trunc i32 %m to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 3
   store i8 %conv, i8* %arrayidx, align 1
   %0 = lshr i32 %m, 8
   %conv3 = trunc i32 %0 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 2
   store i8 %conv3, i8* %arrayidx4, align 1

 ; p[3] = (m >> 0) & 0xFF;
 ; p[2] = (m >> 8) & 0xFF;
 ; *q = 3;
 ; p[1] = (m >> 16) & 0xFF;
 ; p[0] = (m >> 24) & 0xFF;
 define void @store_i32_by_i8_bswap_store_in_between(i32 signext %m, i8* %p, i8* %q) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_store_in_between:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 6, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 3(4)
-; CHECK-PPC64LE-NEXT:    stb 6, 2(4)
-; CHECK-PPC64LE-NEXT:    li 6, 3
-; CHECK-PPC64LE-NEXT:    stb 6, 0(5)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
-; CHECK-PPC64LE-NEXT:    blr
+; CHECK-PPC64LE-NOT:   stwbrx 
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_store_in_between:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    li 6, 3
-; CHECK-PPC64-NEXT:    srwi 7, 3, 8
-; CHECK-PPC64-NEXT:    stb 7, 2(4)
-; CHECK-PPC64-NEXT:    stb 3, 3(4)
-; CHECK-PPC64-NEXT:    stb 6, 0(5)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
-; CHECK-PPC64-NEXT:    blr
+; CHECK-PPC64-NOT:   stw 
 entry:
   %conv = trunc i32 %m to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 3
   store i8 %conv, i8* %arrayidx, align 1
   %0 = lshr i32 %m, 8
   %conv3 = trunc i32 %0 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 2
   store i8 %conv3, i8* %arrayidx4, align 1
   store i8 3, i8* %q, align 1
   %1 = lshr i32 %m, 16
   %conv7 = trunc i32 %1 to i8
   %arrayidx8 = getelementptr inbounds i8, i8* %p, i64 1
   store i8 %conv7, i8* %arrayidx8, align 1
   %2 = lshr i32 %m, 24
   %conv11 = trunc i32 %2 to i8
   store i8 %conv11, i8* %p, align 1
   ret void
 }
 
 define void @store_i32_by_i8_bswap_unrelated_store(i32 signext %m, i8* %p, i8* %q) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_unrelated_store:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 6, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 3(4)
-; CHECK-PPC64LE-NEXT:    stb 6, 2(5)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 1(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
-; CHECK-PPC64LE-NEXT:    blr
+; CHECK-PPC64LE-NOT:   stwbrx 
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_unrelated_store:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 6, 3, 8
-; CHECK-PPC64-NEXT:    stb 3, 3(4)
-; CHECK-PPC64-NEXT:    stb 6, 2(5)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
-; CHECK-PPC64-NEXT:    blr
+; CHECK-PPC64-NOT:   stw 
 entry:
   %conv = trunc i32 %m to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 3
   store i8 %conv, i8* %arrayidx, align 1
   %0 = lshr i32 %m, 8
   %conv3 = trunc i32 %0 to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %q, i64 2
   store i8 %conv3, i8* %arrayidx4, align 1

 ; i8* p;
 ; p[3] = (m >> 8) & 0xFF;
 ; p[4] = (m >> 0) & 0xFF;
 ; p[2] = (m >> 16) & 0xFF;
 ; p[1] = (m >> 24) & 0xFF;
 define void @store_i32_by_i8_bswap_nonzero_offset(i32 signext %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_nonzero_offset:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, 3(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 4(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 2(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 1(4)
+; CHECK-PPC64LE-NEXT:    addi [[REG1:[0-9]+]], 4, 1
+; CHECK-PPC64LE-NEXT:    stwbrx 3, 0, [[REG1]] 
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_nonzero_offset:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 3, 4(4)
-; CHECK-PPC64-NEXT:    stb 5, 3(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, 2(4)
-; CHECK-PPC64-NEXT:    stb 3, 1(4)
+; CHECK-PPC64-NEXT:    stw 3, 1(4)
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %0 = lshr i32 %m, 8
   %conv = trunc i32 %0 to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 3
   store i8 %conv, i8* %arrayidx, align 1
   %conv3 = trunc i32 %m to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 4

 ; i8* p;
 ; p[-3] = (m >> 8) & 0xFF;
 ; p[-4] = (m >> 0) & 0xFF;
 ; p[-2] = (m >> 16) & 0xFF;
 ; p[-1] = (m >> 24) & 0xFF;
 define void @store_i32_by_i8_neg_offset(i32 signext %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_neg_offset:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, -3(4)
-; CHECK-PPC64LE-NEXT:    stb 3, -4(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, -2(4)
-; CHECK-PPC64LE-NEXT:    stb 3, -1(4)
+; CHECK-PPC64LE-NEXT:    stw 3, -4(4)
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_neg_offset:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 3, -4(4)
-; CHECK-PPC64-NEXT:    stb 5, -3(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 3, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, -2(4)
-; CHECK-PPC64-NEXT:    stb 3, -1(4)
+; CHECK-PPC64-NEXT:    addi [[REG1:[0-9]+]], 4, -4
+; CHECK-PPC64-NEXT:    stwbrx 3, 0, [[REG1]] 
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %0 = lshr i32 %m, 8
   %conv = trunc i32 %0 to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 -3
   store i8 %conv, i8* %arrayidx, align 1
   %conv3 = trunc i32 %m to i8
   %arrayidx4 = getelementptr inbounds i8, i8* %p, i64 -4

 ; i8* p;
 ; p[-3] = (m >> 16) & 0xFF;
 ; p[-4] = (m >> 24) & 0xFF;
 ; p[-2] = (m >> 8) & 0xFF;
 ; p[-1] = (m >> 0) & 0xFF;
 define void @store_i32_by_i8_bswap_neg_offset(i32 signext %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_neg_offset:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    stb 5, -3(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, -4(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, -2(4)
-; CHECK-PPC64LE-NEXT:    stb 3, -1(4)
+; CHECK-PPC64LE-NEXT:    addi [[REG1:[0-9]+]], 4, -4
+; CHECK-PPC64LE-NEXT:    stwbrx 3, 0, [[REG1]] 
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_neg_offset:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    srwi 6, 3, 24
-; CHECK-PPC64-NEXT:    stb 5, -3(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 6, -4(4)
-; CHECK-PPC64-NEXT:    stb 5, -2(4)
-; CHECK-PPC64-NEXT:    stb 3, -1(4)
+; CHECK-PPC64-NEXT:    stw 3, -4(4)
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %0 = lshr i32 %m, 16
   %conv = trunc i32 %0 to i8
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 -3
   store i8 %conv, i8* %arrayidx, align 1
   %1 = lshr i32 %m, 24
   %conv3 = trunc i32 %1 to i8

 ; i8* p;
 ; p[i-3] = (m >> 16) & 0xFF;
 ; p[i-4] = (m >> 24) & 0xFF;
 ; p[i-2] = (m >> 8) & 0xFF;
 ; p[i-1] = (m >> 0) & 0xFF;
 define void @store_i32_by_i8_bswap_base_index_offset(i32 %m, i32 %i, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_base_index_offset:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    extsw 4, 4
-; CHECK-PPC64LE-NEXT:    srwi 6, 3, 16
-; CHECK-PPC64LE-NEXT:    add 4, 5, 4
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 6, -3(4)
-; CHECK-PPC64LE-NEXT:    stb 5, -4(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, -2(4)
-; CHECK-PPC64LE-NEXT:    stb 3, -1(4)
+; CHECK-PPC64LE-NEXT:    extsw [[REG1:[0-9]+]], 4
+; CHECK-PPC64LE-NEXT:    add [[REG2:[0-9]+]], 5, [[REG1]] 
+; CHECK-PPC64LE-NEXT:    addi [[REG3:[0-9]+]], [[REG2]], -4
+; CHECK-PPC64LE-NEXT:    stwbrx 3, 0, [[REG3]] 
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_base_index_offset:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    extsw 4, 4
-; CHECK-PPC64-NEXT:    srwi 6, 3, 16
-; CHECK-PPC64-NEXT:    add 4, 5, 4
-; CHECK-PPC64-NEXT:    srwi 5, 3, 24
-; CHECK-PPC64-NEXT:    stb 6, -3(4)
-; CHECK-PPC64-NEXT:    srwi 6, 3, 8
-; CHECK-PPC64-NEXT:    stb 5, -4(4)
-; CHECK-PPC64-NEXT:    stb 6, -2(4)
-; CHECK-PPC64-NEXT:    stb 3, -1(4)
+; CHECK-PPC64-NEXT:    extsw [[REG1:[0-9]+]], 4
+; CHECK-PPC64-NEXT:    add [[REG2:[0-9]+]], 5, [[REG1]] 
+; CHECK-PPC64-NEXT:    stw 3, -4([[REG2]])
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %0 = lshr i32 %m, 16
   %conv = trunc i32 %0 to i8
   %sub = add nsw i32 %i, -3
   %idxprom = sext i32 %sub to i64
   %arrayidx = getelementptr inbounds i8, i8* %p, i64 %idxprom
   store i8 %conv, i8* %arrayidx, align 1

 ; i8 *p3 = p + i + 3;
 ; p0[3] = (m >> 24) & 0xFF;
 ; p1[3] = (m >> 16) & 0xFF;
 ; p2[3] = (m >> 8) & 0xFF;
 ; p3[3] = (m >> 0) & 0xFF;
 define void @store_i32_by_i8_bswap_complicated(i32 %m, i32 %i, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i32_by_i8_bswap_complicated:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    extsw 4, 4
-; CHECK-PPC64LE-NEXT:    add 4, 5, 4
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 24
-; CHECK-PPC64LE-NEXT:    stb 5, 3(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64LE-NEXT:    stb 5, 4(4)
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, 5(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 6(4)
+; CHECK-PPC64LE-NEXT:    extsw [[REG1:[0-9]+]], 4
+; CHECK-PPC64LE-NEXT:    add [[REG2:[0-9]+]], 5, [[REG1]] 
+; CHECK-PPC64LE-NEXT:    addi [[REG3:[0-9]+]], [[REG2]], 3 
+; CHECK-PPC64LE-NEXT:    stwbrx 3, 0, [[REG3]] 
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i32_by_i8_bswap_complicated:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    extsw 4, 4
-; CHECK-PPC64-NEXT:    srwi 6, 3, 24
-; CHECK-PPC64-NEXT:    add 4, 5, 4
-; CHECK-PPC64-NEXT:    srwi 5, 3, 16
-; CHECK-PPC64-NEXT:    stb 6, 3(4)
-; CHECK-PPC64-NEXT:    stb 5, 4(4)
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 5, 5(4)
-; CHECK-PPC64-NEXT:    stb 3, 6(4)
+; CHECK-PPC64-NEXT:    extsw [[REG1:[0-9]+]], 4
+; CHECK-PPC64-NEXT:    add [[REG2:[0-9]+]], 5, [[REG1]] 
+; CHECK-PPC64-NEXT:    stw 3, 3([[REG2]])
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %idx.ext = sext i32 %i to i64
   %add.ptr = getelementptr inbounds i8, i8* %p, i64 %idx.ext
   %add.ptr3 = getelementptr inbounds i8, i8* %add.ptr, i64 1
   %add.ptr6 = getelementptr inbounds i8, i8* %add.ptr, i64 2
   %add.ptr9 = getelementptr inbounds i8, i8* %add.ptr, i64 3
   %0 = lshr i32 %m, 24
entry:
   ret void
 }
 ; i8* p; i32 m;
 ; p[0] = (m >> 8) & 0xFF;
 ; p[1] = (m >> 0) & 0xFF;
 define void @store_i16_by_i8_bswap(i16 %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_i16_by_i8_bswap:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 5, 0(4)
-; CHECK-PPC64LE-NEXT:    stb 3, 1(4)
+; CHECK-PPC64LE-NEXT:    sthbrx 3, 0, 4
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_i16_by_i8_bswap:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 5, 0(4)
-; CHECK-PPC64-NEXT:    stb 3, 1(4)
+; CHECK-PPC64-NEXT:    sth 3, 0(4)
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %0 = lshr i16 %m, 8
   %conv1 = trunc i16 %0 to i8
   store i8 %conv1, i8* %p, align 1
   %conv5 = trunc i16 %m to i8
   %arrayidx6 = getelementptr inbounds i8, i8* %p, i64 1
   store i8 %conv5, i8* %arrayidx6, align 1
   ret void
 }
 ; i8* p; i32 m;
 ; p[0] = (m >> 0) & 0xFF;
 ; p[1] = (m >> 8) & 0xFF;
 define void @store_16_by_i8(i16 %m, i8* %p) {
 ; CHECK-PPC64LE-LABEL: store_16_by_i8:
 ; CHECK-PPC64LE:       # %bb.0: # %entry
-; CHECK-PPC64LE-NEXT:    stb 3, 0(4)
-; CHECK-PPC64LE-NEXT:    srwi 3, 3, 8
-; CHECK-PPC64LE-NEXT:    stb 3, 1(4)
+; CHECK-PPC64LE-NEXT:    sth 3, 0(4)
 ; CHECK-PPC64LE-NEXT:    blr
 ;
 ; CHECK-PPC64-LABEL: store_16_by_i8:
 ; CHECK-PPC64:       # %bb.0: # %entry
-; CHECK-PPC64-NEXT:    srwi 5, 3, 8
-; CHECK-PPC64-NEXT:    stb 3, 0(4)
-; CHECK-PPC64-NEXT:    stb 5, 1(4)
+; CHECK-PPC64-NEXT:    sthbrx 3, 0, 4
 ; CHECK-PPC64-NEXT:    blr
 entry:
   %conv1 = trunc i16 %m to i8
   store i8 %conv1, i8* %p, align 1
   %0 = lshr i16 %m, 8
   %conv5 = trunc i16 %0 to i8
   %arrayidx6 = getelementptr inbounds i8, i8* %p, i64 1
   store i8 %conv5, i8* %arrayidx6, align 1
   ret void
 }