[SLP] Look-ahead operand reordering heuristic.

[vporpo]

Summary

This patch introduces a new heuristic for guiding operand reordering. The new "look-ahead" heuristic can look beyond the immediate predecessors. This helps break ties when the immediate predecessors have identical opcodes (see lit test for an example).

[RKSimon]

lib/Transforms/Vectorize/SLPVectorizer.cpp

149

"slp-max-look-ahead-depth" might be better?

744

These kinds of hard coded costs scare me, especially without any context/description.

[vporpo]

lib/Transforms/Vectorize/SLPVectorizer.cpp

744

I agree, they look quite scary but the values themselves are not very important.
What we we really care about is figuring out whether the values could be potentially vectorized or not. The relative differences between the scores is not very important either. It just shows that matching loads is usually preferable to matching instructions with the same opcode etc.

We could still use the same score of 1 for all of them and we would still get decent reordering. Another alternative would be to check TTI for each potential candidate vector, but it looks like an overkill.

[ABataev]

Tests?

lib/Transforms/Vectorize/SLPVectorizer.cpp

756

Seems to me, the code is not formatted

757

auto *

758

auto *

764

auto *

765

auto *

769

auto *

770

auto *

815

auto *

816

auto *

[ABataev]

Better to commit the test itself at first, without this patch

[vporpo]

Rebased.

[RKSimon]

lib/Transforms/Vectorize/SLPVectorizer.cpp

760

(style) remove outer brackets

773

What happens in the case where we have alt opcodes? Should we have a preference for all the same opcode vs with alt-opcode? Sometimes the alt-opcodes will fold away (shl + mul etc.) - other times it won't (shl + lshr).

[vporpo]

Addressed comments and updated lit test.

lib/Transforms/Vectorize/SLPVectorizer.cpp

773

Hmm good point. Well, currently 'getScoreAtLevelRec()' will simply walk past the alt instructions and will assign them ScoreSameOpcode. This is does not look very accurate because alt opcodes usually require shuffles and should have a lower score.
I introduced a new ScoreAltOpcodes = 1 so that alt opcodes are not given the same score as identical opcodes (please see the new functions in lit test).

As for the alt opcodes that fold away, maybe that should be fixed in the getSameOpcode() and in struct InstructionState ? If the get folded then maybe isAltShuffle() should return false?

[vporpo]

Updated getBestOperand() to use getLookAheadScore() for Load and Constant, not just Opcode.

[RKSimon]

please can you rebase this?

[vporpo]

Rebased.

[dtemirbulatov]

lib/Transforms/Vectorize/SLPVectorizer.cpp

948

one extra space here.

[dtemirbulatov]

oh, I notice some regression with the change for AArch64/matmul.ll and AArch64/transpose.ll. Maybe there is a way to isolate it with heuristics?

[vporpo]

Yes, I will take a look. Maybe it is worth using TTI for the scores after all.

[vporpo]

I investigated the two AArch64 failing tests. These tests feature the exact problem that we are trying to solve with this look-ahead heuristic. A commutative instruction had operands of the same opcode that the current heuristic has no way of reordering in an informed way. The current reordering was just lucky to pick the proper one, while the look-ahead heuristic was reordering the operands according to the score. However, the problem was that the score calculation was not considering external uses and was therefore favoring a sub-optimal operand ordering.

I updated the patch to factor in the cost of external uses, and both failures are now gone. I also updated the lit-test with a test that shows the problem with the external-uses.

[vporpo]

Rebased.

[dtemirbulatov]

hmm, I have another failure with this change on my setup, now it is PR39774.ll. Probably, it might be sort algorithm differents of similar, since it just swap of two inserts. I don't have this failure if PR39774.ll is intact.

[vporpo]

Removed changes in PR39774.ll.

[vporpo]

Hmm I am now getting the same failure as you @dtemirbulatov . I am not sure what was wrong before, but it seems that the change in PR39774.ll is no longer needed.

[dtemirbulatov]

LGTM.

[vporpo]

@ABataev @RKSimon any comments?

[RKSimon]

@vporpo I've committed the lookahead.ll changes at rL363925 with current (trunk) codegen - please can you rebase?

[vporpo]

Rebased

[RKSimon]

LGTM - thanks!

[vporpo]

Thank you for the reviews. Please commit the patch.

[vporpo]

This was reverted in r364111 since it was causing a failure in Chromium reported by @rnk.

[RKSimon]

@rnk do you have a repro yet please?

[vporpo]

Yes, @RKSimon . It was posted in llvm-commits. I did reproduce it and I will update the patch with the fix + lit test.

[vporpo]

Fixed crash in chromium reported by @rnk.

The crash was caused by two call instructions with different number of arguments (see lookahead_crash() function in lit test).

[vporpo]

lib/Transforms/Vectorize/SLPVectorizer.cpp

901

This was the cause of the crash. There was no std::min here, so ToIdx could be OpIdx1 + 1 even if I2 had fewer operands than that.

[RKSimon]

LGTM

[phosek]

This change has caused a massive increase in build times when using LTO. On my workstations when building Clang toolchain itself with LTO, the build time has increased from 50 minutes to 5+ hours. On our continuous builders, we seen timeouts everywhere because they aren't able to finish within the allocated time (5 hours). Would it be possible to revert this change?

[RKSimon]

In D60897#1562427, @phosek wrote:

This change has caused a massive increase in build times when using LTO. On my workstations when building Clang toolchain itself with LTO, the build time has increased from 50 minutes to 5+ hours. On our continuous builders, we seen timeouts everywhere because they aren't able to finish within the allocated time (5 hours). Would it be possible to revert this change?

Reversion makes sense to me - @phosek are you in position to profile it to see where the time is going in SLP please?

[vporpo]

I think There are two possible causes for the compilation time increase:

1. Line 901 : We can restrict the number of operands to a max of 2
2. Line 820: We can restrict the visited users to a ma x of 2.

I can either create a quick patch, or I can revert it. Either is fine.

[phosek]

In D60897#1562519, @vporpo wrote:

I think There are two possible causes for the compilation time increase:

1. Line 901 : We can restrict the number of operands to a max of 2
2. Line 820: We can restrict the visited users to a ma x of 2.

I can either create a quick patch, or I can revert it. Either is fine.

If you can create a quick patch, I'd be happy to try it locally to see if it addresses the problem.

[vporpo]

Thanks, that would be really helpful @phosek . Let me create the quick patch.

[vporpo]

The fix for the compilation-time issue reported by @phosek is in D63948 .

[vporpo]

Fixed the compilation-time issue with the introduction of the user budget limit from D63948. Also added lit test for it.

[RKSimon]

LGTM

[Carrot]

Following is a simplified test case that shows the performance regression of eigen. The number of instructions in inner loop is increased from 85 to 93. Hope it can be helpful.

#include "third_party/eigen3/Eigen/Core"

template <typename ValueType>
inline void CheckNonnegative(ValueType value) {

if (!(value >= 0)) {
  fprintf(stderr, "Check failed.\n");
}

}

template <typename ValueType>
inline decltype(std::norm(ValueType(0))) SquaredMagnitude(ValueType value) {

typedef decltype(std::norm(ValueType(0))) ComponentType;
ComponentType r = std::real(value);
ComponentType i = std::imag(value);
return r * r + i * i;

}

template <typename VectorType2>
class DotEigen {
public:

DotEigen(): first_vector_(16), second_vector_(16) {
}

inline void Execute() {
  result_ = first_vector_.dot(second_vector_);
  CheckNonnegative(SquaredMagnitude(result_));
}

public:

Eigen::Matrix<float, 16, 1> first_vector_;
VectorType2 second_vector_;
std::complex<float> result_;

};

typedef DotEigen<Eigen::Matrix<std::complex<float>, 16, 1>> MOperation;
MOperation* operation;

void BM_Dot_RealComplex_EigenDotFixed(int num_iterations) {

for (int iter = 0; iter < num_iterations; ++iter) {
  operation->Execute();
}

}

[vporpo]

Thanks @Carrot, I will investigate the issue.

[RKSimon]

@Carrot What build settings are you using? I'm not seeing this here: https://godbolt.org/z/4jq0g8

[Carrot]

@RKSimon, my command line options are:

-O3 -m64 -msse4.2 -mpclmul -maes -mprefer-vector-width=128 -fexperimental-new-pass-manager -fPIE

I compared r364964 and r364963.

[vporpo]

This fixes the issue reported by @Carrot. It updates getShallowScore() to better cope with extracts from consecutive locations of the same vector (see last lit test). @Carrot, if you have the time please verify that this patch no longer causes a regression. Thanks!

[Carrot]

@vporpo, it's verified that the eigen test doesn't regress now, actually the inner loop is the same as before.
Thanks

[xbolva00]

Please reland this patch again.

[vporpo]

@RKSimon any comments about the latest changes?

[xbolva00]

Ping @RKSimon

[RKSimon]

@vporpo Please can you rebase this?

[vporpo]

Rebased.

[RKSimon]

LGTM - cheers

lib/Transforms/Vectorize/SLPVectorizer.cpp

 static cl::opt<unsigned> RecursionMaxDepth(
     "slp-recursion-max-depth", cl::init(12), cl::Hidden,
     cl::desc("Limit the recursion depth when building a vectorizable tree"));
 
 static cl::opt<unsigned> MinTreeSize(
     "slp-min-tree-size", cl::init(3), cl::Hidden,
     cl::desc("Only vectorize small trees if they are fully vectorizable"));
 
+// The maximum depth that the look-ahead score heuristic will explore.
+// The higher this value, the higher the compilation time overhead.
+static cl::opt<int> LookAheadMaxDepth(
+    "slp-max-look-ahead-depth", cl::init(2), cl::Hidden,
+    cl::desc("The maximum look-ahead depth for operand reordering scores"));
+
 static cl::opt<bool>
     ViewSLPTree("view-slp-tree", cl::Hidden,
                 cl::desc("Display the SLP trees with Graphviz"));
 
 // Limit the number of alias checks. The limit is chosen so that
 // it has no negative effect on the llvm benchmarks.
 static const unsigned AliasedCheckLimit = 10;
 
void clearUsed() {
           OpsVec[OpIdx][Lane].IsUsed = false;
     }
 
     /// Swap the operand at \p OpIdx1 with that one at \p OpIdx2.
     void swap(unsigned OpIdx1, unsigned OpIdx2, unsigned Lane) {
       std::swap(OpsVec[OpIdx1][Lane], OpsVec[OpIdx2][Lane]);
     }
 
+    // The hard-coded scores listed here are not very important. When computing
+    // the scores of matching one sub-tree with another, we are basically
+    // counting the number of values that are matching. So even if all scores
+    // are set to 1, we would still get a decent matching result.
+    // However, sometimes we have to break ties. For example we may have to
+    // choose between matching loads vs matching opcodes. This is what these
+    // scores are helping us with: they provide the order of preference.
+
+    /// Loads from consecutive memory addresses, e.g. load(A[i]), load(A[i+1]).
+    static const int ScoreConsecutiveLoads = 3;
+    /// Constants.
+    static const int ScoreConstants = 2;
+    /// Instructions with the same opcode.
+    static const int ScoreSameOpcode = 2;
+    /// Identical instructions (a.k.a. splat or broadcast).
+    static const int ScoreSplat = 1;
+    /// Matching with an undef is preferable to failing.
+    static const int ScoreUndef = 1;
+    /// Score for failing to find a decent match.
+    static const int ScoreFail = 0;
+
+    /// \returns the score of placing \p V1 and \p V2 in consecutive lanes.
+    static int getShallowScore(Value *V1, Value *V2, const DataLayout &DL,
+                               ScalarEvolution &SE) {
+      auto *LI1 = dyn_cast<LoadInst>(V1);
+      auto *LI2 = dyn_cast<LoadInst>(V2);
+      if (LI1 && LI2)
+        return (isConsecutiveAccess(LI1, LI2, DL, SE))
+                   ? VLOperands::ScoreConsecutiveLoads
+                   : VLOperands::ScoreFail;
+
+      auto *C1 = dyn_cast<Constant>(V1);
+      auto *C2 = dyn_cast<Constant>(V2);
+      if (C1 && C2)
+        return VLOperands::ScoreConstants;
+
+      auto *I1 = dyn_cast<Instruction>(V1);
+      auto *I2 = dyn_cast<Instruction>(V2);
+      if (I1 && I2) {
+        InstructionsState S = getSameOpcode({I1, I2});
+        if (S.getOpcode())
+          return (I1 == I2) ? VLOperands::ScoreSplat
+                            : VLOperands::ScoreSameOpcode;
+      }
+
+      if (isa<UndefValue>(V2))
+        return VLOperands::ScoreUndef;
+
+      return VLOperands::ScoreFail;
+    }
+
+    /// Go through the operands of \p V1 and \p V2 recursively until \p
+    /// MaxLevel, and return the cummulative score. For example:
+    /// \verbatim
+    ///  A[0]  B[0]  A[1]  B[1]  C[0] D[0]  B[1] A[1]
+    ///     \ /         \ /         \ /        \ /
+    ///      +           +           +          +
+    ///     G1          G2          G3         G4
+    /// \endverbatim
+    /// The getScoreAtLevelRec(G1, G2) function will try to match the nodes at
+    /// each level recursively, accumulating the score. It starts from matching
+    /// the additions at level 0, then moves on to the loads (level 1). The
+    /// score of G1 and G2 is higher than G1 and G3, because {A[0],A[1]} and
+    /// {B[0],B[1]} match with VLOperands::ScoreConsecutiveLoads, while
+    /// {A[0],C[0]} has a score of VLOperands::ScoreFail.
+    /// Please note that the order of the operands does not matter, as we
+    /// evaluate the score of all profitable combinations of operands. In
+    /// other words the score of G1 and G4 is the same as G1 and G2. This
+    /// heuristic is based on ideas described in:
+    ///   Look-ahead SLP: Auto-vectorization in the presence of commutative
+    ///   operations, CGO 2018 by Vasileios Porpodas, Rodrigo C. O. Rocha,
+    ///   Luís F. W. Góes
+    static int getScoreAtLevelRec(Value *V1, Value *V2, int CurrLevel,
+                                  int MaxLevel, const DataLayout &DL,
+                                  ScalarEvolution &SE) {
+      // Get the shallow score of V1 and V2.
+      int ShallowScoreAtThisLevel = getShallowScore(V1, V2, DL, SE);
+
+      // If reached MaxLevel,
+      //  or if V1 and V2 are not instructions,
+      //  or if they are SPLAT,
+      //  or if they are not consecutive, early return the current cost.
+      auto *I1 = dyn_cast<Instruction>(V1);
+      auto *I2 = dyn_cast<Instruction>(V2);
+      if (CurrLevel == MaxLevel || !(I1 && I2) || I1 == I2 ||
+          ShallowScoreAtThisLevel == VLOperands::ScoreFail ||
+          (isa<LoadInst>(I1) && isa<LoadInst>(I2) && ShallowScoreAtThisLevel))
+        return ShallowScoreAtThisLevel;
+      assert(I1 && I2 && "Should have early exited.");
+
+      // Contains the I2 operand indexes that got matched with I1 operands.
+      SmallSet<int, 4> Op2Used;
+
+      // Recursion towards the operands of I1 and I2. We are trying all possbile
+      // operand pairs, and keeping track of the best score.
+      for (int OpIdx1 = 0, NumOperands1 = I1->getNumOperands();
+           OpIdx1 != NumOperands1; ++OpIdx1) {
+        // Try to pair op1I with the best operand of I2.
+        int MaxTmpScore = 0;
+        int MaxOpIdx2 = -1;
+        for (int OpIdx2 = 0, NumOperands2 = I2->getNumOperands();
+             OpIdx2 != NumOperands2; ++OpIdx2) {
+          // Skip operands already paired with OpIdx1.
+          if (Op2Used.count(OpIdx2))
+            continue;
+          // Recursively calculate the cost at each level
+          int TmpScore =
+              getScoreAtLevelRec(I1->getOperand(OpIdx1), I2->getOperand(OpIdx2),
+                                 CurrLevel + 1, MaxLevel, DL, SE);
+          // Look for the best score.
+          if (TmpScore > VLOperands::ScoreFail && TmpScore > MaxTmpScore) {
+            MaxTmpScore = TmpScore;
+            MaxOpIdx2 = OpIdx2;
+          }
+        }
+        if (MaxOpIdx2 >= 0) {
+          // Pair {OpIdx1, MaxOpIdx2} was found to be best. Never revisit it.
+          Op2Used.insert(MaxOpIdx2);
+          ShallowScoreAtThisLevel += MaxTmpScore;
+        }
+      }
+      return ShallowScoreAtThisLevel;
+    }
+
+    /// \returns the look-ahead score, which tells us how much the sub-trees
+    /// rooted at \p LHS and \p RHS match, the more they match the higher the
+    /// score.
+    static int getLookAheadScore(Value *LHS, Value *RHS, const DataLayout &DL,
+                                 ScalarEvolution &SE) {
+      return getScoreAtLevelRec(LHS, RHS, 1, LookAheadMaxDepth, DL, SE);
+    }
+
     // Search all operands in Ops[*][Lane] for the one that matches best
     // Ops[OpIdx][LastLane] and return its opreand index.
     // If no good match can be found, return None.
     Optional<unsigned>
     getBestOperand(unsigned OpIdx, int Lane, int LastLane,
                    ArrayRef<ReorderingMode> ReorderingModes) {
       unsigned NumOperands = getNumOperands();
 
getBestOperand(unsigned OpIdx, int Lane, int LastLane,
       // Iterate through all unused operands and look for the best.
       for (unsigned Idx = 0; Idx != NumOperands; ++Idx) {
         // Get the operand at Idx and Lane.
         OperandData &OpData = getData(Idx, Lane);
         Value *Op = OpData.V;
         bool OpAPO = OpData.APO;
 
         // Skip already selected operands.
         if (OpData.IsUsed)
           continue;
 
         // Skip if we are trying to move the operand to a position with a
         // different opcode in the linearized tree form. This would break the
         // semantics.
         if (OpAPO != OpIdxAPO)
           continue;
 
         // Look for an operand that matches the current mode.
         switch (RMode) {
         case ReorderingMode::Load:
           if (isa<LoadInst>(Op)) {
             // Figure out which is left and right, so that we can check for
             // consecutive loads
             bool LeftToRight = Lane > LastLane;
             Value *OpLeft = (LeftToRight) ? OpLastLane : Op;
             Value *OpRight = (LeftToRight) ? Op : OpLastLane;
             if (isConsecutiveAccess(cast<LoadInst>(OpLeft),
                                     cast<LoadInst>(OpRight), DL, SE))
               BestOp.Idx = Idx;
           }
           break;
-        case ReorderingMode::Opcode:
-          // We accept both Instructions and Undefs, but with different scores.
-          if ((isa<Instruction>(Op) && isa<Instruction>(OpLastLane) &&
-               cast<Instruction>(Op)->getOpcode() ==
-                   cast<Instruction>(OpLastLane)->getOpcode()) ||
-              (isa<UndefValue>(OpLastLane) && isa<Instruction>(Op)) ||
-              isa<UndefValue>(Op)) {
-            // An instruction has a higher score than an undef.
-            unsigned Score = (isa<UndefValue>(Op)) ? GoodScore : BestScore;
-            if (Score > BestOp.Score) {
-              BestOp.Idx = Idx;
-              BestOp.Score = Score;
-            }
-          }
+        case ReorderingMode::Opcode: {
+          unsigned Score = getLookAheadScore(OpLastLane, Op, DL, SE);
            break;
+        }
         case ReorderingMode::Constant:
           if (isa<Constant>(Op)) {
             unsigned Score = (isa<UndefValue>(Op)) ? GoodScore : BestScore;
             if (Score > BestOp.Score) {
               BestOp.Idx = Idx;
               BestOp.Score = Score;
             }
           }
           break;
         case ReorderingMode::Splat:
           if (Op == OpLastLane)
             BestOp.Idx = Idx;
           break;
         case ReorderingMode::Failed:
           return None;
         }
       }
 
       if (BestOp.Idx) {
         getData(BestOp.Idx.getValue(), Lane).IsUsed = true;
         return BestOp.Idx;
       }
       // If we could not find a good match return None.
       return None;
     }

test/Transforms/SLPVectorizer/X86/lookahead.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -slp-vectorizer -S < %s -mtriple=x86_64-unknown-linux -mcpu=corei7-avx | FileCheck %s
+;
+; This checks the look-ahead operand reordering heuristic
+;
+; A[0] B[0] C[0] D[0]  C[1] D[1] A[1] B[1]
+;     \  /   \  /          \  /   \  /
+;       -     -              -     -
+;        \   /                \   /
+;          +                    +
+;          |                    |
+;         S[0]                 S[1]
+;
+define void @test(double* %array) {
+; CHECK-LABEL: @test(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[IDX0:%.*]] = getelementptr inbounds double, double* [[ARRAY:%.*]], i64 0
+; CHECK-NEXT:    [[IDX1:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 1
+; CHECK-NEXT:    [[IDX2:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 2
+; CHECK-NEXT:    [[IDX3:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 3
+; CHECK-NEXT:    [[IDX4:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 4
+; CHECK-NEXT:    [[IDX5:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 5
+; CHECK-NEXT:    [[IDX6:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 6
+; CHECK-NEXT:    [[IDX7:%.*]] = getelementptr inbounds double, double* [[ARRAY]], i64 7
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast double* [[IDX0]] to <2 x double>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x double>, <2 x double>* [[TMP0]], align 8
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast double* [[IDX2]] to <2 x double>*
+; CHECK-NEXT:    [[TMP3:%.*]] = load <2 x double>, <2 x double>* [[TMP2]], align 8
+; CHECK-NEXT:    [[TMP4:%.*]] = bitcast double* [[IDX4]] to <2 x double>*
+; CHECK-NEXT:    [[TMP5:%.*]] = load <2 x double>, <2 x double>* [[TMP4]], align 8
+; CHECK-NEXT:    [[TMP6:%.*]] = bitcast double* [[IDX6]] to <2 x double>*
+; CHECK-NEXT:    [[TMP7:%.*]] = load <2 x double>, <2 x double>* [[TMP6]], align 8
+; CHECK-NEXT:    [[TMP8:%.*]] = fsub fast <2 x double> [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    [[TMP9:%.*]] = fsub fast <2 x double> [[TMP5]], [[TMP7]]
+; CHECK-NEXT:    [[TMP10:%.*]] = fadd fast <2 x double> [[TMP8]], [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = bitcast double* [[IDX0]] to <2 x double>*
+; CHECK-NEXT:    store <2 x double> [[TMP10]], <2 x double>* [[TMP11]], align 8
+; CHECK-NEXT:    ret void
+;
+entry:
+  %idx0 = getelementptr inbounds double, double* %array, i64 0
+  %idx1 = getelementptr inbounds double, double* %array, i64 1
+  %idx2 = getelementptr inbounds double, double* %array, i64 2
+  %idx3 = getelementptr inbounds double, double* %array, i64 3
+  %idx4 = getelementptr inbounds double, double* %array, i64 4
+  %idx5 = getelementptr inbounds double, double* %array, i64 5
+  %idx6 = getelementptr inbounds double, double* %array, i64 6
+  %idx7 = getelementptr inbounds double, double* %array, i64 7
+
+
+  %A_0 = load double, double *%idx0, align 8
+  %A_1 = load double, double *%idx1, align 8
+  %B_0 = load double, double *%idx2, align 8
+  %B_1 = load double, double *%idx3, align 8
+  %C_0 = load double, double *%idx4, align 8
+  %C_1 = load double, double *%idx5, align 8
+  %D_0 = load double, double *%idx6, align 8
+  %D_1 = load double, double *%idx7, align 8
+
+  %subAB_0 = fsub fast double %A_0, %B_0
+  %subCD_0 = fsub fast double %C_0, %D_0
+
+  %subAB_1 = fsub fast double %A_1, %B_1
+  %subCD_1 = fsub fast double %C_1, %D_1
+
+  %addABCD_0 = fadd fast double %subAB_0, %subCD_0
+  %addCDAB_1 = fadd fast double %subCD_1, %subAB_1
+
+  store double %addABCD_0, double *%idx0, align 8
+  store double %addCDAB_1, double *%idx1, align 8
+  ret void
+}