This is an archive of the discontinued LLVM Phabricator instance.

Differential D133657

[Matrix] Optimize matrix transposes around additions
ClosedPublic

Authored by thegameg on Sep 10 2022, 10:44 PM.

Details

Reviewers
fhahn
zjaffal
Commits
rGda09b35334ab: [Matrix] Optimize matrix transposes around additions
Summary

First, sink the transposes to the operands to simplify redudant
ones. Then, lift them to reduce the number of realized transposes.

(A + B)^T -> A^T + B^T -> (A + B)^T

See tests for more examples.

Diff Detail

Unit TestsFailed

TimeTest
60,050 msx64 debian > MLIR.Examples/standalone::test.toy

Event Timeline

thegameg created this revision.Sep 10 2022, 10:44 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 10 2022, 10:44 PM
Herald added subscribers: tschuett, hiraditya. · View Herald Transcript
thegameg requested review of this revision.Sep 10 2022, 10:44 PM
Harbormaster completed remote builds in B186044: Diff 459326.Sep 10 2022, 10:45 PM
thegameg added a parent revision: D133656: [Matrix] Add tests for addition transpose optimizations.Sep 10 2022, 10:46 PM
fhahn added inline comments.Sep 26 2022, 6:41 AM
llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp
747

Now also supports adds.

832

Could the lambda just return Value * instead of Instruction *? setShapeInfo and ReplaceAllUsesWith should work with Value *?

849

Now also handles TT additions.

877

as the loop body gets bigger, it seems like it would be good to move this to a separate function to reduce the indentation level.

Same for the other loop above.

882

Should we introduce a new m_Transpose helper to make this a bit more compact?

891

This is the same as for the earlier if, right? Can we share this cleanup code? It's easy to miss

thegameg updated this revision to Diff 463029.Sep 26 2022, 3:17 PM
thegameg marked 4 inline comments as done.
thegameg added inline comments.
llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp
832

It could, but we would need to then downcast back to Instruction for the iterator update:

// If we replaced I with a new instruction, continue from there.
if (NewInst)
  II = std::next(BasicBlock::reverse_iterator(NewInst));
882

Good idea, let me do it in a separate patch so it doesn't clutter the diff here.

Harbormaster completed remote builds in B188801: Diff 463029.Sep 26 2022, 3:56 PM
thegameg updated this revision to Diff 465468.Oct 5 2022, 10:55 AM

Rebase for build to pass.

Harbormaster completed remote builds in B190538: Diff 465468.Oct 5 2022, 12:25 PM
thegameg added a comment.Oct 11 2022, 2:32 PM

Ping?

thegameg added a comment.Nov 28 2022, 8:53 AM

ping?

thegameg added a comment.Dec 13 2022, 2:22 PM

ping?

fhahn accepted this revision.Jan 11 2023, 8:07 AM

LGTM, thanks!

This revision is now accepted and ready to land.Jan 11 2023, 8:07 AM
Herald added a subscriber: StephenFan. · View Herald TranscriptJan 11 2023, 8:07 AM
thegameg updated this revision to Diff 488279.Jan 11 2023, 10:26 AM

Rebase.

Harbormaster completed remote builds in B207135: Diff 488279.Jan 11 2023, 1:27 PM
This revision was landed with ongoing or failed builds.Jan 11 2023, 3:23 PM
Closed by commit rGda09b35334ab: [Matrix] Optimize matrix transposes around additions (authored by thegameg). · Explain Why
This revision was automatically updated to reflect the committed changes.
thegameg added a commit: rGda09b35334ab: [Matrix] Optimize matrix transposes around additions.

Revision Contents

PathSize
llvm/
lib/
Transforms/
Scalar/
43 lines
test/
Transforms/
LowerMatrixIntrinsics/
36 lines
125 lines

Diff 459326

llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp

Show First 20 Lines • Show All 99 Lines • ▼ Show 20 Lines
} }
/// Match any mul operation (fp or integer). /// Match any mul operation (fp or integer).
template <typename LTy, typename RTy> template <typename LTy, typename RTy>
auto m_AnyMul(const LTy &L, const RTy &R) { auto m_AnyMul(const LTy &L, const RTy &R) {
return m_CombineOr(m_Mul(L, R), m_FMul(L, R)); return m_CombineOr(m_Mul(L, R), m_FMul(L, R));
} }
/// Match any add operation (fp or integer).
template <typename LTy, typename RTy>
auto m_AnyAdd(const LTy &L, const RTy &R) {
return m_CombineOr(m_Add(L, R), m_FAdd(L, R));
}
namespace { namespace {
// Given an element pointer \p BasePtr to the start of a (sub) matrix, compute // Given an element pointer \p BasePtr to the start of a (sub) matrix, compute
// the start address of vector \p VecIdx with type (\p EltType x \p NumElements) // the start address of vector \p VecIdx with type (\p EltType x \p NumElements)
// assuming \p Stride elements between start two consecutive vectors. // assuming \p Stride elements between start two consecutive vectors.
// \p Stride must be >= \p NumElements. // \p Stride must be >= \p NumElements.
// For column-major matrixes, the function computes the address of a column // For column-major matrixes, the function computes the address of a column
// vectors and \p NumElements must be set to the number of elements in a column // vectors and \p NumElements must be set to the number of elements in a column
▲ Show 20 Lines • Show All 617 Lines • ▼ Show 20 Lines auto ReplaceAllUsesWith = [this](Instruction &Old, Value *New) {
if (S != ShapeMap.end()) { if (S != ShapeMap.end()) {
ShapeMap.erase(S); ShapeMap.erase(S);
if (supportsShapeInfo(New)) if (supportsShapeInfo(New))
ShapeMap.insert({New, S->second}); ShapeMap.insert({New, S->second});
} }
Old.replaceAllUsesWith(New); Old.replaceAllUsesWith(New);
}; };
// First sink all transposes inside matmuls, hoping that we end up with NN, // First sink all transposes inside matmuls, hoping that we end up with NN,
fhahnUnsubmitted
Done
ReplyInline Actions

Now also supports adds.

fhahn: Now also supports adds.
// NT or TN variants. // NT or TN variants.
for (BasicBlock &BB : reverse(Func)) { for (BasicBlock &BB : reverse(Func)) {
for (auto II = BB.rbegin(); II != BB.rend();) { for (auto II = BB.rbegin(); II != BB.rend();) {
Instruction &I = *II; Instruction &I = *II;
// We may remove II. By default continue on the next/prev instruction. // We may remove II. By default continue on the next/prev instruction.
++II; ++II;
// If we were to erase II, move again. // If we were to erase II, move again.
auto EraseFromParent = [&II, &BB](Value *V) { auto EraseFromParent = [&II, &BB](Value *V) {
▲ Show 20 Lines • Show All 61 Lines • ▼ Show 20 Lines for (BasicBlock &BB : reverse(Func)) {
: LocalBuilder.CreateMul(T0, T1, "mmul"); : LocalBuilder.CreateMul(T0, T1, "mmul");
auto *Result = cast<Instruction>(Mul); auto *Result = cast<Instruction>(Mul);
setShapeInfo(Result, Shape0); setShapeInfo(Result, Shape0);
return Result; return Result;
}); });
ReplaceAllUsesWith(I, NewInst); ReplaceAllUsesWith(I, NewInst);
EraseFromParent(&I); EraseFromParent(&I);
EraseFromParent(TA); EraseFromParent(TA);
// (A + B)^t -> A^t + B^t
// RxC RxC CxR CxR
} else if (match(TA, m_AnyAdd(m_Value(TAMA), m_Value(TAMB)))) {
IRBuilder<> LocalBuilder(&I);
NewInst = distributeTransposes(
TAMA, {R, C}, TAMB, {R, C}, Builder,
[&](Value *T0, ShapeInfo Shape0, Value *T1, ShapeInfo Shape1) {
auto *FAdd = cast<Instruction>(
fhahnUnsubmitted
Not Done
ReplyInline Actions

Could the lambda just return Value * instead of Instruction *? setShapeInfo and ReplaceAllUsesWith should work with Value *?

fhahn: Could the lambda just return `Value *` instead of `Instruction *`? `setShapeInfo` and…
thegamegAuthorUnsubmitted
Done
ReplyInline Actions

It could, but we would need to then downcast back to Instruction for the iterator update:

// If we replaced I with a new instruction, continue from there.
if (NewInst)
  II = std::next(BasicBlock::reverse_iterator(NewInst));
thegameg: It could, but we would need to then downcast back to `Instruction` for the iterator update…
LocalBuilder.CreateFAdd(T0, T1, "mfadd"));
setShapeInfo(FAdd, Shape0);
return FAdd;
});
ReplaceAllUsesWith(I, NewInst);
EraseFromParent(&I);
EraseFromParent(TA);
} }
} }
// If we replaced I with a new instruction, continue from there. // If we replaced I with a new instruction, continue from there.
if (NewInst) if (NewInst)
II = std::next(BasicBlock::reverse_iterator(NewInst)); II = std::next(BasicBlock::reverse_iterator(NewInst));
} }
} }
// If we have a TT matmul, lift the transpose. We may be able to fold into // If we have a TT matmul, lift the transpose. We may be able to fold into
fhahnUnsubmitted
Done
ReplyInline Actions

Now also handles TT additions.

fhahn: Now also handles TT additions.
// consuming multiply. // consuming multiply.
for (BasicBlock &BB : Func) { for (BasicBlock &BB : Func) {
for (Instruction &I : llvm::make_early_inc_range(BB)) { for (Instruction &I : llvm::make_early_inc_range(BB)) {
Value *A, *B, *AT, *BT; Value *A, *B, *AT, *BT;
ConstantInt *R, *K, *C; ConstantInt *R, *K, *C;
// A^t * B ^t -> (B * A)^t // A^t * B ^t -> (B * A)^t
if (match(&I, m_Intrinsic<Intrinsic::matrix_multiply>( if (match(&I, m_Intrinsic<Intrinsic::matrix_multiply>(
m_Value(A), m_Value(B), m_ConstantInt(R), m_Value(A), m_Value(B), m_ConstantInt(R),
Show All 10 Lines for (BasicBlock &BB : Func) {
ReplaceAllUsesWith(I, NewInst); ReplaceAllUsesWith(I, NewInst);
if (I.use_empty()) if (I.use_empty())
I.eraseFromParent(); I.eraseFromParent();
if (A->use_empty()) if (A->use_empty())
cast<Instruction>(A)->eraseFromParent(); cast<Instruction>(A)->eraseFromParent();
if (A != B && B->use_empty()) if (A != B && B->use_empty())
cast<Instruction>(B)->eraseFromParent(); cast<Instruction>(B)->eraseFromParent();
} }
// A^t + B ^t -> (A + B)^t
else if (match(&I, m_FAdd(m_Value(A), m_Value(B))) &&
fhahnUnsubmitted
Done
ReplyInline Actions

as the loop body gets bigger, it seems like it would be good to move this to a separate function to reduce the indentation level.

Same for the other loop above.

fhahn: as the loop body gets bigger, it seems like it would be good to move this to a separate…
match(A,
m_Intrinsic<Intrinsic::matrix_transpose>(
m_Value(AT), m_ConstantInt(R), m_ConstantInt(C))) &&
match(B,
m_Intrinsic<Intrinsic::matrix_transpose>(
fhahnUnsubmitted
Not Done
ReplyInline Actions

Should we introduce a new m_Transpose helper to make this a bit more compact?

fhahn: Should we introduce a new `m_Transpose` helper to make this a bit more compact?
thegamegAuthorUnsubmitted
Done
ReplyInline Actions

Good idea, let me do it in a separate patch so it doesn't clutter the diff here.

thegameg: Good idea, let me do it in a separate patch so it doesn't clutter the diff here.
m_Value(BT), m_ConstantInt(R), m_ConstantInt(C)))) {
IRBuilder<> Builder(&I);
Value *Add = cast<Instruction>(Builder.CreateFAdd(AT, BT, "mfadd"));
setShapeInfo(Add, {C, R});
MatrixBuilder MBuilder(Builder);
Instruction *NewInst = MBuilder.CreateMatrixTranspose(
Add, C->getZExtValue(), R->getZExtValue(), "mfadd_t");
ReplaceAllUsesWith(I, NewInst);
if (I.use_empty())
fhahnUnsubmitted
Done
ReplyInline Actions

This is the same as for the earlier if, right? Can we share this cleanup code? It's easy to miss

fhahn: This is the same as for the earlier if, right? Can we share this cleanup code? It's easy to miss
I.eraseFromParent();
if (A->use_empty())
cast<Instruction>(A)->eraseFromParent();
if (A != B && B->use_empty())
cast<Instruction>(B)->eraseFromParent();
}
} }
} }
} }
bool Visit() { bool Visit() {
SmallVector<Instruction *, 32> WorkList; SmallVector<Instruction *, 32> WorkList;
// Initially only the shape of matrix intrinsics is known. // Initially only the shape of matrix intrinsics is known.
▲ Show 20 Lines • Show All 1,568 Lines • Show Last 20 Lines

llvm/test/Transforms/LowerMatrixIntrinsics/after-transpose-opts.ll

Show First 20 Lines • Show All 94 Lines • ▼ Show 20 Lines
} }
; A^T + B^T -> (A + B)^T ; A^T + B^T -> (A + B)^T
define void @at_plus_bt(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %C) { define void @at_plus_bt(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %C) {
; CHECK-LABEL: @at_plus_bt( ; CHECK-LABEL: @at_plus_bt(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128 ; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128
; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128 ; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128
; CHECK-NEXT: [[AT:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[A]], i32 3, i32 3) ; CHECK-NEXT: [[MFADD:%.*]] = fadd <9 x double> [[A]], [[B]]
; CHECK-NEXT: [[BT:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[B]], i32 3, i32 3) ; CHECK-NEXT: [[MFADD_T:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[MFADD]], i32 3, i32 3)
; CHECK-NEXT: [[FADD:%.*]] = fadd <9 x double> [[AT]], [[BT]] ; CHECK-NEXT: store <9 x double> [[MFADD_T]], <9 x double>* [[C:%.*]], align 128
; CHECK-NEXT: store <9 x double> [[FADD]], <9 x double>* [[C:%.*]], align 128
; CHECK-NEXT: ret void ; CHECK-NEXT: ret void
; ;
entry: entry:
%a = load <9 x double>, <9 x double>* %Aptr %a = load <9 x double>, <9 x double>* %Aptr
%b = load <9 x double>, <9 x double>* %Bptr %b = load <9 x double>, <9 x double>* %Bptr
%at = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %a, i32 3, i32 3) %at = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %a, i32 3, i32 3)
%bt = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %b, i32 3, i32 3) %bt = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %b, i32 3, i32 3)
%fadd = fadd <9 x double> %at, %bt %fadd = fadd <9 x double> %at, %bt
store <9 x double> %fadd, <9 x double>* %C store <9 x double> %fadd, <9 x double>* %C
ret void ret void
} }
; (A + B)^T -> A^T + B^T -> (A + B)^T ; (A + B)^T -> A^T + B^T -> (A + B)^T
define void @a_plus_b_t(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %C) { define void @a_plus_b_t(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %C) {
; CHECK-LABEL: @a_plus_b_t( ; CHECK-LABEL: @a_plus_b_t(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128 ; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128
; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128 ; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128
; CHECK-NEXT: [[FADD:%.*]] = fadd <9 x double> [[A]], [[B]] ; CHECK-NEXT: [[MFADD1:%.*]] = fadd <9 x double> [[A]], [[B]]
; CHECK-NEXT: [[T:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[FADD]], i32 3, i32 3) ; CHECK-NEXT: [[MFADD_T:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[MFADD1]], i32 3, i32 3)
; CHECK-NEXT: store <9 x double> [[T]], <9 x double>* [[C:%.*]], align 128 ; CHECK-NEXT: store <9 x double> [[MFADD_T]], <9 x double>* [[C:%.*]], align 128
; CHECK-NEXT: ret void ; CHECK-NEXT: ret void
; ;
entry: entry:
%a = load <9 x double>, <9 x double>* %Aptr %a = load <9 x double>, <9 x double>* %Aptr
%b = load <9 x double>, <9 x double>* %Bptr %b = load <9 x double>, <9 x double>* %Bptr
%fadd = fadd <9 x double> %a, %b %fadd = fadd <9 x double> %a, %b
%t = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %fadd, i32 3, i32 3) %t = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %fadd, i32 3, i32 3)
store <9 x double> %t, <9 x double>* %C store <9 x double> %t, <9 x double>* %C
ret void ret void
} }
; A^T * B^T + C^T * D^T -> (B * A + D * C)^T ; A^T * B^T + C^T * D^T -> (B * A + D * C)^T
define void @atbt_plus_ctdt(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %Cptr, <9 x double>* %Dptr, <9 x double>* %E) { define void @atbt_plus_ctdt(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %Cptr, <9 x double>* %Dptr, <9 x double>* %E) {
; CHECK-LABEL: @atbt_plus_ctdt( ; CHECK-LABEL: @atbt_plus_ctdt(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128 ; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128
; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128 ; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128
; CHECK-NEXT: [[C:%.*]] = load <9 x double>, <9 x double>* [[CPTR:%.*]], align 128 ; CHECK-NEXT: [[C:%.*]] = load <9 x double>, <9 x double>* [[CPTR:%.*]], align 128
; CHECK-NEXT: [[D:%.*]] = load <9 x double>, <9 x double>* [[DPTR:%.*]], align 128 ; CHECK-NEXT: [[D:%.*]] = load <9 x double>, <9 x double>* [[DPTR:%.*]], align 128
; CHECK-NEXT: [[TMP0:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[B]], <9 x double> [[A]], i32 3, i32 3, i32 3) ; CHECK-NEXT: [[TMP0:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[B]], <9 x double> [[A]], i32 3, i32 3, i32 3)
; CHECK-NEXT: [[TMP1:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[TMP0]], i32 3, i32 3) ; CHECK-NEXT: [[TMP1:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[D]], <9 x double> [[C]], i32 3, i32 3, i32 3)
; CHECK-NEXT: [[TMP2:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[D]], <9 x double> [[C]], i32 3, i32 3, i32 3) ; CHECK-NEXT: [[MFADD:%.*]] = fadd <9 x double> [[TMP0]], [[TMP1]]
; CHECK-NEXT: [[TMP3:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[TMP2]], i32 3, i32 3) ; CHECK-NEXT: [[MFADD_T:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[MFADD]], i32 3, i32 3)
; CHECK-NEXT: [[FADD:%.*]] = fadd <9 x double> [[TMP1]], [[TMP3]] ; CHECK-NEXT: store <9 x double> [[MFADD_T]], <9 x double>* [[E:%.*]], align 128
; CHECK-NEXT: store <9 x double> [[FADD]], <9 x double>* [[E:%.*]], align 128
; CHECK-NEXT: ret void ; CHECK-NEXT: ret void
; ;
entry: entry:
%a = load <9 x double>, <9 x double>* %Aptr %a = load <9 x double>, <9 x double>* %Aptr
%b = load <9 x double>, <9 x double>* %Bptr %b = load <9 x double>, <9 x double>* %Bptr
%c = load <9 x double>, <9 x double>* %Cptr %c = load <9 x double>, <9 x double>* %Cptr
%d = load <9 x double>, <9 x double>* %Dptr %d = load <9 x double>, <9 x double>* %Dptr
%at = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %a, i32 3, i32 3) %at = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %a, i32 3, i32 3)
Show All 34 Lines
; (A^T * B^T + k * C^T * D^T)^T -> (B * A) + (D * C * k) ; (A^T * B^T + k * C^T * D^T)^T -> (B * A) + (D * C * k)
define void @atbt_plus_kctdt_t(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %Cptr, <9 x double>* %Dptr, double %k, <9 x double>* %E) { define void @atbt_plus_kctdt_t(<9 x double>* %Aptr, <9 x double>* %Bptr, <9 x double>* %Cptr, <9 x double>* %Dptr, double %k, <9 x double>* %E) {
; CHECK-LABEL: @atbt_plus_kctdt_t( ; CHECK-LABEL: @atbt_plus_kctdt_t(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128 ; CHECK-NEXT: [[A:%.*]] = load <9 x double>, <9 x double>* [[APTR:%.*]], align 128
; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128 ; CHECK-NEXT: [[B:%.*]] = load <9 x double>, <9 x double>* [[BPTR:%.*]], align 128
; CHECK-NEXT: [[C:%.*]] = load <9 x double>, <9 x double>* [[CPTR:%.*]], align 128 ; CHECK-NEXT: [[C:%.*]] = load <9 x double>, <9 x double>* [[CPTR:%.*]], align 128
; CHECK-NEXT: [[D:%.*]] = load <9 x double>, <9 x double>* [[DPTR:%.*]], align 128 ; CHECK-NEXT: [[D:%.*]] = load <9 x double>, <9 x double>* [[DPTR:%.*]], align 128
; CHECK-NEXT: [[CT:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[C]], i32 3, i32 3)
; CHECK-NEXT: [[DT:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[D]], i32 3, i32 3)
; CHECK-NEXT: [[TMP0:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[B]], <9 x double> [[A]], i32 3, i32 3, i32 3)
; CHECK-NEXT: [[TMP1:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[TMP0]], i32 3, i32 3)
; CHECK-NEXT: [[VECK:%.*]] = insertelement <9 x double> poison, double [[K:%.*]], i64 0 ; CHECK-NEXT: [[VECK:%.*]] = insertelement <9 x double> poison, double [[K:%.*]], i64 0
; CHECK-NEXT: [[SPLAT:%.*]] = shufflevector <9 x double> [[VECK]], <9 x double> poison, <9 x i32> zeroinitializer ; CHECK-NEXT: [[SPLAT:%.*]] = shufflevector <9 x double> [[VECK]], <9 x double> poison, <9 x i32> zeroinitializer
; CHECK-NEXT: [[KCT:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[SPLAT]], <9 x double> [[CT]], i32 3, i32 3, i32 3) ; CHECK-NEXT: [[MMUL2:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[B]], <9 x double> [[A]], i32 3, i32 3, i32 3)
; CHECK-NEXT: [[KCTDT:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[KCT]], <9 x double> [[DT]], i32 3, i32 3, i32 3) ; CHECK-NEXT: [[MMUL1:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[C]], <9 x double> [[SPLAT]], i32 3, i32 3, i32 3)
; CHECK-NEXT: [[FADD:%.*]] = fadd <9 x double> [[TMP1]], [[KCTDT]] ; CHECK-NEXT: [[MMUL:%.*]] = call <9 x double> @llvm.matrix.multiply.v9f64.v9f64.v9f64(<9 x double> [[D]], <9 x double> [[MMUL1]], i32 3, i32 3, i32 3)
; CHECK-NEXT: [[T:%.*]] = call <9 x double> @llvm.matrix.transpose.v9f64(<9 x double> [[FADD]], i32 3, i32 3) ; CHECK-NEXT: [[MFADD:%.*]] = fadd <9 x double> [[MMUL2]], [[MMUL]]
; CHECK-NEXT: store <9 x double> [[T]], <9 x double>* [[E:%.*]], align 128 ; CHECK-NEXT: store <9 x double> [[MFADD]], <9 x double>* [[E:%.*]], align 128
; CHECK-NEXT: ret void ; CHECK-NEXT: ret void
; ;
entry: entry:
%a = load <9 x double>, <9 x double>* %Aptr %a = load <9 x double>, <9 x double>* %Aptr
%b = load <9 x double>, <9 x double>* %Bptr %b = load <9 x double>, <9 x double>* %Bptr
%c = load <9 x double>, <9 x double>* %Cptr %c = load <9 x double>, <9 x double>* %Cptr
%d = load <9 x double>, <9 x double>* %Dptr %d = load <9 x double>, <9 x double>* %Dptr
%at = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %a, i32 3, i32 3) %at = call <9 x double> @llvm.matrix.transpose.v9f64.v9f64(<9 x double> %a, i32 3, i32 3)
▲ Show 20 Lines • Show All 44 Lines • Show Last 20 Lines

llvm/test/Transforms/LowerMatrixIntrinsics/propagate-backward.ll

; NOTE: Assertions have been autogenerated by utils/update_test_checks.py ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
; RUN: opt -matrix-allow-contract=false -passes='lower-matrix-intrinsics' -S < %s | FileCheck %s ; RUN: opt -matrix-allow-contract=false -passes='lower-matrix-intrinsics' -S < %s | FileCheck %s
define <8 x double> @fadd_transpose(<8 x double> %a, <8 x double> %b) { define <8 x double> @fadd_transpose(<8 x double> %a, <8 x double> %b) {
; CHECK-LABEL: @fadd_transpose( ; CHECK-LABEL: @fadd_transpose(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[SPLIT:%.*]] = shufflevector <8 x double> [[A:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1> ; CHECK-NEXT: [[SPLIT:%.*]] = shufflevector <8 x double> [[A:%.*]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[SPLIT1:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 2, i32 3> ; CHECK-NEXT: [[SPLIT2:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[SPLIT2:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 4, i32 5> ; CHECK-NEXT: [[SPLIT3:%.*]] = shufflevector <8 x double> [[B:%.*]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[SPLIT3:%.*]] = shufflevector <8 x double> [[A]], <8 x double> poison, <2 x i32> <i32 6, i32 7> ; CHECK-NEXT: [[SPLIT4:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[SPLIT4:%.*]] = shufflevector <8 x double> [[B:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1> ; CHECK-NEXT: [[TMP0:%.*]] = fadd <4 x double> [[SPLIT]], [[SPLIT3]]
; CHECK-NEXT: [[SPLIT5:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 2, i32 3> ; CHECK-NEXT: [[TMP1:%.*]] = fadd <4 x double> [[SPLIT2]], [[SPLIT4]]
; CHECK-NEXT: [[SPLIT6:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 4, i32 5> ; CHECK-NEXT: [[TMP2:%.*]] = extractelement <4 x double> [[TMP0]], i64 0
; CHECK-NEXT: [[SPLIT7:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 6, i32 7> ; CHECK-NEXT: [[TMP3:%.*]] = insertelement <2 x double> poison, double [[TMP2]], i64 0
; CHECK-NEXT: [[TMP0:%.*]] = fadd <2 x double> [[SPLIT]], [[SPLIT4]] ; CHECK-NEXT: [[TMP4:%.*]] = extractelement <4 x double> [[TMP1]], i64 0
; CHECK-NEXT: [[TMP1:%.*]] = fadd <2 x double> [[SPLIT1]], [[SPLIT5]] ; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x double> [[TMP3]], double [[TMP4]], i64 1
; CHECK-NEXT: [[TMP2:%.*]] = fadd <2 x double> [[SPLIT2]], [[SPLIT6]] ; CHECK-NEXT: [[TMP6:%.*]] = extractelement <4 x double> [[TMP0]], i64 1
; CHECK-NEXT: [[TMP3:%.*]] = fadd <2 x double> [[SPLIT3]], [[SPLIT7]] ; CHECK-NEXT: [[TMP7:%.*]] = insertelement <2 x double> poison, double [[TMP6]], i64 0
; CHECK-NEXT: [[TMP4:%.*]] = extractelement <2 x double> [[TMP0]], i64 0 ; CHECK-NEXT: [[TMP8:%.*]] = extractelement <4 x double> [[TMP1]], i64 1
; CHECK-NEXT: [[TMP5:%.*]] = insertelement <4 x double> poison, double [[TMP4]], i64 0 ; CHECK-NEXT: [[TMP9:%.*]] = insertelement <2 x double> [[TMP7]], double [[TMP8]], i64 1
; CHECK-NEXT: [[TMP6:%.*]] = extractelement <2 x double> [[TMP1]], i64 0 ; CHECK-NEXT: [[TMP10:%.*]] = extractelement <4 x double> [[TMP0]], i64 2
; CHECK-NEXT: [[TMP7:%.*]] = insertelement <4 x double> [[TMP5]], double [[TMP6]], i64 1 ; CHECK-NEXT: [[TMP11:%.*]] = insertelement <2 x double> poison, double [[TMP10]], i64 0
; CHECK-NEXT: [[TMP8:%.*]] = extractelement <2 x double> [[TMP2]], i64 0 ; CHECK-NEXT: [[TMP12:%.*]] = extractelement <4 x double> [[TMP1]], i64 2
; CHECK-NEXT: [[TMP9:%.*]] = insertelement <4 x double> [[TMP7]], double [[TMP8]], i64 2 ; CHECK-NEXT: [[TMP13:%.*]] = insertelement <2 x double> [[TMP11]], double [[TMP12]], i64 1
; CHECK-NEXT: [[TMP10:%.*]] = extractelement <2 x double> [[TMP3]], i64 0 ; CHECK-NEXT: [[TMP14:%.*]] = extractelement <4 x double> [[TMP0]], i64 3
; CHECK-NEXT: [[TMP11:%.*]] = insertelement <4 x double> [[TMP9]], double [[TMP10]], i64 3 ; CHECK-NEXT: [[TMP15:%.*]] = insertelement <2 x double> poison, double [[TMP14]], i64 0
; CHECK-NEXT: [[TMP12:%.*]] = extractelement <2 x double> [[TMP0]], i64 1 ; CHECK-NEXT: [[TMP16:%.*]] = extractelement <4 x double> [[TMP1]], i64 3
; CHECK-NEXT: [[TMP13:%.*]] = insertelement <4 x double> poison, double [[TMP12]], i64 0 ; CHECK-NEXT: [[TMP17:%.*]] = insertelement <2 x double> [[TMP15]], double [[TMP16]], i64 1
; CHECK-NEXT: [[TMP14:%.*]] = extractelement <2 x double> [[TMP1]], i64 1 ; CHECK-NEXT: [[TMP18:%.*]] = shufflevector <2 x double> [[TMP5]], <2 x double> [[TMP9]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP15:%.*]] = insertelement <4 x double> [[TMP13]], double [[TMP14]], i64 1 ; CHECK-NEXT: [[TMP19:%.*]] = shufflevector <2 x double> [[TMP13]], <2 x double> [[TMP17]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP16:%.*]] = extractelement <2 x double> [[TMP2]], i64 1 ; CHECK-NEXT: [[TMP20:%.*]] = shufflevector <4 x double> [[TMP18]], <4 x double> [[TMP19]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[TMP17:%.*]] = insertelement <4 x double> [[TMP15]], double [[TMP16]], i64 2
; CHECK-NEXT: [[TMP18:%.*]] = extractelement <2 x double> [[TMP3]], i64 1
; CHECK-NEXT: [[TMP19:%.*]] = insertelement <4 x double> [[TMP17]], double [[TMP18]], i64 3
; CHECK-NEXT: [[TMP20:%.*]] = shufflevector <4 x double> [[TMP11]], <4 x double> [[TMP19]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: ret <8 x double> [[TMP20]] ; CHECK-NEXT: ret <8 x double> [[TMP20]]
; ;
entry: entry:
%add = fadd <8 x double> %a, %b %add = fadd <8 x double> %a, %b
%c = call <8 x double> @llvm.matrix.transpose(<8 x double> %add, i32 2, i32 4) %c = call <8 x double> @llvm.matrix.transpose(<8 x double> %add, i32 2, i32 4)
ret <8 x double> %c ret <8 x double> %c
} }
define <8 x double> @load_fadd_transpose(<8 x double>* %A.Ptr, <8 x double> %b) { define <8 x double> @load_fadd_transpose(<8 x double>* %A.Ptr, <8 x double> %b) {
; CHECK-LABEL: @load_fadd_transpose( ; CHECK-LABEL: @load_fadd_transpose(
; CHECK-NEXT: entry: ; CHECK-NEXT: entry:
; CHECK-NEXT: [[TMP0:%.*]] = bitcast <8 x double>* [[A_PTR:%.*]] to double* ; CHECK-NEXT: [[TMP0:%.*]] = bitcast <8 x double>* [[A_PTR:%.*]] to double*
; CHECK-NEXT: [[VEC_CAST:%.*]] = bitcast double* [[TMP0]] to <2 x double>* ; CHECK-NEXT: [[VEC_CAST:%.*]] = bitcast double* [[TMP0]] to <2 x double>*
; CHECK-NEXT: [[COL_LOAD:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST]], align 8 ; CHECK-NEXT: [[COL_LOAD:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST]], align 8
; CHECK-NEXT: [[VEC_GEP:%.*]] = getelementptr double, double* [[TMP0]], i64 2 ; CHECK-NEXT: [[VEC_GEP:%.*]] = getelementptr double, double* [[TMP0]], i64 2
; CHECK-NEXT: [[VEC_CAST1:%.*]] = bitcast double* [[VEC_GEP]] to <2 x double>* ; CHECK-NEXT: [[VEC_CAST2:%.*]] = bitcast double* [[VEC_GEP]] to <2 x double>*
; CHECK-NEXT: [[COL_LOAD2:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST1]], align 8 ; CHECK-NEXT: [[COL_LOAD3:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST2]], align 8
; CHECK-NEXT: [[VEC_GEP3:%.*]] = getelementptr double, double* [[TMP0]], i64 4 ; CHECK-NEXT: [[VEC_GEP4:%.*]] = getelementptr double, double* [[TMP0]], i64 4
; CHECK-NEXT: [[VEC_CAST4:%.*]] = bitcast double* [[VEC_GEP3]] to <2 x double>* ; CHECK-NEXT: [[VEC_CAST5:%.*]] = bitcast double* [[VEC_GEP4]] to <2 x double>*
; CHECK-NEXT: [[COL_LOAD5:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST4]], align 8 ; CHECK-NEXT: [[COL_LOAD6:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST5]], align 8
; CHECK-NEXT: [[VEC_GEP6:%.*]] = getelementptr double, double* [[TMP0]], i64 6 ; CHECK-NEXT: [[VEC_GEP7:%.*]] = getelementptr double, double* [[TMP0]], i64 6
; CHECK-NEXT: [[VEC_CAST7:%.*]] = bitcast double* [[VEC_GEP6]] to <2 x double>* ; CHECK-NEXT: [[VEC_CAST8:%.*]] = bitcast double* [[VEC_GEP7]] to <2 x double>*
; CHECK-NEXT: [[COL_LOAD8:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST7]], align 8 ; CHECK-NEXT: [[COL_LOAD9:%.*]] = load <2 x double>, <2 x double>* [[VEC_CAST8]], align 8
; CHECK-NEXT: [[SPLIT:%.*]] = shufflevector <8 x double> [[B:%.*]], <8 x double> poison, <2 x i32> <i32 0, i32 1> ; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <2 x double> [[COL_LOAD]], <2 x double> [[COL_LOAD3]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[SPLIT9:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 2, i32 3> ; CHECK-NEXT: [[TMP2:%.*]] = shufflevector <2 x double> [[COL_LOAD6]], <2 x double> [[COL_LOAD9]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[SPLIT10:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 4, i32 5> ; CHECK-NEXT: [[TMP3:%.*]] = shufflevector <4 x double> [[TMP1]], <4 x double> [[TMP2]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[SPLIT11:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <2 x i32> <i32 6, i32 7> ; CHECK-NEXT: [[SPLIT:%.*]] = shufflevector <8 x double> [[TMP3]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP1:%.*]] = fadd <2 x double> [[COL_LOAD]], [[SPLIT]] ; CHECK-NEXT: [[SPLIT10:%.*]] = shufflevector <8 x double> [[TMP3]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[TMP2:%.*]] = fadd <2 x double> [[COL_LOAD2]], [[SPLIT9]] ; CHECK-NEXT: [[SPLIT11:%.*]] = shufflevector <8 x double> [[B:%.*]], <8 x double> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP3:%.*]] = fadd <2 x double> [[COL_LOAD5]], [[SPLIT10]] ; CHECK-NEXT: [[SPLIT12:%.*]] = shufflevector <8 x double> [[B]], <8 x double> poison, <4 x i32> <i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: [[TMP4:%.*]] = fadd <2 x double> [[COL_LOAD8]], [[SPLIT11]] ; CHECK-NEXT: [[TMP4:%.*]] = fadd <4 x double> [[SPLIT]], [[SPLIT11]]
; CHECK-NEXT: [[TMP5:%.*]] = extractelement <2 x double> [[TMP1]], i64 0 ; CHECK-NEXT: [[TMP5:%.*]] = fadd <4 x double> [[SPLIT10]], [[SPLIT12]]
; CHECK-NEXT: [[TMP6:%.*]] = insertelement <4 x double> poison, double [[TMP5]], i64 0 ; CHECK-NEXT: [[TMP6:%.*]] = extractelement <4 x double> [[TMP4]], i64 0
; CHECK-NEXT: [[TMP7:%.*]] = extractelement <2 x double> [[TMP2]], i64 0 ; CHECK-NEXT: [[TMP7:%.*]] = insertelement <2 x double> poison, double [[TMP6]], i64 0
; CHECK-NEXT: [[TMP8:%.*]] = insertelement <4 x double> [[TMP6]], double [[TMP7]], i64 1 ; CHECK-NEXT: [[TMP8:%.*]] = extractelement <4 x double> [[TMP5]], i64 0
; CHECK-NEXT: [[TMP9:%.*]] = extractelement <2 x double> [[TMP3]], i64 0 ; CHECK-NEXT: [[TMP9:%.*]] = insertelement <2 x double> [[TMP7]], double [[TMP8]], i64 1
; CHECK-NEXT: [[TMP10:%.*]] = insertelement <4 x double> [[TMP8]], double [[TMP9]], i64 2 ; CHECK-NEXT: [[TMP10:%.*]] = extractelement <4 x double> [[TMP4]], i64 1
; CHECK-NEXT: [[TMP11:%.*]] = extractelement <2 x double> [[TMP4]], i64 0 ; CHECK-NEXT: [[TMP11:%.*]] = insertelement <2 x double> poison, double [[TMP10]], i64 0
; CHECK-NEXT: [[TMP12:%.*]] = insertelement <4 x double> [[TMP10]], double [[TMP11]], i64 3 ; CHECK-NEXT: [[TMP12:%.*]] = extractelement <4 x double> [[TMP5]], i64 1
; CHECK-NEXT: [[TMP13:%.*]] = extractelement <2 x double> [[TMP1]], i64 1 ; CHECK-NEXT: [[TMP13:%.*]] = insertelement <2 x double> [[TMP11]], double [[TMP12]], i64 1
; CHECK-NEXT: [[TMP14:%.*]] = insertelement <4 x double> poison, double [[TMP13]], i64 0 ; CHECK-NEXT: [[TMP14:%.*]] = extractelement <4 x double> [[TMP4]], i64 2
; CHECK-NEXT: [[TMP15:%.*]] = extractelement <2 x double> [[TMP2]], i64 1 ; CHECK-NEXT: [[TMP15:%.*]] = insertelement <2 x double> poison, double [[TMP14]], i64 0
; CHECK-NEXT: [[TMP16:%.*]] = insertelement <4 x double> [[TMP14]], double [[TMP15]], i64 1 ; CHECK-NEXT: [[TMP16:%.*]] = extractelement <4 x double> [[TMP5]], i64 2
; CHECK-NEXT: [[TMP17:%.*]] = extractelement <2 x double> [[TMP3]], i64 1 ; CHECK-NEXT: [[TMP17:%.*]] = insertelement <2 x double> [[TMP15]], double [[TMP16]], i64 1
; CHECK-NEXT: [[TMP18:%.*]] = insertelement <4 x double> [[TMP16]], double [[TMP17]], i64 2 ; CHECK-NEXT: [[TMP18:%.*]] = extractelement <4 x double> [[TMP4]], i64 3
; CHECK-NEXT: [[TMP19:%.*]] = extractelement <2 x double> [[TMP4]], i64 1 ; CHECK-NEXT: [[TMP19:%.*]] = insertelement <2 x double> poison, double [[TMP18]], i64 0
; CHECK-NEXT: [[TMP20:%.*]] = insertelement <4 x double> [[TMP18]], double [[TMP19]], i64 3 ; CHECK-NEXT: [[TMP20:%.*]] = extractelement <4 x double> [[TMP5]], i64 3
; CHECK-NEXT: [[TMP21:%.*]] = shufflevector <4 x double> [[TMP12]], <4 x double> [[TMP20]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7> ; CHECK-NEXT: [[TMP21:%.*]] = insertelement <2 x double> [[TMP19]], double [[TMP20]], i64 1
; CHECK-NEXT: ret <8 x double> [[TMP21]] ; CHECK-NEXT: [[TMP22:%.*]] = shufflevector <2 x double> [[TMP9]], <2 x double> [[TMP13]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP23:%.*]] = shufflevector <2 x double> [[TMP17]], <2 x double> [[TMP21]], <4 x i32> <i32 0, i32 1, i32 2, i32 3>
; CHECK-NEXT: [[TMP24:%.*]] = shufflevector <4 x double> [[TMP22]], <4 x double> [[TMP23]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
; CHECK-NEXT: ret <8 x double> [[TMP24]]
; ;
entry: entry:
%a = load <8 x double>, <8 x double>* %A.Ptr, align 8 %a = load <8 x double>, <8 x double>* %A.Ptr, align 8
%add = fadd <8 x double> %a, %b %add = fadd <8 x double> %a, %b
%c = call <8 x double> @llvm.matrix.transpose(<8 x double> %add, i32 2, i32 4) %c = call <8 x double> @llvm.matrix.transpose(<8 x double> %add, i32 2, i32 4)
ret <8 x double> %c ret <8 x double> %c
▲ Show 20 Lines • Show All 47 Lines • Show Last 20 Lines