This is an archive of the discontinued LLVM Phabricator instance.

Differential D144932

[mlir][sparse] fix performance bug in matmul with a sparse rhs due to suboptimal iteration graphs.
ClosedPublic

Authored by Peiming on Feb 27 2023, 5:18 PM.

Details

Reviewers
aartbik
nicolasvasilache
bixia
wrengr
Commits
rG849529ba8a2e: [mlir][sparse] fix performance bug in matmul with a sparse rhs due to…
Summary

While dense tensors support random accesses, it is critical to visit them in a row-major order for better cache locality. However, we previously consider dense inputs and outputs together when computing constraints for building iteration graph, it could lead us to less efficient iteration graphs.

This patch adds a new SortMask::kIncludeDenseInput to treat dense inputs/outputs separately when building iteration graph, thus increasing the chance for use to construct a better iteration graph.

A more fine-grained approach is to treat each input separately.

Note, related to:
https://github.com/llvm/llvm-project/issues/51651

Diff Detail

Event Timeline

Peiming created this revision.Feb 27 2023, 5:18 PM
Herald added a reviewer: aartbik. · View Herald TranscriptFeb 27 2023, 5:18 PM
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: hanchung, jsetoain, Moerafaat and 23 others. · View Herald Transcript
Peiming requested review of this revision.Feb 27 2023, 5:18 PM
Herald added a reviewer: nicolasvasilache. · View Herald TranscriptFeb 27 2023, 5:18 PM
Herald added a project: Restricted Project. · View Herald Transcript
Herald added subscribers: stephenneuendorffer, nicolasvasilache. · View Herald Transcript
Peiming added reviewers: bixia, wrengr.Feb 27 2023, 5:18 PM
Peiming updated this revision to Diff 500983.Feb 27 2023, 5:28 PM

update comments.

Peiming added inline comments.Feb 27 2023, 5:30 PM
mlir/test/Dialect/SparseTensor/sparse_2d.mlir
1034

Inefficient accessing pattern

mlir/test/Dialect/SparseTensor/sparse_kernels.mlir
177

Inefficient accessing pattern

226–274

Inefficient access pattern

mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir
90

Inefficient access pattern

wrengr added inline comments.Feb 27 2023, 6:03 PM
mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
458–472

Since the bitmasks overlap, it's a bit hard to reason about these three conditions, and it'd be better to rearrange them to make things easier to follow. If I'm reading things correctly then the following should be logically equivalent to the current code:

if (!enc) {
  const bool lacksDenseInput = !(mask & kIncludeDenseInput);
  const bool lacksDenseOutput = !(mask & kIncludeDenseOutput);
  const bool hasDpsInput = env.op().isDpsInput(&t);
  if ( (lacksDenseInput && lacksDenseOutput)
    || (lacksDenseInput && hasDpsInput)
    || (lacksDenseOutput && !hasDpsInput)
  ) continue;
}

That's still not particularly easy to follow, but it at least helps clarify things, and it gives a good place to start for rearranging the condition to be easier to reason about.

Harbormaster completed remote builds in B216366: Diff 500983.Feb 27 2023, 6:13 PM
aartbik added a comment.Feb 27 2023, 6:41 PM

please add some rationale to the description on what has changed
(we have gone back and forth on loop ordering quite a bit already ;-)

mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
50

I would organize these now as follows

  • individual tests b001 b010 b100
  • sets passed for progressive lowering b111 (all) bxxx (some other sets) b000 (none, only sparse)

or, if you don't like having them in the same place,
define the individual bits in the SortMask and define some consts for the "sets"

459

note that we also have mask testing at L515 and such, making this all a bit hard to follow
(since some masks are mutually exclusive dense/sparse but some apply to both)

Peiming edited the summary of this revision. (Show Details)Feb 27 2023, 8:25 PM
Peiming updated this revision to Diff 501187.Feb 28 2023, 9:36 AM
Peiming marked an inline comment as done.

address comments.

Peiming marked 2 inline comments as done.Feb 28 2023, 9:36 AM
Harbormaster completed remote builds in B216502: Diff 501187.Feb 28 2023, 9:36 AM
aartbik edited the summary of this revision. (Show Details)Feb 28 2023, 11:43 AM
aartbik added inline comments.Feb 28 2023, 11:47 AM
mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
52

maybe add a comment line seperator, as in

The individual mask bits.
1, 2, 4
 The subsets of mask bits
7, 3, 0

465

I also feel like we could do without this at some point ;-)

1550–1555

The earlier a constraint mask ....

But... I would say this is still very heuristic driven, no guarantees ;-)

mlir/test/Dialect/SparseTensor/sparse_2d.mlir
1012

why did you remove the sparsity CHECK

That seems very unrelated to this revision

mlir/test/Dialect/SparseTensor/sparse_kernels.mlir
206–210

same question

257

and here

mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir
72

and here

Peiming added inline comments.Feb 28 2023, 11:51 AM
mlir/test/Dialect/SparseTensor/sparse_2d.mlir
1012

Simply because it is too long... I can add them back if you want

Peiming updated this revision to Diff 501252.Feb 28 2023, 12:05 PM
Peiming marked 4 inline comments as done.
Peiming edited the summary of this revision. (Show Details)

address comment and revert unrelated changes.

Harbormaster completed remote builds in B216552: Diff 501252.Feb 28 2023, 12:06 PM
aartbik accepted this revision.Feb 28 2023, 12:10 PM
aartbik added inline comments.
mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp
1550–1555

typo remains: the eariler a constrait mask

This revision is now accepted and ready to land.Feb 28 2023, 12:10 PM
Peiming updated this revision to Diff 501256.Feb 28 2023, 12:10 PM
Peiming marked an inline comment as done.

fix typo

Harbormaster completed remote builds in B216555: Diff 501256.Feb 28 2023, 12:11 PM
Peiming marked 2 inline comments as done.Feb 28 2023, 12:13 PM
This revision was landed with ongoing or failed builds.Feb 28 2023, 1:02 PM
Closed by commit rG849529ba8a2e: [mlir][sparse] fix performance bug in matmul with a sparse rhs due to… (authored by Peiming). · Explain Why
This revision was automatically updated to reflect the committed changes.
Peiming added a commit: rG849529ba8a2e: [mlir][sparse] fix performance bug in matmul with a sparse rhs due to….

Revision Contents

PathSize
mlir/
lib/
Dialect/
SparseTensor/
Transforms/
33 lines
test/
Dialect/
SparseTensor/
69 lines
176 lines
83 lines

Diff 500983

mlir/lib/Dialect/SparseTensor/Transforms/Sparsification.cpp

Show First 20 Lines • Show All 41 Lines • ▼ Show 20 Lines
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
// Declarations // Declarations
//===----------------------------------------------------------------------===// //===----------------------------------------------------------------------===//
namespace { namespace {
/// Iteration graph sorting. /// Iteration graph sorting.
enum SortMask { enum SortMask {
kSparseOnly = 0x0, kSparseOnly = 0x0, // b000
aartbikUnsubmitted
Done
ReplyInline Actions

I would organize these now as follows

  • individual tests b001 b010 b100
  • sets passed for progressive lowering b111 (all) bxxx (some other sets) b000 (none, only sparse)

or, if you don't like having them in the same place,
define the individual bits in the SortMask and define some consts for the "sets"

aartbik: I would organize these now as follows --- individual tests b001 b010 b100 --- sets…
kIncludeDense = 0x1, kIncludeDenseOutput = 0x1, // b001
kIncludeUndef = 0x2, kIncludeDenseInput = 0x2, // b010
aartbikUnsubmitted
Done
ReplyInline Actions

maybe add a comment line seperator, as in

The individual mask bits.
1, 2, 4
 The subsets of mask bits
7, 3, 0

aartbik: maybe add a comment line seperator, as in // The individual mask bits. 1, 2, 4 // The subsets…
kIncludeAll = 0x3 kIncludeDense = 0x3, // b011
kIncludeUndef = 0x4, // b100
kIncludeAll = 0x7 // b111
}; };
/// A helper class that visits an affine expression and tries to find an /// A helper class that visits an affine expression and tries to find an
/// AffineDimExpr to which the corresponding iterator from a GenericOp matches /// AffineDimExpr to which the corresponding iterator from a GenericOp matches
/// the desired iterator type. /// the desired iterator type.
class AffineDimFinder : public AffineExprVisitor<AffineDimFinder> { class AffineDimFinder : public AffineExprVisitor<AffineDimFinder> {
public: public:
explicit AffineDimFinder(linalg::GenericOp op) explicit AffineDimFinder(linalg::GenericOp op)
▲ Show 20 Lines • Show All 386 Lines • ▼ Show 20 Linesstatic bool computeIterationGraph(CodegenEnv &env, unsigned mask,
std::vector<unsigned> inDegree(n, 0); // in-degree of each node. std::vector<unsigned> inDegree(n, 0); // in-degree of each node.
const auto iteratorTypes = env.op().getIteratorTypesArray(); const auto iteratorTypes = env.op().getIteratorTypesArray();
// Iterate over the indexing maps of every tensor in the tensor expression. // Iterate over the indexing maps of every tensor in the tensor expression.
for (OpOperand &t : env.op()->getOpOperands()) { for (OpOperand &t : env.op()->getOpOperands()) {
// Get map and encoding. // Get map and encoding.
const auto map = env.op().getMatchingIndexingMap(&t); const auto map = env.op().getMatchingIndexingMap(&t);
const auto enc = getSparseTensorEncoding(t.get().getType()); const auto enc = getSparseTensorEncoding(t.get().getType());
assert(map.getNumDims() + getNumCompoundAffineOnSparseDims(env.op()) == n); assert(map.getNumDims() + getNumCompoundAffineOnSparseDims(env.op()) == n);
// Skip dense tensor constraints when not requested.
// Skips dense tensor constraints when not requested.
aartbikUnsubmitted
Done
ReplyInline Actions

note that we also have mask testing at L515 and such, making this all a bit hard to follow
(since some masks are mutually exclusive dense/sparse but some apply to both)

aartbik: note that we also have mask testing at L515 and such, making this all a bit hard to follow…
if (!(mask & SortMask::kIncludeDense) && !enc) if (!(mask & SortMask::kIncludeDense) && !enc)
continue; continue;
// Skips dense input when not requested.
if (!(mask & SortMask::kIncludeDenseInput) && !enc &&
env.op().isDpsInput(&t)) {
aartbikUnsubmitted
Done
ReplyInline Actions

I also feel like we could do without this at some point ;-)

aartbik: I also feel like we could do without this at some point ;-)
continue;
}
// Skips dense output when not requested.
if (!(mask & SortMask::kIncludeDenseOutput) && !enc &&
!env.op().isDpsInput(&t)) {
continue;
wrengrUnsubmitted
Done
ReplyInline Actions

Since the bitmasks overlap, it's a bit hard to reason about these three conditions, and it'd be better to rearrange them to make things easier to follow. If I'm reading things correctly then the following should be logically equivalent to the current code:

if (!enc) {
  const bool lacksDenseInput = !(mask & kIncludeDenseInput);
  const bool lacksDenseOutput = !(mask & kIncludeDenseOutput);
  const bool hasDpsInput = env.op().isDpsInput(&t);
  if ( (lacksDenseInput && lacksDenseOutput)
    || (lacksDenseInput && hasDpsInput)
    || (lacksDenseOutput && !hasDpsInput)
  ) continue;
}

That's still not particularly easy to follow, but it at least helps clarify things, and it gives a good place to start for rearranging the condition to be easier to reason about.

wrengr: Since the bitmasks overlap, it's a bit hard to reason about these three conditions, and it'd be…
}
// Each tensor expression and optional dimension ordering (row-major // Each tensor expression and optional dimension ordering (row-major
// by default) puts an ordering constraint on the loop indices. For // by default) puts an ordering constraint on the loop indices. For
// example, the tensor expresion A_ijk forces the ordering i < j < k // example, the tensor expresion A_ijk forces the ordering i < j < k
// on the loop indices if no explicit dimension ordering is given. // on the loop indices if no explicit dimension ordering is given.
const Level lvlRank = map.getNumResults(); const Level lvlRank = map.getNumResults();
assert(!enc || lvlRank == enc.getLvlRank()); assert(!enc || lvlRank == enc.getLvlRank());
for (Level l = 0; l < lvlRank; l++) { for (Level l = 0; l < lvlRank; l++) {
// FIXME: `toOrigDim` is deprecated. // FIXME: `toOrigDim` is deprecated.
▲ Show 20 Lines • Show All 1,060 Lines • ▼ Show 20 Lines LogicalResult matchAndRewrite(linalg::GenericOp op,
// constraints until an acyclic iteration graph results, such that sparse // constraints until an acyclic iteration graph results, such that sparse
// code generation can proceed. As a last resort, an attempt is made // code generation can proceed. As a last resort, an attempt is made
// to resolve cycles by inserting a conversion. // to resolve cycles by inserting a conversion.
bool isAdmissible = false; bool isAdmissible = false;
bool hasCycle = true; bool hasCycle = true;
// An const list of all masks that we used for interation graph // An const list of all masks that we used for interation graph
// computation. Must be ordered from more strict to less strict. // computation. Must be ordered from more strict to less strict.
const auto allMask = {SortMask::kIncludeAll, SortMask::kIncludeUndef, // The eariler a constrait mask can be satisfied, the faster the
SortMask::kIncludeDense, SortMask::kSparseOnly}; // generated kernel will be.
const auto allMask = {
SortMask::kIncludeAll, SortMask::kIncludeDense,
SortMask::kIncludeDenseInput, SortMask::kIncludeDenseOutput,
SortMask::kIncludeUndef, SortMask::kSparseOnly};
aartbikUnsubmitted
Done
ReplyInline Actions

The earlier a constraint mask ....

But... I would say this is still very heuristic driven, no guarantees ;-)

aartbik: The earlier a constraint mask .... But... I would say this is still very heuristic driven, no…
aartbikUnsubmitted
Done
ReplyInline Actions

typo remains: the eariler a constrait mask

aartbik: typo remains: the eariler a constrait mask
for (auto mask : allMask) { for (auto mask : allMask) {
if (computeIterationGraph(env, mask)) { if (computeIterationGraph(env, mask)) {
hasCycle = false; hasCycle = false;
if (env.isAdmissibleTopoOrder()) { if (env.isAdmissibleTopoOrder()) {
isAdmissible = true; isAdmissible = true;
break; break;
} }
// else try a set of less strict constraints. // else try a set of less strict constraints.
▲ Show 20 Lines • Show All 66 Lines • Show Last 20 Lines

mlir/test/Dialect/SparseTensor/sparse_2d.mlir

Show First 20 Lines • Show All 996 Lines • ▼ Show 20 Lines indexing_maps = [
affine_map<(i,j,k) -> (i,k)>, // A affine_map<(i,j,k) -> (i,k)>, // A
affine_map<(i,j,k) -> (k,j)>, // B affine_map<(i,j,k) -> (k,j)>, // B
affine_map<(i,j,k) -> (i,j)> // X (out) affine_map<(i,j,k) -> (i,j)> // X (out)
], ],
iterator_types = ["parallel", "parallel", "reduction"], iterator_types = ["parallel", "parallel", "reduction"],
doc = "X(i,j) += S(i,j) SUM_k A(i,k) B(k,j)" doc = "X(i,j) += S(i,j) SUM_k A(i,k) B(k,j)"
} }
// CHECK-LABEL: func @sampled_dense_dense( // CHECK-LABEL: func.func @sampled_dense_dense(
// CHECK-SAME: %[[VAL_0:.*0]]: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>, // CHECK-SAME: %[[VAL_0:.*0]]: tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>,
// CHECK-SAME: %[[VAL_1:.*1]]: tensor<?x?xf32>, // CHECK-SAME: %[[VAL_1:.*1]]: tensor<?x?xf32>,
// CHECK-SAME: %[[VAL_2:.*2]]: tensor<?x?xf32>, // CHECK-SAME: %[[VAL_2:.*2]]: tensor<?x?xf32>,
// CHECK-SAME: %[[VAL_3:.*3]]: tensor<?x?xf32>) -> tensor<?x?xf32> { // CHECK-SAME: %[[VAL_3:.*3]]: tensor<?x?xf32>) -> tensor<?x?xf32> {
// CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index // CHECK-DAG: %[[VAL_4:.*]] = arith.constant 1 : index
// CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index // CHECK-DAG: %[[VAL_5:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_6:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index}
aartbikUnsubmitted
Done
ReplyInline Actions

why did you remove the sparsity CHECK

That seems very unrelated to this revision

aartbik: why did you remove the sparsity CHECK That seems very unrelated to this revision
PeimingAuthorUnsubmitted
Done
ReplyInline Actions

Simply because it is too long... I can add them back if you want

Peiming: Simply because it is too long... I can add them back if you want
// CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index}
// CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<?x?xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xf32> // CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_0]]
// CHECK-DAG: %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_1]] : memref<?x?xf32> // CHECK-DAG: %[[VAL_11:.*]] = tensor.dim %[[VAL_1]], %[[VAL_4]] : tensor<?x?xf32>
// CHECK-DAG: %[[VAL_12:.*]] = tensor.dim %[[VAL_1]], %[[VAL_5]] : tensor<?x?xf32> // CHECK-DAG: %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_1]] : memref<?x?xf32>
// CHECK-DAG: %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<?x?xf32> // CHECK-DAG: %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<?x?xf32>
// CHECK-DAG: %[[VAL_17:.*]] = bufferization.to_memref %[[VAL_3]] : memref<?x?xf32> // CHECK-DAG: %[[VAL_14:.*]] = bufferization.to_memref %[[VAL_3]] : memref<?x?xf32>
// CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex> // CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
// CHECK: %[[VAL_19:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex> // CHECK: %[[VAL_16:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_20:.*]] = %[[VAL_18]] to %[[VAL_19]] step %[[VAL_5]] { // CHECK: scf.for %[[VAL_17:.*]] = %[[VAL_15]] to %[[VAL_16]] step %[[VAL_4]] {
// CHECK: %[[VAL_21:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_20]]] : memref<?xindex> // CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_17]]] : memref<?xindex>
// CHECK: %[[VAL_22:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_20]]] : memref<?xindex> // CHECK: scf.for %[[VAL_19:.*]] = %[[VAL_5]] to %[[VAL_11]] step %[[VAL_4]] {
// CHECK: %[[VAL_23:.*]] = arith.addi %[[VAL_20]], %[[VAL_5]] : index // CHECK: %[[VAL_20:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_18]], %[[VAL_19]]] : memref<?x?xf32>
// CHECK: %[[VAL_24:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_23]]] : memref<?xindex> // CHECK: %[[VAL_21:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_17]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_25:.*]] = %[[VAL_22]] to %[[VAL_24]] step %[[VAL_5]] { // CHECK: %[[VAL_22:.*]] = arith.addi %[[VAL_17]], %[[VAL_4]] : index
// CHECK-DAG: %[[VAL_26:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_25]]] : memref<?xindex> // CHECK: %[[VAL_23:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_22]]] : memref<?xindex>
// CHECK-DAG: %[[VAL_27:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_25]]] : memref<?xf32> // CHECK: scf.for %[[VAL_24:.*]] = %[[VAL_21]] to %[[VAL_23]] step %[[VAL_4]] {
// CHECK-DAG: %[[VAL_28:.*]] = memref.load %[[VAL_17]]{{\[}}%[[VAL_21]], %[[VAL_26]]] : memref<?x?xf32> // CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_24]]] : memref<?xindex>
// CHECK: %[[VAL_29:.*]] = scf.for %[[VAL_30:.*]] = %[[VAL_4]] to %[[VAL_12]] step %[[VAL_5]] iter_args(%[[VAL_31:.*]] = %[[VAL_28]]) -> (f32) { // CHECK: %[[VAL_26:.*]] = memref.load %[[VAL_14]]{{\[}}%[[VAL_18]], %[[VAL_25]]] : memref<?x?xf32>
// CHECK: %[[VAL_32:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_21]], %[[VAL_30]]] : memref<?x?xf32> // CHECK: %[[VAL_27:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_24]]] : memref<?xf32>
// CHECK: %[[VAL_33:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_30]], %[[VAL_26]]] : memref<?x?xf32> // CHECK: %[[VAL_28:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_19]], %[[VAL_25]]] : memref<?x?xf32>
PeimingAuthorUnsubmitted
Done
ReplyInline Actions

Inefficient accessing pattern

Peiming: Inefficient accessing pattern
// CHECK: %[[VAL_34:.*]] = arith.mulf %[[VAL_32]], %[[VAL_33]] : f32 // CHECK: %[[VAL_29:.*]] = arith.mulf %[[VAL_20]], %[[VAL_28]] : f32
// CHECK: %[[VAL_35:.*]] = arith.mulf %[[VAL_27]], %[[VAL_34]] : f32 // CHECK: %[[VAL_30:.*]] = arith.mulf %[[VAL_27]], %[[VAL_29]] : f32
// CHECK: %[[VAL_36:.*]] = arith.addf %[[VAL_31]], %[[VAL_35]] : f32 // CHECK: %[[VAL_31:.*]] = arith.addf %[[VAL_26]], %[[VAL_30]] : f32
// CHECK: scf.yield %[[VAL_36]] : f32 // CHECK: memref.store %[[VAL_31]], %[[VAL_14]]{{\[}}%[[VAL_18]], %[[VAL_25]]] : memref<?x?xf32>
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: memref.store %[[VAL_29]], %[[VAL_17]]{{\[}}%[[VAL_21]], %[[VAL_26]]] : memref<?x?xf32> // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: %[[VAL_32:.*]] = bufferization.to_tensor %[[VAL_14]] : memref<?x?xf32>
// CHECK: %[[VAL_38:.*]] = bufferization.to_tensor %[[VAL_17]] : memref<?x?xf32> // CHECK: return %[[VAL_32]] : tensor<?x?xf32>
// CHECK: return %[[VAL_38]] : tensor<?x?xf32>
// CHECK: } // CHECK: }
func.func @sampled_dense_dense(%args: tensor<?x?xf32, #Tss>, func.func @sampled_dense_dense(%args: tensor<?x?xf32, #Tss>,
%arga: tensor<?x?xf32>, %arga: tensor<?x?xf32>,
%argb: tensor<?x?xf32>, %argb: tensor<?x?xf32>,
%argx: tensor<?x?xf32>) -> tensor<?x?xf32> { %argx: tensor<?x?xf32>) -> tensor<?x?xf32> {
%0 = linalg.generic #trait_sampled_dense_dense %0 = linalg.generic #trait_sampled_dense_dense
ins(%args, %arga, %argb: tensor<?x?xf32, #Tss>, tensor<?x?xf32>, tensor<?x?xf32>) ins(%args, %arga, %argb: tensor<?x?xf32, #Tss>, tensor<?x?xf32>, tensor<?x?xf32>)
outs(%argx: tensor<?x?xf32>) { outs(%argx: tensor<?x?xf32>) {
▲ Show 20 Lines • Show All 234 Lines • Show Last 20 Lines

mlir/test/Dialect/SparseTensor/sparse_kernels.mlir

Show First 20 Lines • Show All 45 Lines • ▼ Show 20 Linesfunc.func @matmul1(%a: tensor<10x20xf32, #DCSR>,
%b: tensor<20x30xf32>, %b: tensor<20x30xf32>,
%c: tensor<10x30xf32>) -> tensor<10x30xf32> { %c: tensor<10x30xf32>) -> tensor<10x30xf32> {
%0 = linalg.matmul %0 = linalg.matmul
ins(%a, %b: tensor<10x20xf32, #DCSR>, tensor<20x30xf32>) ins(%a, %b: tensor<10x20xf32, #DCSR>, tensor<20x30xf32>)
outs(%c: tensor<10x30xf32>) -> tensor<10x30xf32> outs(%c: tensor<10x30xf32>) -> tensor<10x30xf32>
return %0 : tensor<10x30xf32> return %0 : tensor<10x30xf32>
} }
// CHECK-LABEL: func.func @matmul_sparse_rhs(
// CHECK-SAME: %[[VAL_0:.*]]: tensor<10x20xf32>,
// CHECK-SAME: %[[VAL_1:.*]]: tensor<20x30xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>,
// CHECK-SAME: %[[VAL_2:.*]]: tensor<10x30xf32>) -> tensor<10x30xf32> {
// CHECK-DAG: %[[VAL_3:.*]] = arith.constant 10 : index
// CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index
// CHECK-DAG: %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<10x20xf32>
// CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index}
// CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index}
// CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]]
// CHECK: %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<10x30xf32>
// CHECK: scf.for %[[VAL_13:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
// CHECK: %[[VAL_14:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_5]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_16:.*]] = %[[VAL_14]] to %[[VAL_15]] step %[[VAL_5]] {
// CHECK: %[[VAL_17:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_16]]] : memref<?xindex>
// CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_13]], %[[VAL_17]]] : memref<10x20xf32>
// CHECK: %[[VAL_19:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_16]]] : memref<?xindex>
// CHECK: %[[VAL_20:.*]] = arith.addi %[[VAL_16]], %[[VAL_5]] : index
// CHECK: %[[VAL_21:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_20]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_22:.*]] = %[[VAL_19]] to %[[VAL_21]] step %[[VAL_5]] {
// CHECK: %[[VAL_23:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_22]]] : memref<?xindex>
// CHECK: %[[VAL_24:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_13]], %[[VAL_23]]] : memref<10x30xf32>
// CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_22]]] : memref<?xf32>
// CHECK: %[[VAL_26:.*]] = arith.mulf %[[VAL_18]], %[[VAL_25]] : f32
// CHECK: %[[VAL_27:.*]] = arith.addf %[[VAL_24]], %[[VAL_26]] : f32
// CHECK: memref.store %[[VAL_27]], %[[VAL_12]]{{\[}}%[[VAL_13]], %[[VAL_23]]] : memref<10x30xf32>
// CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: %[[VAL_28:.*]] = bufferization.to_tensor %[[VAL_12]] : memref<10x30xf32>
// CHECK: return %[[VAL_28]] : tensor<10x30xf32>
// CHECK: }
// IMPORTANT! Ensures that dense input are visit in row-major order.
func.func @matmul_sparse_rhs(%a: tensor<10x20xf32>,
%b: tensor<20x30xf32, #DCSR>,
%c: tensor<10x30xf32>) -> tensor<10x30xf32> {
%0 = linalg.matmul
ins(%a, %b: tensor<10x20xf32>, tensor<20x30xf32,#DCSR>)
outs(%c: tensor<10x30xf32>) -> tensor<10x30xf32>
return %0 : tensor<10x30xf32>
}
// //
// Computes C = A x B with all matrices sparse (SpMSpM) in DCSR. // Computes C = A x B with all matrices sparse (SpMSpM) in DCSR.
// //
// CHECK-LABEL: func.func @matmul2( // CHECK-LABEL: func.func @matmul2(
// CHECK-SAME: %[[VAL_0:.*]]: tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>, // CHECK-SAME: %[[VAL_0:.*]]: tensor<4x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>,
// CHECK-SAME: %[[VAL_1:.*]]: tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>) -> tensor<4x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> { // CHECK-SAME: %[[VAL_1:.*]]: tensor<8x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>) -> tensor<4x4xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> {
// CHECK-DAG: %[[VAL_2:.*]] = arith.constant 0 : index // CHECK-DAG: %[[VAL_2:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_3:.*]] = arith.constant 1 : index // CHECK-DAG: %[[VAL_3:.*]] = arith.constant 1 : index
▲ Show 20 Lines • Show All 89 Lines • ▼ Show 20 Lines
// CHECK-LABEL: func.func @conv2d( // CHECK-LABEL: func.func @conv2d(
// CHECK-SAME: %[[VAL_0:.*]]: tensor<8x8xi32>, // CHECK-SAME: %[[VAL_0:.*]]: tensor<8x8xi32>,
// CHECK-SAME: %[[VAL_1:.*]]: tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>, // CHECK-SAME: %[[VAL_1:.*]]: tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>,
// CHECK-SAME: %[[VAL_2:.*]]: tensor<6x6xi32>) -> tensor<6x6xi32> { // CHECK-SAME: %[[VAL_2:.*]]: tensor<6x6xi32>) -> tensor<6x6xi32> {
// CHECK-DAG: %[[VAL_3:.*]] = arith.constant 6 : index // CHECK-DAG: %[[VAL_3:.*]] = arith.constant 6 : index
// CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index // CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index // CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index
// CHECK-DAG: %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<8x8xi32> // CHECK-DAG: %[[VAL_6:.*]] = bufferization.to_memref %[[VAL_0]] : memref<8x8xi32>
// CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index}
// CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index}
// CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<3x3xi32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xi32> // CHECK-DAG: %[[VAL_11:.*]] = sparse_tensor.values %[[VAL_1]]
aartbikUnsubmitted
Done
ReplyInline Actions

same question

aartbik: same question
// CHECK: %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<6x6xi32> // CHECK: %[[VAL_12:.*]] = bufferization.to_memref %[[VAL_2]] : memref<6x6xi32>
// CHECK: %[[VAL_13:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_4]]] : memref<?xindex> // CHECK: scf.for %[[VAL_13:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
// CHECK: %[[VAL_14:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_5]]] : memref<?xindex> // CHECK: %[[VAL_14:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_15:.*]] = %[[VAL_13]] to %[[VAL_14]] step %[[VAL_5]] { // CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_5]]] : memref<?xindex>
// CHECK: %[[VAL_16:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_15]]] : memref<?xindex> // CHECK: scf.for %[[VAL_16:.*]] = %[[VAL_14]] to %[[VAL_15]] step %[[VAL_5]] {
// CHECK: %[[VAL_17:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_15]]] : memref<?xindex> // CHECK: %[[VAL_17:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_16]]] : memref<?xindex>
// CHECK: %[[VAL_18:.*]] = arith.addi %[[VAL_15]], %[[VAL_5]] : index // CHECK: scf.for %[[VAL_18:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
// CHECK: %[[VAL_19:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_18]]] : memref<?xindex> // CHECK: %[[VAL_19:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_13]], %[[VAL_18]]] : memref<6x6xi32>
// CHECK: scf.for %[[VAL_20:.*]] = %[[VAL_17]] to %[[VAL_19]] step %[[VAL_5]] { // CHECK: %[[VAL_20:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_16]]] : memref<?xindex>
// CHECK: %[[VAL_21:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_20]]] : memref<?xindex> // CHECK: %[[VAL_21:.*]] = arith.addi %[[VAL_16]], %[[VAL_5]] : index
// CHECK: %[[VAL_22:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_20]]] : memref<?xi32> // CHECK: %[[VAL_22:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_21]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_23:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] { // CHECK: %[[VAL_23:.*]] = scf.for %[[VAL_24:.*]] = %[[VAL_20]] to %[[VAL_22]] step %[[VAL_5]] iter_args(%[[VAL_25:.*]] = %[[VAL_19]]) -> (i32) {
// CHECK: scf.for %[[VAL_24:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] { // CHECK: %[[VAL_26:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_24]]] : memref<?xindex>
// CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_24]], %[[VAL_23]]] : memref<6x6xi32> // CHECK: %[[VAL_27:.*]] = arith.addi %[[VAL_13]], %[[VAL_17]] : index
PeimingAuthorUnsubmitted
Done
ReplyInline Actions

Inefficient accessing pattern

Peiming: Inefficient accessing pattern
// CHECK: %[[VAL_26:.*]] = arith.addi %[[VAL_24]], %[[VAL_16]] : index // CHECK: %[[VAL_28:.*]] = arith.addi %[[VAL_18]], %[[VAL_26]] : index
// CHECK: %[[VAL_27:.*]] = arith.addi %[[VAL_23]], %[[VAL_21]] : index // CHECK: %[[VAL_29:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_27]], %[[VAL_28]]] : memref<8x8xi32>
// CHECK: %[[VAL_28:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_26]], %[[VAL_27]]] : memref<8x8xi32> // CHECK: %[[VAL_30:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_24]]] : memref<?xi32>
// CHECK: %[[VAL_29:.*]] = arith.muli %[[VAL_28]], %[[VAL_22]] : i32 // CHECK: %[[VAL_31:.*]] = arith.muli %[[VAL_29]], %[[VAL_30]] : i32
// CHECK: %[[VAL_30:.*]] = arith.addi %[[VAL_25]], %[[VAL_29]] : i32 // CHECK: %[[VAL_32:.*]] = arith.addi %[[VAL_25]], %[[VAL_31]] : i32
// CHECK: memref.store %[[VAL_30]], %[[VAL_12]]{{\[}}%[[VAL_24]], %[[VAL_23]]] : memref<6x6xi32> // CHECK: scf.yield %[[VAL_32]] : i32
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: memref.store %[[VAL_33:.*]], %[[VAL_12]]{{\[}}%[[VAL_13]], %[[VAL_18]]] : memref<6x6xi32>
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: %[[VAL_31:.*]] = bufferization.to_tensor %[[VAL_12]] : memref<6x6xi32> // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: return %[[VAL_31]] : tensor<6x6xi32> // CHECK: %[[VAL_34:.*]] = bufferization.to_tensor %[[VAL_12]] : memref<6x6xi32>
// CHECK: return %[[VAL_34]] : tensor<6x6xi32>
// CHECK: } // CHECK: }
func.func @conv2d(%input: tensor<8x8xi32>, func.func @conv2d(%input: tensor<8x8xi32>,
%filter: tensor<3x3xi32, #DCSR>, %filter: tensor<3x3xi32, #DCSR>,
%output: tensor<6x6xi32>) -> tensor<6x6xi32> { %output: tensor<6x6xi32>) -> tensor<6x6xi32> {
%0 = linalg.conv_2d %0 = linalg.conv_2d
ins (%input, %filter: tensor<8x8xi32>, tensor<3x3xi32, #DCSR>) ins (%input, %filter: tensor<8x8xi32>, tensor<3x3xi32, #DCSR>)
outs (%output: tensor<6x6xi32>) -> tensor<6x6xi32> outs (%output: tensor<6x6xi32>) -> tensor<6x6xi32>
return %0 : tensor<6x6xi32> return %0 : tensor<6x6xi32>
} }
// CHECK-LABEL: func.func @quantized_matmul( // CHECK-LABEL: func.func @quantized_matmul(
// CHECK-SAME: %[[VAL_0:.*]]: tensor<5x3xi8>, // CHECK-SAME: %[[VAL_0:.*]]: tensor<5x3xi8>,
// CHECK-SAME: %[[VAL_1:.*]]: tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>, // CHECK-SAME: %[[VAL_1:.*]]: tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>,
// CHECK-SAME: %[[VAL_2:.*]]: tensor<5x6xi64>) -> tensor<5x6xi64> { // CHECK-SAME: %[[VAL_2:.*]]: tensor<5x6xi64>) -> tensor<5x6xi64> {
// CHECK-DAG: %[[VAL_3:.*]] = arith.constant 5 : index // CHECK: %[[VAL_3:.*]] = arith.constant 5 : index
// CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index // CHECK: %[[VAL_4:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index // CHECK: %[[VAL_5:.*]] = arith.constant 1 : index
// CHECK-DAG: %[[VAL_6:.*]] = arith.constant 2 : i64 // CHECK: %[[VAL_6:.*]] = arith.constant 2 : i64
// CHECK-DAG: %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_0]] : memref<5x3xi8> // CHECK: %[[VAL_7:.*]] = bufferization.to_memref %[[VAL_0]] : memref<5x3xi8>
// CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK: %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index}
aartbikUnsubmitted
Done
ReplyInline Actions

and here

aartbik: and here
// CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK: %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index}
// CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK: %[[VAL_10:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index} : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xindex> // CHECK: %[[VAL_11:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 1 : index}
// CHECK-DAG: %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<3x6xi8, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>> to memref<?xi8> // CHECK: %[[VAL_12:.*]] = sparse_tensor.values %[[VAL_1]]
// CHECK: %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<5x6xi64> // CHECK: %[[VAL_13:.*]] = bufferization.to_memref %[[VAL_2]] : memref<5x6xi64>
// CHECK: %[[VAL_14:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_4]]] : memref<?xindex> // CHECK: scf.for %[[VAL_14:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
// CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_5]]] : memref<?xindex> // CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_16:.*]] = %[[VAL_14]] to %[[VAL_15]] step %[[VAL_5]] { // CHECK: %[[VAL_16:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_5]]] : memref<?xindex>
// CHECK: %[[VAL_17:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_16]]] : memref<?xindex> // CHECK: scf.for %[[VAL_17:.*]] = %[[VAL_15]] to %[[VAL_16]] step %[[VAL_5]] {
// CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_16]]] : memref<?xindex> // CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_17]]] : memref<?xindex>
// CHECK: %[[VAL_19:.*]] = arith.addi %[[VAL_16]], %[[VAL_5]] : index // CHECK: %[[VAL_19:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_14]], %[[VAL_18]]] : memref<5x3xi8>
// CHECK: %[[VAL_20:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_19]]] : memref<?xindex> // CHECK: %[[VAL_20:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_17]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_21:.*]] = %[[VAL_18]] to %[[VAL_20]] step %[[VAL_5]] { // CHECK: %[[VAL_21:.*]] = arith.addi %[[VAL_17]], %[[VAL_5]] : index
// CHECK: %[[VAL_22:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_21]]] : memref<?xindex> // CHECK: %[[VAL_22:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_21]]] : memref<?xindex>
// CHECK: %[[VAL_23:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_21]]] : memref<?xi8> // CHECK: scf.for %[[VAL_23:.*]] = %[[VAL_20]] to %[[VAL_22]] step %[[VAL_5]] {
// CHECK: scf.for %[[VAL_24:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] { // CHECK: %[[VAL_24:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_23]]] : memref<?xindex>
// CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_24]], %[[VAL_22]]] : memref<5x6xi64> // CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_14]], %[[VAL_24]]] : memref<5x6xi64>
// CHECK: %[[VAL_26:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_24]], %[[VAL_17]]] : memref<5x3xi8> // CHECK: %[[VAL_26:.*]] = arith.extsi %[[VAL_19]] : i8 to i64
// CHECK: %[[VAL_27:.*]] = arith.extsi %[[VAL_26]] : i8 to i64 // CHECK: %[[VAL_27:.*]] = arith.subi %[[VAL_26]], %[[VAL_6]] : i64
// CHECK: %[[VAL_28:.*]] = arith.subi %[[VAL_27]], %[[VAL_6]] : i64 // CHECK: %[[VAL_28:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_23]]] : memref<?xi8>
// CHECK: %[[VAL_29:.*]] = arith.extsi %[[VAL_23]] : i8 to i64 // CHECK: %[[VAL_29:.*]] = arith.extsi %[[VAL_28]] : i8 to i64
// CHECK: %[[VAL_30:.*]] = arith.muli %[[VAL_28]], %[[VAL_29]] : i64 // CHECK: %[[VAL_30:.*]] = arith.muli %[[VAL_27]], %[[VAL_29]] : i64
// CHECK: %[[VAL_31:.*]] = arith.addi %[[VAL_25]], %[[VAL_30]] : i64 // CHECK: %[[VAL_31:.*]] = arith.addi %[[VAL_25]], %[[VAL_30]] : i64
// CHECK: memref.store %[[VAL_31]], %[[VAL_13]]{{\[}}%[[VAL_24]], %[[VAL_22]]] : memref<5x6xi64> // CHECK: memref.store %[[VAL_31]], %[[VAL_13]]{{\[}}%[[VAL_14]], %[[VAL_24]]] : memref<5x6xi64>
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: %[[VAL_32:.*]] = bufferization.to_tensor %[[VAL_13]] : memref<5x6xi64> // CHECK: %[[VAL_32:.*]] = bufferization.to_tensor %[[VAL_13]] : memref<5x6xi64>
// CHECK: return %[[VAL_32]] : tensor<5x6xi64> // CHECK: return %[[VAL_32]] : tensor<5x6xi64>
// CHECK: } // CHECK: }
func.func @quantized_matmul(%input1: tensor<5x3xi8>, func.func @quantized_matmul(%input1: tensor<5x3xi8>,
%input2: tensor<3x6xi8, #DCSR>, %input2: tensor<3x6xi8, #DCSR>,
%output: tensor<5x6xi64>) -> tensor<5x6xi64> { %output: tensor<5x6xi64>) -> tensor<5x6xi64> {
%c0 = arith.constant 0 : i32 %c0 = arith.constant 0 : i32
%c2 = arith.constant 2 : i32 %c2 = arith.constant 2 : i32
%0 = linalg.quantized_matmul %0 = linalg.quantized_matmul
ins(%input1, %input2, %c2, %c0 : tensor<5x3xi8>, tensor<3x6xi8, #DCSR>, i32, i32) ins(%input1, %input2, %c2, %c0 : tensor<5x3xi8>, tensor<3x6xi8, #DCSR>, i32, i32)
outs(%output : tensor<5x6xi64>) -> tensor<5x6xi64> outs(%output : tensor<5x6xi64>) -> tensor<5x6xi64>
return %0: tensor<5x6xi64> return %0: tensor<5x6xi64>
} }
// CHECK-LABEL: func.func @sparse_dot( // CHECK-LABEL: func.func @sparse_dot(
// CHECK-SAME: %[[VAL_0:.*0]]: tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>, // CHECK-SAME: %[[VAL_0:.*0]]: tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>,
// CHECK-SAME: %[[VAL_1:.*1]]: tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>, // CHECK-SAME: %[[VAL_1:.*1]]: tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>>,
// CHECK-SAME: %[[VAL_2:.*2]]: tensor<f32>) -> tensor<f32> { // CHECK-SAME: %[[VAL_2:.*2]]: tensor<f32>) -> tensor<f32> {
// CHECK-DAG: %[[VAL_3:.*]] = arith.constant 0 : index // CHECK-DAG: %[[VAL_3:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_4:.*]] = arith.constant 1 : index // CHECK-DAG: %[[VAL_4:.*]] = arith.constant 1 : index
// CHECK-DAG: %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_5:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
// CHECK-DAG: %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_6:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
// CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32> // CHECK-DAG: %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
// CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_8:.*]] = sparse_tensor.pointers %[[VAL_1]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
// CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex> // CHECK-DAG: %[[VAL_9:.*]] = sparse_tensor.indices %[[VAL_1]] {dimension = 0 : index} : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xindex>
// CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32> // CHECK-DAG: %[[VAL_10:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<1024xf32, #sparse_tensor.encoding<{ dimLevelType = [ "compressed" ] }>> to memref<?xf32>
// CHECK: %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<f32> // CHECK: %[[VAL_11:.*]] = bufferization.to_memref %[[VAL_2]] : memref<f32>
// CHECK: %[[VAL_12:.*]] = memref.load %[[VAL_11]][] : memref<f32> // CHECK: %[[VAL_12:.*]] = memref.load %[[VAL_11]][] : memref<f32>
// CHECK: %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex> // CHECK: %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_3]]] : memref<?xindex>
// CHECK: %[[VAL_14:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_4]]] : memref<?xindex> // CHECK: %[[VAL_14:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_3]]] : memref<?xindex> // CHECK: %[[VAL_15:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_3]]] : memref<?xindex>
// CHECK: %[[VAL_16:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_4]]] : memref<?xindex> // CHECK: %[[VAL_16:.*]] = memref.load %[[VAL_8]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: %[[VAL_17:.*]]:3 = scf.while (%[[VAL_18:.*]] = %[[VAL_13]], %[[VAL_19:.*]] = %[[VAL_15]], %[[VAL_20:.*]] = %[[VAL_12]]) : (index, index, f32) -> (index, index, f32) { // CHECK: %[[VAL_17:.*]]:3 = scf.while (%[[VAL_18:.*]] = %[[VAL_13]], %[[VAL_19:.*]] = %[[VAL_15]], %[[VAL_20:.*]] = %[[VAL_12]]) : (index, index, f32) -> (index, index, f32) {
// CHECK: %[[VAL_21:.*]] = arith.cmpi ult, %[[VAL_18]], %[[VAL_14]] : index // CHECK: %[[VAL_21:.*]] = arith.cmpi ult, %[[VAL_18]], %[[VAL_14]] : index
// CHECK: %[[VAL_22:.*]] = arith.cmpi ult, %[[VAL_19]], %[[VAL_16]] : index // CHECK: %[[VAL_22:.*]] = arith.cmpi ult, %[[VAL_19]], %[[VAL_16]] : index
// CHECK: %[[VAL_23:.*]] = arith.andi %[[VAL_21]], %[[VAL_22]] : i1 // CHECK: %[[VAL_23:.*]] = arith.andi %[[VAL_21]], %[[VAL_22]] : i1
// CHECK: scf.condition(%[[VAL_23]]) %[[VAL_18]], %[[VAL_19]], %[[VAL_20]] : index, index, f32 // CHECK: scf.condition(%[[VAL_23]]) %[[VAL_18]], %[[VAL_19]], %[[VAL_20]] : index, index, f32
// CHECK: } do { // CHECK: } do {
PeimingAuthorUnsubmitted
Done
ReplyInline Actions

Inefficient access pattern

Peiming: Inefficient access pattern
// CHECK: ^bb0(%[[VAL_24:.*]]: index, %[[VAL_25:.*]]: index, %[[VAL_26:.*]]: f32): // CHECK: ^bb0(%[[VAL_24:.*]]: index, %[[VAL_25:.*]]: index, %[[VAL_26:.*]]: f32):
// CHECK: %[[VAL_27:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_24]]] : memref<?xindex> // CHECK: %[[VAL_27:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_24]]] : memref<?xindex>
// CHECK: %[[VAL_28:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_25]]] : memref<?xindex> // CHECK: %[[VAL_28:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_25]]] : memref<?xindex>
// CHECK: %[[VAL_29:.*]] = arith.cmpi ult, %[[VAL_28]], %[[VAL_27]] : index // CHECK: %[[VAL_29:.*]] = arith.cmpi ult, %[[VAL_28]], %[[VAL_27]] : index
// CHECK: %[[VAL_30:.*]] = arith.select %[[VAL_29]], %[[VAL_28]], %[[VAL_27]] : index // CHECK: %[[VAL_30:.*]] = arith.select %[[VAL_29]], %[[VAL_28]], %[[VAL_27]] : index
// CHECK: %[[VAL_31:.*]] = arith.cmpi eq, %[[VAL_27]], %[[VAL_30]] : index // CHECK: %[[VAL_31:.*]] = arith.cmpi eq, %[[VAL_27]], %[[VAL_30]] : index
// CHECK: %[[VAL_32:.*]] = arith.cmpi eq, %[[VAL_28]], %[[VAL_30]] : index // CHECK: %[[VAL_32:.*]] = arith.cmpi eq, %[[VAL_28]], %[[VAL_30]] : index
// CHECK: %[[VAL_33:.*]] = arith.andi %[[VAL_31]], %[[VAL_32]] : i1 // CHECK: %[[VAL_33:.*]] = arith.andi %[[VAL_31]], %[[VAL_32]] : i1
Show All 29 Lines

mlir/test/Dialect/SparseTensor/sparse_sddmm.mlir

Show First 20 Lines • Show All 50 Lines • ▼ Show 20 Lines %1 = linalg.generic {indexing_maps = [affine_map<(d0) -> (d0)>],
iterator_types = ["parallel"]} iterator_types = ["parallel"]}
outs(%0 : tensor<32xf64>) { outs(%0 : tensor<32xf64>) {
^bb0(%x: f64): ^bb0(%x: f64):
linalg.yield %cst : f64 linalg.yield %cst : f64
} -> tensor<32xf64> } -> tensor<32xf64>
return %1 : tensor<32xf64> return %1 : tensor<32xf64>
} }
// CHECK-LABEL: func.func @sampled_dd_unfused( // CHECK-LABEL: func.func @sampled_dd_unfused(
// CHECK-SAME: %[[VAL_0:.*]]: tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>>, // CHECK-SAME: %[[VAL_0:.*]]: tensor<8x8xf64, #sparse_tensor.encoding<{ dimLevelType = [ "compressed", "compressed" ] }>>,
// CHECK-SAME: %[[VAL_1:.*]]: tensor<8x8xf64>, // CHECK-SAME: %[[VAL_1:.*]]: tensor<8x8xf64>,
// CHECK-SAME: %[[VAL_2:.*]]: tensor<8x8xf64>) -> tensor<8x8xf64> { // CHECK-SAME: %[[VAL_2:.*]]: tensor<8x8xf64>) -> tensor<8x8xf64> {
// CHECK-DAG: %[[VAL_3:.*]] = arith.constant 8 : index // CHECK-DAG: %[[VAL_3:.*]] = arith.constant 8 : index
// CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index // CHECK-DAG: %[[VAL_4:.*]] = arith.constant 0 : index
// CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index // CHECK-DAG: %[[VAL_5:.*]] = arith.constant 1 : index
// CHECK-DAG: %[[VAL_6:.*]] = arith.constant dense<0.000000e+00> : tensor<8x8xf64> // CHECK-DAG: %[[VAL_6:.*]] = arith.constant dense<0.000000e+00> : tensor<8x8xf64>
// CHECK: %[[VAL_7:.*]] = bufferization.alloc_tensor() copy(%[[VAL_6]]) {bufferization.escape = [false]} : tensor<8x8xf64> // CHECK-DAG: %[[VAL_7:.*]] = bufferization.alloc_tensor() copy(%[[VAL_6]]) {bufferization.escape = [false]} : tensor<8x8xf64>
// CHECK: %[[VAL_8:.*]] = bufferization.alloc_tensor() copy(%[[VAL_6]]) {bufferization.escape = [false], memory_space = 0 : i64} : tensor<8x8xf64> // CHECK-DAG: %[[VAL_8:.*]] = bufferization.alloc_tensor() copy(%[[VAL_6]]) {bufferization.escape = [false], memory_space = 0 : i64} : tensor<8x8xf64>
// CHECK: %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<8x8xf64> // CHECK-DAG: %[[VAL_9:.*]] = bufferization.to_memref %[[VAL_1]] : memref<8x8xf64>
// CHECK: %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<8x8xf64> // CHECK-DAG: %[[VAL_10:.*]] = bufferization.to_memref %[[VAL_2]] : memref<8x8xf64>
// CHECK: %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex> // CHECK-DAG: %[[VAL_11:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 0 : index}
// CHECK: %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex> // CHECK-DAG: %[[VAL_12:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 0 : index}
aartbikUnsubmitted
Done
ReplyInline Actions

and here

aartbik: and here
// CHECK: %[[VAL_13:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex> // CHECK-DAG: %[[VAL_13:.*]] = sparse_tensor.pointers %[[VAL_0]] {dimension = 1 : index}
// CHECK: %[[VAL_14:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index} : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xindex> // CHECK-DAG: %[[VAL_14:.*]] = sparse_tensor.indices %[[VAL_0]] {dimension = 1 : index}
// CHECK: %[[VAL_15:.*]] = sparse_tensor.values %[[VAL_0]] : tensor<8x8xf64, #sparse_tensor.encoding<{{.*}}>> to memref<?xf64> // CHECK-DAG: %[[VAL_15:.*]] = sparse_tensor.values %[[VAL_0]]
// CHECK: %[[VAL_16:.*]] = bufferization.to_memref %[[VAL_8]] : memref<8x8xf64> // CHECK: %[[VAL_16:.*]] = bufferization.to_memref %[[VAL_8]] : memref<8x8xf64>
// CHECK: %[[VAL_17:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_4]]] : memref<?xindex> // CHECK: %[[VAL_17:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_4]]] : memref<?xindex>
// CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_5]]] : memref<?xindex> // CHECK: %[[VAL_18:.*]] = memref.load %[[VAL_11]]{{\[}}%[[VAL_5]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_19:.*]] = %[[VAL_17]] to %[[VAL_18]] step %[[VAL_5]] { // CHECK: scf.for %[[VAL_19:.*]] = %[[VAL_17]] to %[[VAL_18]] step %[[VAL_5]] {
// CHECK: %[[VAL_20:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_19]]] : memref<?xindex> // CHECK: %[[VAL_20:.*]] = memref.load %[[VAL_12]]{{\[}}%[[VAL_19]]] : memref<?xindex>
// CHECK: %[[VAL_21:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_19]]] : memref<?xindex> // CHECK: scf.for %[[VAL_21:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] {
// CHECK: %[[VAL_22:.*]] = arith.addi %[[VAL_19]], %[[VAL_5]] : index // CHECK: %[[VAL_22:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_20]], %[[VAL_21]]] : memref<8x8xf64>
// CHECK: %[[VAL_23:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_22]]] : memref<?xindex> // CHECK: %[[VAL_23:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_19]]] : memref<?xindex>
// CHECK: scf.for %[[VAL_24:.*]] = %[[VAL_21]] to %[[VAL_23]] step %[[VAL_5]] { // CHECK: %[[VAL_24:.*]] = arith.addi %[[VAL_19]], %[[VAL_5]] : index
// CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_14]]{{\[}}%[[VAL_24]]] : memref<?xindex> // CHECK: %[[VAL_25:.*]] = memref.load %[[VAL_13]]{{\[}}%[[VAL_24]]] : memref<?xindex>
// CHECK: %[[VAL_26:.*]] = memref.load %[[VAL_16]]{{\[}}%[[VAL_20]], %[[VAL_25]]] : memref<8x8xf64> // CHECK: scf.for %[[VAL_26:.*]] = %[[VAL_23]] to %[[VAL_25]] step %[[VAL_5]] {
// CHECK: %[[VAL_27:.*]] = memref.load %[[VAL_15]]{{\[}}%[[VAL_24]]] : memref<?xf64> // CHECK: %[[VAL_27:.*]] = memref.load %[[VAL_14]]{{\[}}%[[VAL_26]]] : memref<?xindex>
// CHECK: %[[VAL_28:.*]] = scf.for %[[VAL_29:.*]] = %[[VAL_4]] to %[[VAL_3]] step %[[VAL_5]] iter_args(%[[VAL_30:.*]] = %[[VAL_26]]) -> (f64) { // CHECK: %[[VAL_28:.*]] = memref.load %[[VAL_16]]{{\[}}%[[VAL_20]], %[[VAL_27]]] : memref<8x8xf64>
// CHECK: %[[VAL_31:.*]] = memref.load %[[VAL_9]]{{\[}}%[[VAL_20]], %[[VAL_29]]] : memref<8x8xf64> // CHECK: %[[VAL_29:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_21]], %[[VAL_27]]] : memref<8x8xf64>
// CHECK: %[[VAL_32:.*]] = memref.load %[[VAL_10]]{{\[}}%[[VAL_29]], %[[VAL_25]]] : memref<8x8xf64> // CHECK: %[[VAL_30:.*]] = arith.mulf %[[VAL_22]], %[[VAL_29]] : f64
PeimingAuthorUnsubmitted
Done
ReplyInline Actions

Inefficient access pattern

Peiming: Inefficient access pattern
// CHECK: %[[VAL_33:.*]] = arith.mulf %[[VAL_31]], %[[VAL_32]] : f64 // CHECK: %[[VAL_31:.*]] = memref.load %[[VAL_15]]{{\[}}%[[VAL_26]]] : memref<?xf64>
// CHECK: %[[VAL_34:.*]] = arith.mulf %[[VAL_33]], %[[VAL_27]] : f64 // CHECK: %[[VAL_32:.*]] = arith.mulf %[[VAL_30]], %[[VAL_31]] : f64
// CHECK: %[[VAL_35:.*]] = arith.addf %[[VAL_30]], %[[VAL_34]] : f64 // CHECK: %[[VAL_33:.*]] = arith.addf %[[VAL_28]], %[[VAL_32]] : f64
// CHECK: scf.yield %[[VAL_35]] : f64 // CHECK: memref.store %[[VAL_33]], %[[VAL_16]]{{\[}}%[[VAL_20]], %[[VAL_27]]] : memref<8x8xf64>
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: memref.store %[[VAL_24:.*]], %[[VAL_16]]{{\[}}%[[VAL_20]], %[[VAL_25]]] : memref<8x8xf64> // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: } {"Emitted from" = "linalg.generic"}
// CHECK: } // CHECK: %[[VAL_34:.*]] = bufferization.to_tensor %[[VAL_16]] : memref<8x8xf64>
// CHECK: %[[VAL_37:.*]] = bufferization.to_tensor %[[VAL_16]] : memref<8x8xf64> // CHECK: return %[[VAL_34]] : tensor<8x8xf64>
// CHECK: return %[[VAL_37]] : tensor<8x8xf64>
// CHECK: } // CHECK: }
func.func @sampled_dd_unfused(%args: tensor<8x8xf64, #SM>, func.func @sampled_dd_unfused(%args: tensor<8x8xf64, #SM>,
%arga: tensor<8x8xf64>, %arga: tensor<8x8xf64>,
%argb: tensor<8x8xf64>) -> tensor<8x8xf64> { %argb: tensor<8x8xf64>) -> tensor<8x8xf64> {
// Perform dense-dense matrix matrix multiplication. // Perform dense-dense matrix matrix multiplication.
%1 = arith.constant dense<0.0> : tensor<8x8xf64> %1 = arith.constant dense<0.0> : tensor<8x8xf64>
%2 = linalg.generic #trait_matmul %2 = linalg.generic #trait_matmul
ins(%arga, %argb : tensor<8x8xf64>, tensor<8x8xf64>) ins(%arga, %argb : tensor<8x8xf64>, tensor<8x8xf64>)
outs(%1 : tensor<8x8xf64>) { outs(%1 : tensor<8x8xf64>) {
▲ Show 20 Lines • Show All 98 Lines • Show Last 20 Lines