[mlir][sparse] Expose SparseTensor passes as enums instead of opaque numbers for vectorization and parallelization options.

[nkreeger]

Summary

Fixes https://github.com/llvm/llvm-project/issues/53389

The SparseTensor passes currently use opaque numbers for the CLI, despite using an enum internally. This patch exposes the enums instead of numbered items that are matched back to the enum.

Please also see:
https://reviews.llvm.org/D118379
https://reviews.llvm.org/D117919

[mehdi_amini]

Nice! LGTM but please for Aart to review.

Do you have commit access or do you need someone to land this for you?

[nkreeger]

In D123876#3454790, @mehdi_amini wrote:

Nice! LGTM but please for Aart to review.

Do you have commit access or do you need someone to land this for you?

Yeah Aart is on the review - I don't have commit access, first patch :-)

[aartbik]

Thanks for the change. One nit question.

mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td

70

Is this "Some desc here", here and below, a TBD, or an oversight?

[nkreeger]

In D123876#3455626, @aartbik wrote:

Thanks for the change. One nit question.

Yes, sorry an oversight. Updated with some comments in the .td file.

[rriddle]

mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td

69

FYI, You can use [{ <string-contents-here> }] to create a multi line string literal.

69

That would remove the need to escape the inner strings as well.

[nkreeger]

Updated - thanks for the eyes

[aartbik]

Thanks for doing this. I have a few nits on the description text, please fix those before submitting, but no need for a re-review.

mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td

79

Typo: L79 should reay "Enable *dense* ...."
Also, I would not repeat "strategy" here

90

Same here and below, just say

Enable vectorization for ....

[nkreeger]

Updated diff - sorry I don't have commit access.

[nkreeger]

Friendly ping here - lemme know if I need to something more to get this thing landed. I don't have LLVM commit rights. Thanks!

[mehdi_amini]

Sorry, I missed this... Can you please rebase? There is a conflict in a test.

Also: please email Chris to get commit access if you're planning to continue send more changes :)

[nkreeger]

Updated - thanks Mehdi - I'll drop Chris a note.

[nkreeger]

Looks like build-bot didn't like this test for NVIDIA: https://lab.llvm.org/buildbot/#/builders/61/builds/25447

[aartbik]

In D123876#3469139, @nkreeger wrote:

Friendly ping here - lemme know if I need to something more to get this thing landed. I don't have LLVM commit rights. Thanks!

Oops, I missed this follow up discussion. Apologies. I will commit this!

[nkreeger]

Hey Aart - the integration tests haven't passed on GPU - I'm working on a
fix but I haven't had a chance to get back to it this week.

Additionally, I have commit access now thanks to Chris :-)

[aartbik]

In D123876#3477552, @nkreeger wrote:

Additionally, I have commit access now thanks to Chris :-)

In that case, I will let you do the honors! (after the fix of course ;-)

[aartbik]

Just curious on the status of this change?

[nkreeger]

In D123876#3593116, @aartbik wrote:

Just curious on the status of this change?

It is stuck on a command line parse error with the integration tests with the enum names. I haven't had a good chance to sit in the debugger and figure out what is going on. I'll try and circle back next week on it.

[aartbik]

In D123876#3593149, @nkreeger wrote:

It is stuck on a command line parse error with the integration tests with the enum names. I haven't had a good chance to sit in the debugger and figure out what is going on. I'll try and circle back next week on it.

Thanks for the update. No rush. I was just going through my "old" list of revisions to check ;-)

[nkreeger]

Updated patch - passes integration tests.

Problem was that mlir/include/mlir/Dialect/SparseTensor/Pipelines/Passes.h needed to add all the options in struct SparseCompilerOptions.

@aartbik lemme know if you want me to file an issue on that. @Mogball helped me run that down.

[aartbik]

Terribly sorry, but can you please rebase this with main? I think some changes will need resolution first.

[nkreeger]

rebased diff.

[nkreeger]

In D123876#3756434, @aartbik wrote:

Terribly sorry, but can you please rebase this with main? I think some changes will need resolution first.

Let me know if this updated one looks good and I'll land it finally :-)

[aartbik]

I am fine submitting this, since I have kept you waiting for (too) long.
But the duplication of the options in td/h file worries me, that seems like a consistency nightmare.
Can you please file an issue on that, and add a TODO in the header file (as in, // TODO(issue): need to duplicate enums, but will be fixed....

[nkreeger]

Done, filed https://github.com/llvm/llvm-project/issues/57514 - from when I tried to run it down it looked like a parser bug in mlir.

I'll land later today or tomorrow - thanks!

[nkreeger]

Looks like CI is funny again. I'll circle back on Monday and try again.

[nkreeger]

header madness needed to move below declared structs.

mlir/benchmark/python/common.py

     """Setup pass pipeline parameters for benchmark functions.
     """
     opt = (
-        "parallelization-strategy=0"
-        " vectorization-strategy=0 vl=1 enable-simd-index32=False"
+        "parallelization-strategy=none"
+        " vectorization-strategy=none vl=1 enable-simd-index32=False"
     )
     pipeline = f"sparse-compiler{{{opt}}}"
     PassManager.parse(pipeline).run(mlir_module)

mlir/include/mlir/Dialect/SparseTensor/Pipelines/Passes.h

 struct SparseCompilerOptions
     : public PassPipelineOptions<SparseCompilerOptions> {
   // These options must be kept in sync with `SparsificationBase`.
-  PassOptions::Option<int32_t> parallelization{
+  // TODO(57514): These options are duplicated in Passes.td.
+  PassOptions::Option<mlir::SparseParallelizationStrategy> parallelization{
       *this, "parallelization-strategy",
-      desc("Set the parallelization strategy"), init(0)};
-  PassOptions::Option<int32_t> vectorization{
-      *this, "vectorization-strategy", desc("Set the vectorization strategy"),
-      init(0)};
+      ::llvm::cl::desc("Set the parallelization strategy"),
+      ::llvm::cl::init(mlir::SparseParallelizationStrategy::kNone),
+      llvm::cl::values(
+          clEnumValN(mlir::SparseParallelizationStrategy::kNone, "none",
+                     "Turn off sparse parallelization."),
+          clEnumValN(mlir::SparseParallelizationStrategy::kDenseOuterLoop,
+                     "dense-outer-loop",
+                     "Enable dense outer loop sparse parallelization."),
+          clEnumValN(mlir::SparseParallelizationStrategy::kAnyStorageOuterLoop,
+                     "any-storage-outer-loop",
+                     "Enable sparse parallelization regardless of storage for "
+                     "the outer loop."),
+          clEnumValN(mlir::SparseParallelizationStrategy::kDenseAnyLoop,
+                     "dense-any-loop",
+                     "Enable dense parallelization for any loop."),
+          clEnumValN(
+              mlir::SparseParallelizationStrategy::kAnyStorageAnyLoop,
+              "any-storage-any-loop",
+              "Enable sparse parallelization for any storage and loop."))};
+  PassOptions::Option<mlir::SparseVectorizationStrategy> vectorization{
+      *this, "vectorization-strategy",
+      ::llvm::cl::desc("Set the vectorization strategy"),
+      ::llvm::cl::init(mlir::SparseVectorizationStrategy::kNone),
+      llvm::cl::values(
+          clEnumValN(mlir::SparseVectorizationStrategy::kNone, "none",
+                     "Turn off sparse vectorization."),
+          clEnumValN(mlir::SparseVectorizationStrategy::kDenseInnerLoop,
+                     "dense-inner-loop",
+                     "Enable vectorization for dense inner loops."),
+          clEnumValN(mlir::SparseVectorizationStrategy::kAnyStorageInnerLoop,
+                     "any-storage-inner-loop",
+                     "Enable sparse vectorization for inner loops with any "
+                     "storage."))};
+
   PassOptions::Option<int32_t> vectorLength{
       *this, "vl", desc("Set the vector length"), init(1)};
   PassOptions::Option<bool> enableSIMDIndex32{

 
   /// Projects out the options for `createSparsificationPass`.
   SparsificationOptions sparsificationOptions() const {
-    return SparsificationOptions(sparseParallelizationStrategy(parallelization),
-                                 sparseVectorizationStrategy(vectorization),
-                                 vectorLength, enableSIMDIndex32,
-                                 enableVLAVectorization);
+    return SparsificationOptions(parallelization, vectorization, vectorLength,
+                                 enableSIMDIndex32, enableVLAVectorization);
   }
 
   // These options must be kept in sync with `SparseTensorConversionBase`.

mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.h

   // TODO: support reduction parallelization too?
 };
 
-/// Converts command-line parallelization flag to the strategy enum.
-SparseParallelizationStrategy sparseParallelizationStrategy(int32_t flag);
-
 /// Defines a vectorization strategy. Any inner loop is a candidate (full SIMD
 /// for parallel loops and horizontal SIMD for reduction loops). A loop is
 /// actually vectorized if (1) allowed by the strategy, and (2) the emitted

   kAnyStorageInnerLoop
 };
 
-/// Converts command-line vectorization flag to the strategy enum.
-SparseVectorizationStrategy sparseVectorizationStrategy(int32_t flag);
-
 /// Options for the Sparsification pass.
 struct SparsificationOptions {
   SparsificationOptions(SparseParallelizationStrategy p,

mlir/include/mlir/Dialect/SparseTensor/Transforms/Passes.td

     "sparse_tensor::SparseTensorDialect",
     "vector::VectorDialect",
   ];
+  // TODO(57514): These enum options are duplicated in Passes.h.
   let options = [
-    Option<"parallelization", "parallelization-strategy", "int32_t", "0",
-           "Set the parallelization strategy">,
-    Option<"vectorization", "vectorization-strategy", "int32_t", "0",
-           "Set the vectorization strategy">,
+    Option<"parallelization", "parallelization-strategy", "mlir::SparseParallelizationStrategy",
+           "mlir::SparseParallelizationStrategy::kNone",
+           "Set the parallelization strategy", [{llvm::cl::values(
+             clEnumValN(mlir::SparseParallelizationStrategy::kNone, "none",
+                        "Turn off sparse parallelization."),
+             clEnumValN(mlir::SparseParallelizationStrategy::kDenseOuterLoop,
+                        "dense-outer-loop",
+                        "Enable dense outer loop sparse parallelization."),
+             clEnumValN(mlir::SparseParallelizationStrategy::kAnyStorageOuterLoop,
+                        "any-storage-outer-loop",
+                        "Enable sparse parallelization regardless of storage for the outer loop."),
+             clEnumValN(mlir::SparseParallelizationStrategy::kDenseAnyLoop,
+                        "dense-any-loop",
+                        "Enable dense parallelization for any loop."),
+             clEnumValN(mlir::SparseParallelizationStrategy::kAnyStorageAnyLoop,
+                        "any-storage-any-loop",
+                        "Enable sparse parallelization for any storage and loop."))}]>,
+    Option<"vectorization", "vectorization-strategy", "mlir::SparseVectorizationStrategy",
+           "mlir::SparseVectorizationStrategy::kNone",
+           "Set the vectorization strategy", [{llvm::cl::values(
+             clEnumValN(mlir::SparseVectorizationStrategy::kNone, "none",
+                        "Turn off sparse vectorization."),
+             clEnumValN(mlir::SparseVectorizationStrategy::kDenseInnerLoop,
+                        "dense-inner-loop",
+                        "Enable vectorization for dense inner loops."),
+             clEnumValN(mlir::SparseVectorizationStrategy::kAnyStorageInnerLoop,
+                        "any-storage-inner-loop",
+                        "Enable sparse vectorization for inner loops with any storage."))}]>,
     Option<"vectorLength", "vl", "int32_t", "1",
            "Set the vector length">,
     Option<"enableSIMDIndex32", "enable-simd-index32", "bool", "false",

mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp

   SparsificationPass() = default;
   SparsificationPass(const SparsificationPass &pass) = default;
   SparsificationPass(const SparsificationOptions &options) {
-    parallelization = static_cast<int32_t>(options.parallelizationStrategy);
-    vectorization = static_cast<int32_t>(options.vectorizationStrategy);
+    parallelization = options.parallelizationStrategy;
+    vectorization = options.vectorizationStrategy;
     vectorLength = options.vectorLength;
     enableSIMDIndex32 = options.enableSIMDIndex32;
     enableVLAVectorization = options.enableVLAVectorization;

     populateSparseTensorRewriting(prePatterns);
     (void)applyPatternsAndFoldGreedily(getOperation(), std::move(prePatterns));
     // Translate strategy flags to strategy options.
-    SparsificationOptions options(
-        sparseParallelizationStrategy(parallelization),
-        sparseVectorizationStrategy(vectorization), vectorLength,
-        enableSIMDIndex32, enableVLAVectorization);
+    SparsificationOptions options(parallelization, vectorization, vectorLength,
+                                  enableSIMDIndex32, enableVLAVectorization);
     // Apply sparsification and vector cleanup rewriting.
     RewritePatternSet patterns(ctx);
     populateSparsificationPatterns(patterns, options);

 // Strategy flag methods.
 //===----------------------------------------------------------------------===//
 
-SparseParallelizationStrategy
-mlir::sparseParallelizationStrategy(int32_t flag) {
-  switch (flag) {
-  default:
-    return SparseParallelizationStrategy::kNone;
-  case 1:
-    return SparseParallelizationStrategy::kDenseOuterLoop;
-  case 2:
-    return SparseParallelizationStrategy::kAnyStorageOuterLoop;
-  case 3:
-    return SparseParallelizationStrategy::kDenseAnyLoop;
-  case 4:
-    return SparseParallelizationStrategy::kAnyStorageAnyLoop;
-  }
-}
-
-SparseVectorizationStrategy mlir::sparseVectorizationStrategy(int32_t flag) {
-  switch (flag) {
-  default:
-    return SparseVectorizationStrategy::kNone;
-  case 1:
-    return SparseVectorizationStrategy::kDenseInnerLoop;
-  case 2:
-    return SparseVectorizationStrategy::kAnyStorageInnerLoop;
-  }
-}
-
 SparseToSparseConversionStrategy
 mlir::sparseToSparseConversionStrategy(int32_t flag) {
   switch (flag) {

mlir/test/Dialect/SparseTensor/sparse_parallel.mlir

-// RUN: mlir-opt %s -sparsification="parallelization-strategy=0" | \
+// RUN: mlir-opt %s -sparsification="parallelization-strategy=none" | \
 // RUN:   FileCheck %s --check-prefix=CHECK-PAR0
-// RUN: mlir-opt %s -sparsification="parallelization-strategy=1" | \
+// RUN: mlir-opt %s -sparsification="parallelization-strategy=dense-outer-loop" | \
 // RUN:   FileCheck %s --check-prefix=CHECK-PAR1
-// RUN: mlir-opt %s -sparsification="parallelization-strategy=2" | \
+// RUN: mlir-opt %s -sparsification="parallelization-strategy=any-storage-outer-loop" | \
 // RUN:   FileCheck %s --check-prefix=CHECK-PAR2
-// RUN: mlir-opt %s -sparsification="parallelization-strategy=3" | \
+// RUN: mlir-opt %s -sparsification="parallelization-strategy=dense-any-loop" | \
 // RUN:   FileCheck %s --check-prefix=CHECK-PAR3
-// RUN: mlir-opt %s -sparsification="parallelization-strategy=4" | \
+// RUN: mlir-opt %s -sparsification="parallelization-strategy=any-storage-any-loop" | \
 // RUN:   FileCheck %s --check-prefix=CHECK-PAR4
 
 #DenseMatrix = #sparse_tensor.encoding<{

mlir/test/Dialect/SparseTensor/sparse_vector.mlir

-// RUN: mlir-opt %s -sparsification="vectorization-strategy=0 vl=16" -cse -split-input-file | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=none vl=16" -cse -split-input-file | \
 // RUN:   FileCheck %s --check-prefix=CHECK-VEC0
-// RUN: mlir-opt %s -sparsification="vectorization-strategy=1 vl=16" -cse -split-input-file | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=dense-inner-loop vl=16" -cse -split-input-file | \
 // RUN:   FileCheck %s --check-prefix=CHECK-VEC1
-// RUN: mlir-opt %s -sparsification="vectorization-strategy=2 vl=16" -cse -split-input-file | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=any-storage-inner-loop vl=16" -cse -split-input-file | \
 // RUN:   FileCheck %s --check-prefix=CHECK-VEC2
-// RUN: mlir-opt %s -sparsification="vectorization-strategy=2 vl=16 enable-simd-index32=true" -cse -split-input-file | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=any-storage-inner-loop vl=16 enable-simd-index32=true" -cse -split-input-file | \
 // RUN:   FileCheck %s --check-prefix=CHECK-VEC3
-// RUN: mlir-opt %s -sparsification="vectorization-strategy=2 vl=4 enable-vla-vectorization=true" -cse -split-input-file | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=any-storage-inner-loop vl=4 enable-vla-vectorization=true" -cse -split-input-file | \
 // RUN:   FileCheck %s --check-prefix=CHECK-VEC4
 
 #DenseVector = #sparse_tensor.encoding<{ dimLevelType = [ "dense" ] }>

mlir/test/Dialect/SparseTensor/sparse_vector_chain.mlir

 // NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
 
-// RUN: mlir-opt %s -sparsification="vectorization-strategy=2 vl=8" -canonicalize | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=any-storage-inner-loop vl=8" -canonicalize | \
 // RUN:   FileCheck %s
 
 #SparseMatrix = #sparse_tensor.encoding<{dimLevelType = ["dense","compressed"]}>

mlir/test/Dialect/SparseTensor/sparse_vector_index.mlir

 // about what constitutes a good test! The CHECK should be
 // minimized and named to reflect the test intent.
 
-// RUN: mlir-opt %s -sparsification="vectorization-strategy=2 vl=8" -canonicalize | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=any-storage-inner-loop vl=8" -canonicalize | \
 // RUN:   FileCheck %s
 
 #SparseVector = #sparse_tensor.encoding<{

mlir/test/Dialect/SparseTensor/sparse_vector_peeled.mlir

-// RUN: mlir-opt %s -sparsification="vectorization-strategy=2 vl=16" -scf-for-loop-peeling -canonicalize | \
+// RUN: mlir-opt %s -sparsification="vectorization-strategy=any-storage-inner-loop vl=16" -scf-for-loop-peeling -canonicalize | \
 // RUN:   FileCheck %s
 
 #SparseVector = #sparse_tensor.encoding<{

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_cast.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=2" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=2" | \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_filter_conv2d.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=2" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=2" | \
 // RUN: mlir-cpu-runner -e entry -entry-point-result=void \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
 // RUN: FileCheck %s

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_flatten.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=4" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=4" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/test.tns" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index_dense.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=4" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=4" | \
 // RUN: mlir-cpu-runner -e entry -entry-point-result=void \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
 // RUN: FileCheck %s

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matvec.mlir

 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
 // RUN: mlir-opt %s \
-// RUN:   --sparse-compiler="vectorization-strategy=2 vl=16 enable-simd-index32" | \
+// RUN:   --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=16 enable-simd-index32" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/wide.mtx" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_mttkrp.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=4" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=4" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/mttkrp_b.tns" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_out_simple.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=4" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=4" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/test.mtx" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_quantized_matmul.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=2" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=2" | \
 // RUN: mlir-cpu-runner -e entry -entry-point-result=void \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
 // RUN: FileCheck %s

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_reductions.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=8" | \
+// RUN: mlir-opt %s -sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=8" | \
 // RUN: mlir-cpu-runner -e entry -entry-point-result=void \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
 // RUN: FileCheck %s

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_matmul.mlir

 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
 // RUN: mlir-opt %s \
-// RUN:   --sparse-compiler="vectorization-strategy=2 vl=4 enable-simd-index32" | \
+// RUN:   --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=4 enable-simd-index32" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/test.mtx" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_mm_fusion.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=8" | \
+// RUN: mlir-opt %s -sparse-compiler="vl=8" | \
 // RUN: mlir-cpu-runner -e entry -entry-point-result=void \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
 // RUN: FileCheck %s

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_scale.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=4" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=4" | \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \
 // RUN:  -shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_spmm.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=2" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=2" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/wide.mtx" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sum.mlir

 //
 // Do the same run, but now with SIMDization as well. This should not change the outcome.
 //
-// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=2 vl=2" | \
+// RUN: mlir-opt %s --sparse-compiler="vectorization-strategy=any-storage-inner-loop vl=2" | \
 // RUN: TENSOR0="%mlir_integration_test_dir/data/test_symmetric.mtx" \
 // RUN: mlir-cpu-runner \
 // RUN:  -e entry -entry-point-result=void  \

mlir/test/Integration/Dialect/SparseTensor/python/test_SDDMM.py

             ir.AffineMap.get_permutation([0, 1]),
             ir.AffineMap.get_permutation([1, 0])
         ]
+        vec_strategy = [
+          'none', 'dense-inner-loop'
+        ]
         for level in levels:
             for ordering in orderings:
                 for pwidth in [32]:
                     for iwidth in [32]:
-                        for par in [0]:
-                            for vec in [0, 1]:
-                                for e in [True]:
-                                    vl = 1 if vec == 0 else 16
-                                    attr = st.EncodingAttr.get(level, ordering, pwidth, iwidth)
-                                    opt = (f'parallelization-strategy={par} '
-                                           f'vectorization-strategy={vec} '
-                                           f'vl={vl} enable-simd-index32={e}')
-                                    compiler = sparse_compiler.SparseCompiler(
-                                        options=opt, opt_level=0, shared_libs=[support_lib])
-                                    build_compile_and_run_SDDMMM(attr, compiler)
-                                    count = count + 1
+                        for vec in vec_strategy:
+                            for e in [True]:
+                                vl = 1 if vec == 0 else 16
+                                attr = st.EncodingAttr.get(level, ordering, pwidth, iwidth)
+                                opt = (f'parallelization-strategy=none '
+                                       f'vectorization-strategy={vec} '
+                                       f'vl={vl} enable-simd-index32={e}')
+                                compiler = sparse_compiler.SparseCompiler(
+                                    options=opt, opt_level=0, shared_libs=[support_lib])
+                                build_compile_and_run_SDDMMM(attr, compiler)
+                                count = count + 1
     # CHECK: Passed 16 tests
     print('Passed ', count, 'tests')
 

mlir/test/Integration/Dialect/SparseTensor/python/test_SpMM.py

         # a *single* sparse tensor. Note that we deliberate do not exhaustively
         # search the full state space to reduce runtime of the test. It is
         # straightforward to adapt the code below to explore more combinations.
-        par = 0
-        vec = 0
         vl = 1
         e = False
-        opt = (f'parallelization-strategy={par} '
-               f'vectorization-strategy={vec} '
+        opt = (f'parallelization-strategy=none '
+               f'vectorization-strategy=none '
                f'vl={vl} enable-simd-index32={e}')
         levels = [[st.DimLevelType.dense, st.DimLevelType.dense],
                   [st.DimLevelType.dense, st.DimLevelType.compressed],

mlir/test/Integration/Dialect/SparseTensor/python/test_stress.py

   # CHECK-LABEL: TEST: test_stress
   print("\nTEST: test_stress")
   with ir.Context() as ctx, ir.Location.unknown():
-    par = 0
-    vec = 0
     vl = 1
     e = False
     # Disable direct sparse2sparse conversion, because it doubles the time!

     # `s2s=0` on a regular basis, to ensure that it does continue to work.
     s2s = 1
     sparsification_options = (
-        f'parallelization-strategy={par} '
-        f'vectorization-strategy={vec} '
+        f'parallelization-strategy=none '
+        f'vectorization-strategy=none '
         f'vl={vl} '
         f'enable-simd-index32={e} '
         f's2s-strategy={s2s}')