[mlir] Narrow bitwidth emulation for MemRef load

[yzhang93]

Summary

This patch adds support for narrow bitwidth storage emulation. The goal is to support sub-byte type
codegen for LLVM CPU. Specifically, a type converter is added to convert memref of narrow bitwidth
(e.g., i4) into supported wider bitwidth (e.g., i8). Another focus of this patch is to populate the
pattern for int4 memref.load. memref.store pattern should be added in a seperate patch.

[tschuett]

The bigger issue is:
https://llvm.org/docs/CodingStandards.html#anonymous-namespaces

mlir/include/mlir/Dialect/Arith/Transforms/NarrowIntEmulationConverter.h

27

I have a personal issue with the missing new line between the class and the namespace.

mlir/test/Dialect/MemRef/emulate-narrow-int.mlir

82

Newline please.

[hanchung]

Nice work! I dropped few comments inline. :)

Inspired by IREE project patch, I found that some people would like to emulate f16 computation on f32 domain: https://github.com/openxla/iree/pull/13808

I think we can rename the EmulateNarrowInt to EmulateNarrowType or EmulateNarrowNumerics, and they can add such patterns to it later.

mlir/lib/Dialect/Arith/Transforms/EmulateNarrowInt.cpp

40–68

Can we move the conversion out of constructor? Users want to control the conversions themselves. I think we can move them to TestEmulateNarrowInt.cpp.

46–49

We don't tie the type conversion to targetWideInt. Instead, we leave the decision to users. The targetWideInt controls how we load a int4, but not the computation domain. Think the case that we want to use byte load for int4, but leave the computation on int4. We can force the test to convert int4 to int8, or control it with a flag (like int4-arith-bitwidth=8).

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowInt.cpp

90–94

I think we should check if converted MemRef type is as same as original type. The emulation is needed only if memref types mismatch.

The newResTy can be the same in the scenario of using byte load but operating on int4 domain. IMO, the working flow is:

1. Check if the original memref type match converted memref type. If they mismatch, we need the emulation and apply the pattern.
2. Load the value using converted memref type. (which is already implemented below)
3. Cast the load value to newResTy if the types mismatch. In the scenario, a int8 value is loaded and we need to cast it to int4 (i.e., newResTy).

100

Can we add a comment to elaborate the core idea? And some comments about why/how we compute linearized_size, linearized_offset, and %stride#1.

%0 = memref.load %0[%v0][%v1] : memref<?x?xi4, strided<[?, ?], offset = ?>>

can be replaced with

%b, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %0
%linearized_offset = /// %v0 * %stride#0 + %v1 * %stride#1
%linearized_size = /// %size0 * %size1
%linearized = memref.reinterpret_cast %b, offset = [%offset], sizes = [%linearized_size], strides = [%stride#1] 
%load = memref.load %linearized[%linearized_offset] : memref<?xi4, strided<?, offset = ?>>

131–132

style nit: names should be in camelCase, i.e., we should name them to linearizedOffset and linearizedSize.

https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly

141–142

style nit: use auto because XXX::get already spells the type.

https://llvm.org/docs/CodingStandards.html#use-auto-type-deduction-to-make-code-more-readable

200–202

IMO, this should be derived from targetBitwidth. If the bitwidth is less than targetBitwidth, we use targetBitwidth.

mlir/test/lib/Dialect/MemRef/TestEmulateNarrowInt.cpp

67

I can't map the comment to populate functions, maybe we can just remove the comment... It's pretty clear (from the function name) that we populate the patterns to do narrow type emulation.

[yzhang93]

In D151519#4376791, @hanchung wrote:

Nice work! I dropped few comments inline. :)

Inspired by IREE project patch, I found that some people would like to emulate f16 computation on f32 domain: https://github.com/openxla/iree/pull/13808

I think we can rename the EmulateNarrowInt to EmulateNarrowType or EmulateNarrowNumerics, and they can add such patterns to it later.

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowInt.cpp

90–94

Thanks for your review and suggestions! I have modified the codes accordingly.

[kuhar]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowInt.cpp

90–95

Same comment as in https://reviews.llvm.org/D151827

[kuhar]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowInt.cpp

90–95

Also, we should return failure() when there was no rewrite

212–214

nit: when else is wrapped with braces so should the then body

[hanchung]

Overall looks good, just some nits. Please also consider renaming the pass name and file name, thanks!

mlir/include/mlir/Dialect/Arith/Transforms/NarrowIntEmulationConverter.h

16–18

please update the comment as well.

22

There are target bitwidth, one is for load/store emulation, and the other is for arith computation domain. This is the one related to load/store emulation, please update the variable name more concretely.

mlir/include/mlir/Dialect/Arith/Transforms/Passes.h

39–40

Please add a comment about "users need to add conversions about the computation domain of narrow types".

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowInt.cpp

90–94

I think we should if op.getMemRefType() is as same as typeConverter->convertType(op.getMemRefType()). If they are not the same, we need the emulation.

109–119

Can you format it in a better way? Adding spaces and new lines could help, IMO. maybe something like: https://github.com/llvm/llvm-project/blob/7f374b6902fad9caed41284a57d573abe9ada9d1/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h#L450-L476

180–182

It should be typeConverter->convertType(oldElementType).

mlir/test/lib/Dialect/MemRef/TestEmulateNarrowInt.cpp

103–105

The naming is ambiguous (and mismatch between the name and flag), can we rename it to something like loadEmulationBitwidth?

[hanchung]

LGTM if the comments are addressed.

mlir/lib/Dialect/Arith/Transforms/EmulateNarrowType.cpp

10–24 ↗

(On Diff #527745)

I think we can remove VectorOps.h from the includes.

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

98–99 ↗

(On Diff #527745)

we can remove the declaration of source and write it like

MemRefType sourceType = adaptor.getMemRefType();

103–105 ↗

(On Diff #527745)

This can be simplified to op.getMemRefType().getElementType().getIntOrFloatBitWidth()

131 ↗

(On Diff #527745)

I would just use adaptor.getMemref() here

189–191 ↗

(On Diff #527745)

The trunci op is not needed if they have the same number of bits.

224–226 ↗

(On Diff #527745)

can this be auto?

mlir/test/lib/Dialect/MemRef/TestEmulateNarrowType.cpp

57–62 ↗

(On Diff #527745)

the if-else already covers all the cases, this can be simplified.

71–79 ↗

(On Diff #527745)

ditto, this can be simplified

[yzhang93]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

98–99 ↗

(On Diff #527745)

Looks like there's no member named 'getMemRefType' in 'mlir::memref::LoadOpAdaptor'. I'll keep the use of what I have.

[hanchung]

just two nits

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

230–239 ↗

(On Diff #527998)

I think we don't need else, this can save us one level of indents.

mlir/test/lib/Dialect/MemRef/TestEmulateNarrowType.cpp

71–79 ↗

(On Diff #527745)

can we remove the else keyword? that would save us a level of indent. same for above one.

[mravishankar]

Nice! Mostly looks good. Just a few comments.

mlir/include/mlir/Dialect/Arith/Transforms/NarrowTypeEmulationConverter.h

23 ↗

(On Diff #528020)

Make this private and add a getLoadStoreBitwidth method.

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

29–39 ↗

(On Diff #528020)

Language nit :

/// When data is loaded/stored in `targetBits` granularity, but is used in `sourceBits` granularity
/// (`sourceBits` < `targetBits`), the `targetBits` is treated as an array of elements of width `sourceBits`.
/// Return the bit offset of the value at position `srcIdx`. For example, if
/// `sourceBits` equals to 4 and `targetBits` equals to 8, the x-th element is
/// located at (x % 2) * 4. Because there are two elements in one i8, and one
/// element has 4 bits.

90 ↗

(On Diff #528020)

Nit : For statements spanning multiple lines, still it is recommended to use braces.

105 ↗

(On Diff #528020)

Instead of assert just return a failure

return notifyMatchFailure(op, "only dstBits %srcBits == 0 supported");

139 ↗

(On Diff #528020)

I think this needs to happen on the linearizedOffset. Basically

1. find the linearizedOffset.
2. Divide by the scaling factor (which is dstBits / srcBits)
3. Load the value.
4. Get the offset in bits

184 ↗

(On Diff #528020)

Note: This is only relevant for big-endian... Maybe add a comment somewhere that this is the only mode supported for now. Another robust option is to allow setting this in the TypeConverter, and assert that it is the endian-ness expected. Without that it can lead to subtle bugs.

192 ↗

(On Diff #528020)

I am trying to understand when this case happens. The resultType

[yzhang93]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

139 ↗

(On Diff #528020)

@hanchung and I discussed about this before and we thought only the last index needs to be modified. However, I just rethink about this and I agree with you that the scaling needs to happen after the offset is linearized. @hanchung let me know if this makes sense to you.

192 ↗

(On Diff #528020)

This happens when the load bitwidth and computation bitwidth are the same, e.g., when we specify --test-emulate-narrow-int="arith-compute-bitwidth=8 memref-load-bitwidth=8"

[mravishankar]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

192 ↗

(On Diff #528020)

this looks like a premature optimization to me. If the result width and compute width is the same, then there should not be a need to do this.... If the compute width is higher, then the trunc and ext should be folded away as a canonicalization. In any case I actually dont see a test with the --test-emulate-narrow-int="arith-compute-bitwidth=8 memref-load-bitwidth=8". Maybe we just do

if (resultTy != srcElementType) {
   result = rewriter.create<arith::TruncIOp>(loc, resultTy, bitsLoad);
 }

106 ↗

(On Diff #528616)

Nit: This statement spans two lines. Please use braces.

153 ↗

(On Diff #528616)

Two things here

1. First not sure why you need to special case sourceRank == 1?
2. I think this computation is very different from what was there before. The

linearizedOffset = (adjustedOffset[0] + adjustedOffset[1] + ...) * srcBits / dstBits
Here you seem to be dividing the scalar (= dstBits / srcBits) as many times as the sourceRank which seems off.

[yzhang93]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

192 ↗

(On Diff #528020)

My idea behind this is if the emulated memref load bits and the computation bits are the same, e.g., 8 bits, and the actual load is 4 bits in the example. We'll need to return a 8 bits data but only the last 4 bits are the data we needed. So that's why I added a mask to make the first 4 bits zero, and only the last 4 bits are valid. I also added a test for the "arith-compute-bitwidth=8 memref-load-bitwidth=8" case. We can chat in detail if this doesn't make sense to you.

153 ↗

(On Diff #528616)

My bad. Thanks for pointing this out.

[mravishankar]

Thanks Vivian, there are a couple of more bugs in this patch... also left a suggestion to use makeComposedAffineApplyOp which will make the code and IR more readable.

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

152 ↗

(On Diff #529694)

THanks for the changes. Now I understand better. I think I found an issue (sorry if it was triggered by a suggestion from me). This will segfault for zero-rank memrefs.

So this has to be

Value linearizedOffset = builder.create<arith::ConstantIndexOp>(loc, 0).;
Value linearizedSize = builder.create<arith::ConstantIndexOp>(loc, 1);
for (int i = 0; i < sourceRank; ++i) {
  linearizedOffset = rewriter.create<arith::AddIOp>(loc, linearizedOffset, adjustedOffsets[i]);
  linearizedSize = rewriter.create<arith::MulIOp>(loc, linearizedSize, baseSizes[i]);
}

Better yet... instead of creating all these ops we can use makeComposedAffineApplyOp

OpFoldResult linearizedOffset = rewriter.getIndexAttr(0);
OpFoldResult linearizedSize = rewriter.getIndexAttr(1);
AffineExpr s0, s1, s2;
bindSymbols(s0, s1, s2);
for (auto i : llvm::seq<int>(0, sourceRank)) {
  linearizedOffset = makeComposedAffineApplyOp(rewriter, loc, s0 + s1 * s2, {linearizedOffset, indices[i], baseStrides[i]);
  linearizedSize = makeComposedAffineApplyOp(rewriter, loc, s0 * s1, {linearizedSize, baseSizes[i]});
}
OpFoldResult scaler =rewriter.getIndexAttr(dstBits/srcBits);
linearizedOffset = makeComposedAffineApply(rewriter, loc, s0 floorDiv s1, {linearizedOffset, scaler});

Then you can get the Value for linearizedOffset/linearizedSize using getOrCreateConstantIndexOp.

This will fold away any statically known values, and will also make the code easier to read, the IR easier to read, while reducing index arithmetic overhead.

172 ↗

(On Diff #529694)

If I am not mistaken, baseOffset also needs to be scaled.

192 ↗

(On Diff #528020)

Maybe... But i cant think of a valid program where the load is in 4 bits but the use of it is directly in 8-bits....

[yzhang93]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

152 ↗

(On Diff #529694)

Thanks for pointing out the zero-rank problem and I appreciate your suggestions.

I tried what you suggested with AffineApplyOp, but kept having this "error: failed to legalize operation 'memref.load' that was explicitly marked illegal %1 = memref.load %0[%arg0] : memref<4xi4>" on the test even with the simplest test. I'm not sure what caused the error, but if you know any potential issue and the way to fix it please let me know.

Currently I refactor the codes and add the case for sourceRank==0. I think we probably want to treat these cases separately, because when sourceRank==0 we don't need to do linearization with memref.reinterpret_cast op.

[yzhang93]

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

152 ↗

(On Diff #529694)

In the latest revision, I refactored the codes of linearization part with AffineApplyOp as suggested. I also added the conversion pattern for memref::AssumeAlignmentOp, as this is required for e2e test. The tests were updated accordingly.

[mravishankar]

Thanks! This revision looks good!

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowType.cpp

61 ↗

(On Diff #533814)

Nit: Avoid using auto here. It is only used in LLVM when the type is obvious from the context, and here it is not.

85 ↗

(On Diff #533814)

Nit: avoid using the same variable in two contexts.

263 ↗

(On Diff #533814)

I am still not sure about this one... Not really sure this actually happens in practice, but harmless enough. (Could you just leave a comment explaining it isnt clear that this is needed, or something to record this discussion.

mlir/include/mlir/Dialect/Arith/Transforms/NarrowIntEmulationConverter.h

+//===- NarrowIntEmulationConverter.h - Type Converter for NIE -----*- C++
+//-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_ARITH_NARROW_INT_EMULATION_CONVERTER_H_
+#define MLIR_DIALECT_ARITH_NARROW_INT_EMULATION_CONVERTER_H_
+
+#include "mlir/Transforms/DialectConversion.h"
+
+namespace mlir::arith {
+/// Converts integer types that are too narrow and not supported by the target
+/// hardware. Currently, we only handle power-of-two integer types and convert
+/// them to wider integers that are equal or larger than 8 bits.
+class NarrowIntEmulationConverter : public TypeConverter {
+public:
+  explicit NarrowIntEmulationConverter(unsigned targetWideInt);
+  unsigned targetBitwidth;
+};
+} // namespace mlir::arith
+
+#endif // MLIR_DIALECT_ARITH_NARROW_INT_EMULATION_CONVERTER_H_

mlir/include/mlir/Dialect/Arith/Transforms/Passes.h

 namespace arith {
 
 #define GEN_PASS_DECL
 #include "mlir/Dialect/Arith/Transforms/Passes.h.inc"
 #define GEN_PASS_DECL_ARITHINTRANGEOPTS
 #include "mlir/Dialect/Arith/Transforms/Passes.h.inc"
 
 class WideIntEmulationConverter;
+class NarrowIntEmulationConverter;
 
 /// Create a pass to bufferize Arith ops.
 std::unique_ptr<Pass> createArithBufferizePass();
 
 /// Create a pass to bufferize arith.constant ops.
 std::unique_ptr<Pass> createConstantBufferizePass(uint64_t alignment = 0);
 
 /// Adds patterns to emulate wide Arith and Function ops over integer
 /// types into supported ones. This is done by splitting original power-of-two
 /// i2N integer types into two iN halves.
 void populateArithWideIntEmulationPatterns(
     WideIntEmulationConverter &typeConverter, RewritePatternSet &patterns);
 
+/// Adds patterns to emulate narrow Arith and Function ops
+/// into wide supported integer types.
+void populateArithNarrowIntEmulationPatterns(
+    NarrowIntEmulationConverter &typeConverter, RewritePatternSet &patterns);
+
 /// Add patterns to expand Arith ceil/floor division ops.
 void populateCeilFloorDivExpandOpsPatterns(RewritePatternSet &patterns);
 
 /// Add patterns to expand Arith bf16 patterns to lower level bitcasts/shifts.
 void populateExpandBFloat16Patterns(RewritePatternSet &patterns);
 
 /// Add patterns to expand Arith ops.
 void populateArithExpandOpsPatterns(RewritePatternSet &patterns);

mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h

 class OpBuilder;
 class RewritePatternSet;
 class RewriterBase;
 class Value;
 class ValueRange;
 
 namespace arith {
 class WideIntEmulationConverter;
+class NarrowIntEmulationConverter;
 } // namespace arith
 
 namespace memref {
 class AllocOp;
 class AllocaOp;
 
 //===----------------------------------------------------------------------===//
 // Patterns
void populateMemRefWideIntEmulationPatterns(
     arith::WideIntEmulationConverter &typeConverter,
     RewritePatternSet &patterns);
 
 /// Appends type converions for emulating wide integer memref operations with
 /// ops over narrowe integer types.
 void populateMemRefWideIntEmulationConversions(
     arith::WideIntEmulationConverter &typeConverter);
 
+/// Appends patterns for emulating narrow integer memref operations with ops
+/// over wider integer types.
+void populateMemRefNarrowIntEmulationPatterns(
+    arith::NarrowIntEmulationConverter &typeConverter,
+    RewritePatternSet &patterns);
+
+/// Appends type converions for emulating narrow integer memref operations with
+/// ops over wider integer types.
+void populateMemRefNarrowIntEmulationConversions(
+    arith::NarrowIntEmulationConverter &typeConverter);
+
 /// Transformation to do multi-buffering/array expansion to remove dependencies
 /// on the temporary allocation between consecutive loop iterations.
 /// It returns the new allocation if the original allocation was multi-buffered
 /// and returns failure() otherwise.
 /// When `skipOverrideAnalysis`, the pass will apply the transformation
 /// without checking thwt the buffer is overrided at the beginning of each
 /// iteration. This implies that user knows that there is no data carried across
 /// loop iterations. Example:

mlir/lib/Dialect/Arith/Transforms/CMakeLists.txt

 add_mlir_dialect_library(MLIRArithTransforms
   BufferizableOpInterfaceImpl.cpp
   Bufferize.cpp
   EmulateWideInt.cpp
+  EmulateNarrowInt.cpp
   ExpandOps.cpp
   IntNarrowing.cpp
   IntRangeOptimizations.cpp
   ReifyValueBounds.cpp
   UnsignedWhenEquivalent.cpp
 
   ADDITIONAL_HEADER_DIRS
   {$MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/Arith/Transforms

mlir/lib/Dialect/Arith/Transforms/EmulateNarrowInt.cpp

+//===- EmulateNarrowInt.cpp - Narrow integer operation emulation ----*- C++
+//-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Arith/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Transforms/NarrowIntEmulationConverter.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/Func/Transforms/FuncConversions.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/TypeUtilities.h"
+#include "mlir/Support/LogicalResult.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/APInt.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/MathExtras.h"
+#include <cassert>
+
+using namespace mlir;
+
+//===----------------------------------------------------------------------===//
+// Public Interface Definition
+//===----------------------------------------------------------------------===//
+
+arith::NarrowIntEmulationConverter::NarrowIntEmulationConverter(
+    unsigned targetWideInt)
+    : targetBitwidth(targetWideInt) {
+  assert(llvm::isPowerOf2_32(targetWideInt) &&
+         "Only power-of-two integers are supported");
+
+  // Allow unknown types.
+  addConversion([](Type ty) -> std::optional<Type> { return ty; });
+
+  // Function case.
+  addConversion([this](FunctionType ty) -> std::optional<Type> {
+    SmallVector<Type> inputs;
+    if (failed(convertTypes(ty.getInputs(), inputs)))
+      return std::nullopt;
+
+    SmallVector<Type> results;
+    if (failed(convertTypes(ty.getResults(), results)))
+      return std::nullopt;
+
+    return FunctionType::get(ty.getContext(), inputs, results);
+  });
+}
+
+void arith::populateArithNarrowIntEmulationPatterns(
+    NarrowIntEmulationConverter &typeConverter, RewritePatternSet &patterns) {
+  // Populate `func.*` conversion patterns.
+  populateFunctionOpInterfaceTypeConversionPattern<func::FuncOp>(patterns,
+                                                                 typeConverter);
+  populateCallOpTypeConversionPattern(patterns, typeConverter);
+  populateReturnOpTypeConversionPattern(patterns, typeConverter);
+}

mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt

 add_mlir_dialect_library(MLIRMemRefTransforms
   BufferizableOpInterfaceImpl.cpp
   ComposeSubView.cpp
   ExpandOps.cpp
   ExpandStridedMetadata.cpp
   EmulateWideInt.cpp
+  EmulateNarrowInt.cpp
   ExtractAddressComputations.cpp
   FoldMemRefAliasOps.cpp
   IndependenceTransforms.cpp
   MultiBuffer.cpp
   NormalizeMemRefs.cpp
   ResolveShapedTypeResultDims.cpp
   RuntimeOpVerification.cpp
 

mlir/lib/Dialect/MemRef/Transforms/EmulateNarrowInt.cpp

+//===- EmulateNarrowInt.cpp - Narrow integer operation emulation ----*- C++
+//-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Transforms/NarrowIntEmulationConverter.h"
+#include "mlir/Dialect/Arith/Transforms/Passes.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/MemRef/Transforms/Passes.h"
+#include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/MathExtras.h"
+#include <cassert>
+
+using namespace mlir;
+
+//===----------------------------------------------------------------------===//
+// Utility functions
+//===----------------------------------------------------------------------===//
+
+/// Returns the offset of the value in `targetBits` representation.
+///
+/// `srcIdx` is an index into a 1-D array with each element having `sourceBits`.
+/// It's assumed to be non-negative.
+///
+/// When accessing an element in the array treating as having elements of
+/// `targetBits`, multiple values are loaded in the same time. The method
+/// returns the offset where the `srcIdx` locates in the value. For example, if
+/// `sourceBits` equals to 4 and `targetBits` equals to 8, the x-th element is
+/// located at (x % 2) * 4. Because there are two elements in one i8, and one
+/// element has 4 bits.
+static Value getOffsetForBitwidth(Location loc, Value srcIdx, int sourceBits,
+                                  int targetBits, OpBuilder &builder) {
+  assert(targetBits % sourceBits == 0);
+  IntegerType targetType = builder.getIntegerType(targetBits);
+  IntegerAttr idxAttr =
+      builder.getIntegerAttr(targetType, targetBits / sourceBits);
+  auto idx = builder.create<arith::ConstantOp>(loc, targetType, idxAttr);
+  IntegerAttr srcBitsAttr = builder.getIntegerAttr(targetType, sourceBits);
+  auto srcBitsValue =
+      builder.create<arith::ConstantOp>(loc, targetType, srcBitsAttr);
+  auto m = builder.create<arith::RemUIOp>(loc, srcIdx, idx);
+  return builder.create<arith::MulIOp>(loc, targetType, m, srcBitsValue);
+}
+
+namespace {
+
+//===----------------------------------------------------------------------===//
+// ConvertMemRefAlloc
+//===----------------------------------------------------------------------===//
+
+struct ConvertMemRefAlloc final : OpConversionPattern<memref::AllocOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(memref::AllocOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type newTy = getTypeConverter()->convertType(op.getType());
+    if (!newTy)
+      return rewriter.notifyMatchFailure(
+          op->getLoc(),
+          llvm::formatv("failed to convert memref type: {0}", op.getType()));
+
+    rewriter.replaceOpWithNewOp<memref::AllocOp>(
+        op, newTy, adaptor.getDynamicSizes(), adaptor.getSymbolOperands(),
+        adaptor.getAlignmentAttr());
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// ConvertMemRefLoad
+//===----------------------------------------------------------------------===//
+
+struct ConvertMemRefLoad final : OpConversionPattern<memref::LoadOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(memref::LoadOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type newTy = getTypeConverter()->convertType(op.getType());
+    if (!newTy)
+      return rewriter.notifyMatchFailure(
+          op->getLoc(), llvm::formatv("failed to convert memref type: {0}",
+                                      op.getMemRefType()));
+    if (op.getMemRefType() == newTy)
+      return failure();
+
+    auto loc = op.getLoc();
+    Value source = adaptor.getMemref();
+    auto sourceType = cast<MemRefType>(source.getType());
+    auto srcElementType = sourceType.getElementType();
+    unsigned sourceRank = sourceType.getRank();
+
+    auto oldElementType =
+        cast<MemRefType>(op.getMemref().getType()).getElementType();
+    int srcBits = oldElementType.getIntOrFloatBitWidth();
+    int dstBits = srcElementType.getIntOrFloatBitWidth();
+    assert(dstBits % srcBits == 0);
+
+    // The emulation only works on 1D memref types. To make this work on N-D
+    // memref, we need to linearize the offset.
+    // Specifically, %0 = memref.load %0[%v0][%v1] :
+    // memref<?x?xi4, strided<[?, ?], offset = ?>> can be replaced with
+    // %b, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %0
+    // %linearized_offset = %v0 * %stride#0 + %scaled_v1 * %stride#1
+    // where %scaled_v1 = v1 / targetBits * sourceBits
+    // %linearized_size = %size0 * %size1
+    // %linearized = memref.reinterpret_cast %b, offset = [%offset], sizes =
+    // [%linearized_size], strides = [%stride#1] %load = memref.load
+    // %linearized[%linearized_offset] : memref<?xi4, strided<?, offset = ?>>
+    auto stridedMetadata =
+        rewriter.create<memref::ExtractStridedMetadataOp>(loc, source);
+    auto baseBuffer = stridedMetadata.getBaseBuffer();
+    auto baseSizes = stridedMetadata.getSizes();
+    auto baseStrides = stridedMetadata.getStrides();
+    auto baseOffset = stridedMetadata.getOffset();
+
+    SmallVector<Value> indices = adaptor.getIndices();
+    assert(indices.size() == baseStrides.size());
+    assert(indices.size() == sourceRank);
+
+    // Only the last index is modified to load the bits needed.
+    IndexType targetType = rewriter.getIndexType();
+    IntegerAttr attr = rewriter.getIndexAttr(dstBits / srcBits);
+    auto scaler = rewriter.create<arith::ConstantOp>(loc, targetType, attr);
+    indices.back() =
+        rewriter.create<arith::DivUIOp>(loc, indices.back(), scaler);
+
+    SmallVector<Value> adjustOffsets;
+    for (unsigned i = 0; i < sourceRank; ++i) {
+      adjustOffsets.push_back(
+          rewriter.create<arith::MulIOp>(loc, indices[i], baseStrides[i]));
+    }
+
+    // Linearize offset and sizes.
+    Value linearizedOffset = adjustOffsets[0];
+    Value linearizedSize = baseSizes[0];
+    for (unsigned i = 1; i < sourceRank; ++i) {
+      linearizedOffset = rewriter.create<arith::AddIOp>(loc, linearizedOffset,
+                                                        adjustOffsets[i]);
+      linearizedSize =
+          rewriter.create<arith::MulIOp>(loc, linearizedSize, baseSizes[i]);
+    }
+
+    // Flatten n-D MemRef to 1-D MemRef.
+    auto layoutAttr = StridedLayoutAttr::get(
+        sourceType.getContext(), ShapedType::kDynamic, {ShapedType::kDynamic});
+    int64_t staticShape = sourceType.hasStaticShape()
+                              ? sourceType.getNumElements()
+                              : ShapedType::kDynamic;
+    auto flattenMemrefType = MemRefType::get(
+        staticShape, srcElementType, layoutAttr, sourceType.getMemorySpace());
+
+    auto reinterpret = rewriter.create<memref::ReinterpretCastOp>(
+        loc, flattenMemrefType, baseBuffer, baseOffset, linearizedSize,
+        baseStrides.back());
+
+    auto newLoad = rewriter.create<memref::LoadOp>(
+        loc, srcElementType, reinterpret.getResult(), linearizedOffset,
+        op.getNontemporal());
+
+    // Get the offset and shift the bits to the rightmost.
+    auto lastIdx = rewriter.create<arith::IndexCastUIOp>(
+        loc, srcElementType, adaptor.getIndices().back());
+    Value BitwidthOffset =
+        getOffsetForBitwidth(loc, lastIdx, srcBits, dstBits, rewriter);
+    auto bitsLoad =
+        rewriter.create<arith::ShRSIOp>(loc, newLoad, BitwidthOffset);
+
+    // Get the low bits by truncating the result.
+    auto result =
+        rewriter.create<arith::TruncIOp>(loc, oldElementType, bitsLoad);
+    rewriter.replaceOp(op, result.getResult());
+
+    return success();
+  }
+};
+} // end anonymous namespace
+
+//===----------------------------------------------------------------------===//
+// Public Interface Definition
+//===----------------------------------------------------------------------===//
+
+void memref::populateMemRefNarrowIntEmulationPatterns(
+    arith::NarrowIntEmulationConverter &typeConverter,
+    RewritePatternSet &patterns) {
+
+  // Populate `memref.*` conversion patterns.
+  patterns.add<ConvertMemRefAlloc, ConvertMemRefLoad>(typeConverter,
+                                                      patterns.getContext());
+}
+
+void memref::populateMemRefNarrowIntEmulationConversions(
+    arith::NarrowIntEmulationConverter &typeConverter) {
+  typeConverter.addConversion(
+      [&typeConverter](MemRefType ty) -> std::optional<Type> {
+        auto intTy = dyn_cast<IntegerType>(ty.getElementType());
+        if (!intTy)
+          return ty;
+
+        unsigned width = intTy.getWidth();
+        if (width >= typeConverter.targetBitwidth) {
+          return ty;
+        } else {
+          Type newElemTy =
+              IntegerType::get(ty.getContext(), typeConverter.targetBitwidth,
+                               intTy.getSignedness());
+          if (!newElemTy)
+            return std::nullopt;
+          return ty.cloneWith(std::nullopt, newElemTy);
+        }
+      });
+}

mlir/test/Dialect/Arith/emulate-narrow-int.mlir

+// RUN: mlir-opt --test-emulate-narrow-int="arith-int-bitwidth=8" %s | FileCheck %s
+
+// Expect no conversions, f32 is not an integer type.
+// CHECK-LABEL: func @identity_f32
+// CHECK-SAME:    ([[ARG:%.+]]: f32) -> f32
+// CHECK-NEXT:    return [[ARG]] : f32
+func.func @identity_f32(%a : f32) -> f32 {
+    return %a : f32
+}
+
+// Expect no conversions, i32 is supported.
+// CHECK-LABEL: func @identity_i32
+// CHECK-SAME:    ([[ARG:%.+]]: vector<2xi32>) -> vector<2xi32>
+// CHECK-NEXT:    return [[ARG]] : vector<2xi32>
+func.func @identity_i32(%a : vector<2xi32>) -> vector<2xi32> {
+    return %a : vector<2xi32>
+}
+
+// CHECK-LABEL: func @identity_scalar
+// CHECK-SAME:     ([[ARG:%.+]]: i8) -> i8
+// CHECK-NEXT:     return [[ARG]] : i8
+func.func @identity_scalar(%x : i4) -> i4 {
+    return %x : i4
+}
+
+// CHECK-LABEL: func @identity_vector
+// CHECK-SAME:     ([[ARG:%.+]]: vector<4xi8>) -> vector<4xi8>
+// CHECK-NEXT:     return [[ARG]] : vector<4xi8>
+func.func @identity_vector(%x : vector<4xi4>) -> vector<4xi4> {
+    return %x : vector<4xi4>
+}
+
+// CHECK-LABEL: func @identity_vector2d
+// CHECK-SAME:     ([[ARG:%.+]]: vector<3x4xi8>) -> vector<3x4xi8>
+// CHECK-NEXT:     return [[ARG]] : vector<3x4xi8>
+func.func @identity_vector2d(%x : vector<3x4xi4>) -> vector<3x4xi4> {
+    return %x : vector<3x4xi4>
+}
+
+// CHECK-LABEL: func @call
+// CHECK-SAME:     ([[ARG:%.+]]: vector<4xi8>) -> vector<4xi8>
+// CHECK-NEXT:     [[RES:%.+]] = call @identity_vector([[ARG]]) : (vector<4xi8>) -> vector<4xi8>
+// CHECK-NEXT:     return [[RES]] : vector<4xi8>
+func.func @call(%a : vector<4xi4>) -> vector<4xi4> {
+    %res = func.call @identity_vector(%a) : (vector<4xi4>) -> vector<4xi4>
+    return %res : vector<4xi4>
+}

mlir/test/Dialect/MemRef/emulate-narrow-int.mlir

+// RUN: mlir-opt --test-emulate-narrow-int="arith-int-bitwidth=4 memref-target-bits=8" %s | FileCheck %s
+
+// Expect no conversions, i32 is supported.
+// CHECK-LABEL: func @memref_i32
+// CHECK:         [[M:%.+]] = memref.alloc() : memref<4xi32, 1>
+// CHECK-NEXT:    [[V:%.+]] = memref.load [[M]][{{%.+}}] : memref<4xi32, 1>
+// CHECK-NEXT:    memref.store {{%.+}}, [[M]][{{%.+}}] : memref<4xi32, 1>
+// CHECK-NEXT:    return
+func.func @memref_i32() {
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : i32
+    %m = memref.alloc() : memref<4xi32, 1>
+    %v = memref.load %m[%c0] : memref<4xi32, 1>
+    memref.store %c1, %m[%c0] : memref<4xi32, 1>
+    return
+}
+
+// Expect no conversions, f32 is not an integer type.
+// CHECK-LABEL: func @memref_f32
+// CHECK:         [[M:%.+]] = memref.alloc() : memref<4xf32, 1>
+// CHECK-NEXT:    [[V:%.+]] = memref.load [[M]][{{%.+}}] : memref<4xf32, 1>
+// CHECK-NEXT:    memref.store {{%.+}}, [[M]][{{%.+}}] : memref<4xf32, 1>
+// CHECK-NEXT:    return
+func.func @memref_f32() {
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1.0 : f32
+    %m = memref.alloc() : memref<4xf32, 1>
+    %v = memref.load %m[%c0] : memref<4xf32, 1>
+    memref.store %c1, %m[%c0] : memref<4xf32, 1>
+    return
+}
+
+// CHECK-LABEL: func @memref_load_i4
+// CHECK-SAME:    (%[[ARG:.*]]: index)
+// CHECK-NEXT:    %[[M:.*]]  = memref.alloc() : memref<4xi8>
+// CHECK-NEXT:    %[[BASE:.*]], %[[OFFSET:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[M]] : memref<4xi8> -> memref<i8>, index, index, index
+// CHECK-NEXT:    %[[CI:.*]] = arith.constant 2 : index
+// CHECK-NEXT:    %[[SCALE:.*]] = arith.divui %[[ARG]], %[[CI]] : index
+// CHECK-NEXT:    %[[INDEX:.*]] = arith.muli %[[SCALE]], %[[STRIDES]] : index
+// CHECK-NEXT:    %[[CAST:.*]] = memref.reinterpret_cast %[[BASE]] to offset: [%[[OFFSET]]], sizes: [%[[SIZES]]], strides: [%[[STRIDES]]] : memref<i8> to memref<4xi8, strided<[?], offset: ?>>
+// CHECK-NEXT:    %[[LOAD:.*]] = memref.load %[[CAST]][%[[INDEX]]] : memref<4xi8, strided<[?], offset: ?>>
+// CHECK-NEXT:    %[[I:.*]] = arith.index_castui %[[ARG]] : index to i8
+// CHECK-NEXT:    %[[C2:.*]] = arith.constant 2 : i8
+// CHECK-NEXT:    %[[C4:.*]] = arith.constant 4 : i8
+// CHECK-NEXT:    %[[REM:.*]] = arith.remui %[[I]], %[[C2]] : i8
+// CHECK-NEXT:    %[[STEP:.*]] = arith.muli %[[REM]], %[[C4]] : i8
+// CHECK-NEXT:    %[[SHIFT:.*]] = arith.shrsi %[[LOAD]], %[[STEP]] : i8
+// CHECK-NEXT:    %[[RES:.*]] = arith.trunci %[[SHIFT]] : i8 to i4
+// CHECK-NEXT:    return
+func.func @memref_load_i4(%arg0: index) {
+    %0 = memref.alloc() : memref<4xi4>
+    %1 = memref.load %0[%arg0] : memref<4xi4>
+    return
+}
+
+
+// CHECK-LABEL: func @memref_load_i4_rank2
+// CHECK-SAME:    (%[[ARG0:.*]]: index, %[[ARG1:.*]]: index)
+// CHECK-NEXT:    %[[M:.*]]  = memref.alloc() : memref<4x4xi8>
+// CHECK-NEXT:    %[[BASE:.*]], %[[OFFSET:.*]], %[[SIZES:.*]]:2, %[[STRIDES:.*]]:2 = memref.extract_strided_metadata %[[M]] : memref<4x4xi8> -> memref<i8>, index, index, index, index, index
+// CHECK-NEXT:    %[[CI:.*]] = arith.constant 2 : index
+// CHECK-NEXT:    %[[SCALE:.*]] = arith.divui %[[ARG1]], %[[CI]] : index
+// CHECK-NEXT:    %[[IDX1:.*]] = arith.muli %[[ARG0]], %[[STRIDES]]#0 : index
+// CHECK-NEXT:    %[[IDX2:.*]] = arith.muli %[[SCALE]], %[[STRIDES]]#1 : index
+// CHECK-NEXT:    %[[INDEX:.*]] = arith.addi %[[IDX1]], %[[IDX2]] : index
+// CHECK-NEXT:    %[[LSIZE:.*]] = arith.muli %[[SIZES]]#0, %[[SIZES]]#1 : index
+// CHECK-NEXT:    %[[CAST:.*]] = memref.reinterpret_cast %[[BASE]] to offset: [%[[OFFSET]]], sizes: [%[[LSIZE]]], strides: [%[[STRIDES]]#1] : memref<i8> to memref<16xi8, strided<[?], offset: ?>>
+// CHECK-NEXT:    %[[LOAD:.*]] = memref.load %[[CAST]][%[[INDEX]]] : memref<16xi8, strided<[?], offset: ?>>
+// CHECK-NEXT:    %[[I:.*]] = arith.index_castui %[[ARG1]] : index to i8
+// CHECK-NEXT:    %[[C2:.*]] = arith.constant 2 : i8
+// CHECK-NEXT:    %[[C4:.*]] = arith.constant 4 : i8
+// CHECK-NEXT:    %[[REM:.*]] = arith.remui %[[I]], %[[C2]] : i8
+// CHECK-NEXT:    %[[STEP:.*]] = arith.muli %[[REM]], %[[C4]] : i8
+// CHECK-NEXT:    %[[SHIFT:.*]] = arith.shrsi %[[LOAD]], %[[STEP]] : i8
+// CHECK-NEXT:    %[[RES:.*]] = arith.trunci %[[SHIFT]] : i8 to i4
+// CHECK-NEXT:    return
+func.func @memref_load_i4_rank2(%arg0: index, %arg1: index) {
+    %0 = memref.alloc() : memref<4x4xi4>
+    %1 = memref.load %0[%arg0,%arg1] : memref<4x4xi4>
+    return
+}

mlir/test/lib/Dialect/MemRef/CMakeLists.txt

 # Exclude tests from libMLIR.so
 add_mlir_library(MLIRMemRefTestPasses
   TestComposeSubView.cpp
+  TestEmulateNarrowInt.cpp
   TestMultiBuffer.cpp
 
   EXCLUDE_FROM_LIBMLIR
 
   LINK_LIBS PUBLIC
   MLIRPass
   MLIRMemRefDialect
   MLIRMemRefTransforms
   MLIRTestDialect
   )
 
 target_include_directories(MLIRMemRefTestPasses
   PRIVATE
   ${CMAKE_CURRENT_SOURCE_DIR}/../Test
   ${CMAKE_CURRENT_BINARY_DIR}/../Test
   )

mlir/test/lib/Dialect/MemRef/TestEmulateNarrowInt.cpp

+//===- TestEmulateNarrowInt.cpp - Test Narrow Int Emulation  ------*- c++
+//-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Transforms/NarrowIntEmulationConverter.h"
+#include "mlir/Dialect/Arith/Transforms/Passes.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/DialectConversion.h"
+
+using namespace mlir;
+
+namespace {
+
+struct TestEmulateNarrowIntPass
+    : public PassWrapper<TestEmulateNarrowIntPass,
+                         OperationPass<func::FuncOp>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TestEmulateNarrowIntPass)
+
+  TestEmulateNarrowIntPass() = default;
+  TestEmulateNarrowIntPass(const TestEmulateNarrowIntPass &pass)
+      : PassWrapper(pass) {}
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<arith::ArithDialect, func::FuncDialect,
+                    memref::MemRefDialect, vector::VectorDialect>();
+  }
+  StringRef getArgument() const final { return "test-emulate-narrow-int"; }
+  StringRef getDescription() const final {
+    return "Function pass to test Narrow Integer Emulation";
+  }
+
+  void runOnOperation() override {
+    if (!llvm::isPowerOf2_32(targetWideInt) || targetWideInt < 8) {
+      signalPassFailure();
+      return;
+    }
+
+    Operation *op = getOperation();
+    MLIRContext *ctx = op->getContext();
+
+    arith::NarrowIntEmulationConverter typeConverter(targetWideInt);
+
+    // Convert scalar type.
+    typeConverter.addConversion([this](IntegerType ty) -> std::optional<Type> {
+      unsigned width = ty.getWidth();
+      if (width >= arithBitwidth)
+        return ty;
+      else
+        return IntegerType::get(ty.getContext(), arithBitwidth);
+
+      return std::nullopt;
+    });
+
+    // Convert vector type.
+    typeConverter.addConversion([this](VectorType ty) -> std::optional<Type> {
+      auto intTy = dyn_cast<IntegerType>(ty.getElementType());
+      if (!intTy)
+        return ty;
+
+      unsigned width = intTy.getWidth();
+      if (width >= arithBitwidth)
+        return ty;
+      else
+        return VectorType::get(
+            to_vector(ty.getShape()),
+            IntegerType::get(ty.getContext(), arithBitwidth));
+
+      return std::nullopt;
+    });
+
+    memref::populateMemRefNarrowIntEmulationConversions(typeConverter);
+    ConversionTarget target(*ctx);
+    target.addDynamicallyLegalOp<func::FuncOp>([&typeConverter](Operation *op) {
+      return typeConverter.isLegal(cast<func::FuncOp>(op).getFunctionType());
+    });
+    auto opLegalCallback = [&typeConverter](Operation *op) {
+      return typeConverter.isLegal(op);
+    };
+    target.addDynamicallyLegalOp<func::CallOp, func::ReturnOp>(opLegalCallback);
+    target.addDynamicallyLegalDialect<
+        arith::ArithDialect, vector::VectorDialect, memref::MemRefDialect>(
+        [&typeConverter](Operation *op) { return typeConverter.isLegal(op); });
+
+    RewritePatternSet patterns(ctx);
+
+    arith::populateArithNarrowIntEmulationPatterns(typeConverter, patterns);
+    memref::populateMemRefNarrowIntEmulationPatterns(typeConverter, patterns);
+
+    if (failed(applyPartialConversion(op, target, std::move(patterns))))
+      signalPassFailure();
+  }
+
+  Option<unsigned> targetWideInt{
+      *this, "memref-target-bits",
+      llvm::cl::desc("Target memref integer bit width"), llvm::cl::init(8)};
+
+  Option<unsigned> arithBitwidth{
+      *this, "arith-int-bitwidth",
+      llvm::cl::desc("Target arith integer bit width"), llvm::cl::init(4)};
+};
+} // namespace
+
+namespace mlir::test {
+void registerTestEmulateNarrowIntPass() {
+  PassRegistration<TestEmulateNarrowIntPass>();
+}
+} // namespace mlir::test

mlir/tools/mlir-opt/mlir-opt.cpp

 void registerTestDataLayoutPropagation();
 void registerTestDataLayoutQuery();
 void registerTestDeadCodeAnalysisPass();
 void registerTestDecomposeCallGraphTypes();
 void registerTestDiagnosticsPass();
 void registerTestDialectConversionPasses();
 void registerTestDominancePass();
 void registerTestDynamicPipelinePass();
+void registerTestEmulateNarrowIntPass();
 void registerTestExpandMathPass();
 void registerTestFooAnalysisPass();
 void registerTestComposeSubView();
 void registerTestMultiBuffering();
 void registerTestIntRangeInference();
 void registerTestIRVisitorsPass();
 void registerTestGenericIRVisitorsPass();
 void registerTestGenericIRVisitorsInterruptPass();
#if MLIR_ROCM_CONVERSIONS_ENABLED
   mlir::test::registerTestGpuSerializeToHsacoPass();
 #endif
   mlir::test::registerTestDecomposeCallGraphTypes();
   mlir::test::registerTestDataLayoutPropagation();
   mlir::test::registerTestDataLayoutQuery();
   mlir::test::registerTestDeadCodeAnalysisPass();
   mlir::test::registerTestDominancePass();
   mlir::test::registerTestDynamicPipelinePass();
+  mlir::test::registerTestEmulateNarrowIntPass();
   mlir::test::registerTestExpandMathPass();
   mlir::test::registerTestFooAnalysisPass();
   mlir::test::registerTestComposeSubView();
   mlir::test::registerTestMultiBuffering();
   mlir::test::registerTestIntRangeInference();
   mlir::test::registerTestIRVisitorsPass();
   mlir::test::registerTestGenericIRVisitorsPass();
   mlir::test::registerTestInterfaces();