[WebAssembly] Bulk memory intrinsics and builtins

[tlively]

Summary

implements llvm intrinsics and clang intrinsics for
memory.init and data.drop.

[aheejin]

clang/include/clang/Basic/BuiltinsWebAssembly.def

29

• When do we use Ii instead of i?
• Shouldn't we add the memory index field as well, even if that means a user always has to set it to 0? Are we gonna add a new builtin once multiple memories are available?
• Shouldn't the segment index, segment offset, and the size operands be Ui because they cannot be negative?

30

The same thing..

• Shouldn't the segment index be Ui?
• Shouldn't we add the memory index field as well?

clang/lib/CodeGen/CGBuiltin.cpp

13483

Not sure if we can use llvm_unreachable here, because we can certainly reach here if a user uses this builtin with a non-const variable. In this file people often just used assert for user errors, which is not ideal either.

I haven't used it myself, but looking at the code, the recommended way to signal an error looks like to use [[ https://github.com/llvm/llvm-project/blob/db7fbcb038f095622a3e6847ecd6ff80bdc2483a/clang/lib/CodeGen/CodeGenFunction.h#L2092-L2094 | CodeGenFunction::ErrorUnsupported ]] function, as in here. We used llvm_unreachable for SIMD builtins too, but maybe we can fix it later.

13494

Do we need to pass types here to make intrinsic names overloaded like llvm.wasm.memory.init.i32.i32.i8, unless this intrinsic also support operands of other types, which it doesn't? The same for data.drop builtin.

llvm/include/llvm/IR/IntrinsicsWebAssembly.td

119

• Why are the first two immediates anyint? The spec says the segment index is varuint32 (so that will be llvm_i32_ty here), and for the memory index I don't think we are ever gonna need something larger than i32 either, but maybe it is better to be clarified in the spec too though.
• For the pointer type, I guess llvm_ptr_ty will be sufficient, unless we have multiple overloaded intrinsics of the same name.

121

• Isn't the pointer argument number not 1 but 2?
• I guess this should have IntrHasSideEffects as well, as in data.drop?
• I don't know much how they are handled differently in compilation, but because we can access some other memory, which holds the segment part, during execution, how about IntrInaccessibleMemOrArgMemOnly instead of IntrArgMemOnly?

122

I told you we needed this, but on second thought, because we don't really use MachineMemOperand, maybe we don't need it..? :$ The standard memcpy/memmove/memset intrinsics don't have it either. So if the code runs ok without these I think we can delete it. The same for data.drop intrinsic. Sorry for the incorrect info and confusion.

125

The same, llvm_i32_ty

[tlively]

• Address comments
  • Use Ui in builtin signatures
  • Remove unnecessary intrinsic polymorphism
  • Tweak intrinsic properties

clang/include/clang/Basic/BuiltinsWebAssembly.def

29

We use Ii instead of i when the argument needs to be a compile time constant because it is encoded as a literal in the corresponding instruction.

I was imagining that we could add a new builtin once multiple memories are available, but perhaps it would be better to add an argument to the builtin now and error out if it is not zero. I will do that.

Yes, Ui seems reasonable.

30

• Done
• This intrinsic doesn't have a memory index, since it doesn't write anything into memory.

clang/lib/CodeGen/CGBuiltin.cpp

13483

llvm_unreachable is appropriate here because non-constant expressions will have been caught earlier by the type checking.

A follow-up PR updating SIMD intrinsics to use Ui and appropriate error handling sounds good.

13494

Fixed. I had been copying llvm.memcpy, but I agree it is better to not be polymorphic.

llvm/include/llvm/IR/IntrinsicsWebAssembly.td

121

• Yes, good catch.

• Ok, I'm not sure it's necessary but it can't hurt.

• I'm not sure this is necessary either, but again it can't hurt. Better safe than sorry.

122

No problem!

[tlively]

• Update tests to check full polymorphism of llvm.memcpy and llvm.memmove

[aheejin]

clang/lib/CodeGen/CGBuiltin.cpp

13483

llvm_unreachable is appropriate here because non-constant expressions will have been caught earlier by the type checking.

Oh, does Ii ensures that? That's cool...

clang/include/clang/Basic/BuiltinsWebAssembly.def

 #endif
 
 // Query the current memory size, and increase the current memory size.
 // Note that memory.size is not "c" (readnone) because it must be sequenced
 // with respect to memory.grow calls.
 BUILTIN(__builtin_wasm_memory_size, "zIi", "n")
 BUILTIN(__builtin_wasm_memory_grow, "zIiz", "n")
 
+// Bulk memory builtins
+TARGET_BUILTIN(__builtin_wasm_memory_init, "vIUiIUiv*UiUi", "", "bulk-memory")
+TARGET_BUILTIN(__builtin_wasm_data_drop, "vIUi", "", "bulk-memory")
+
 // Floating point min/max
 BUILTIN(__builtin_wasm_min_f32, "fff", "nc")
 BUILTIN(__builtin_wasm_max_f32, "fff", "nc")
 BUILTIN(__builtin_wasm_min_f64, "ddd", "nc")
 BUILTIN(__builtin_wasm_max_f64, "ddd", "nc")
 
 // Exception handling builtins.
 TARGET_BUILTIN(__builtin_wasm_throw, "vUiv*", "r", "exception-handling")

clang/lib/CodeGen/CGBuiltin.cpp

case WebAssembly::BI__builtin_wasm_memory_grow: {
     llvm::Type *ResultType = ConvertType(E->getType());
     Value *Args[] = {
       EmitScalarExpr(E->getArg(0)),
       EmitScalarExpr(E->getArg(1))
     };
     Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_memory_grow, ResultType);
     return Builder.CreateCall(Callee, Args);
   }
+  case WebAssembly::BI__builtin_wasm_memory_init: {
+    llvm::APSInt SegConst;
+    if (!E->getArg(0)->isIntegerConstantExpr(SegConst, getContext()))
+      llvm_unreachable("Constant arg isn't actually constant?");
+    llvm::APSInt MemConst;
+    if (!E->getArg(1)->isIntegerConstantExpr(MemConst, getContext()))
+      llvm_unreachable("Constant arg isn't actually constant?");
+    if (!MemConst.isNullValue())
+      ErrorUnsupported(E, "non-zero memory index");
+    Value *Args[] = {llvm::ConstantInt::get(getLLVMContext(), SegConst),
+                     llvm::ConstantInt::get(getLLVMContext(), MemConst),
+                     EmitScalarExpr(E->getArg(2)), EmitScalarExpr(E->getArg(3)),
+                     EmitScalarExpr(E->getArg(4))};
+    Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_memory_init);
+    return Builder.CreateCall(Callee, Args);
+  }
+  case WebAssembly::BI__builtin_wasm_data_drop: {
+    llvm::APSInt SegConst;
+    if (!E->getArg(0)->isIntegerConstantExpr(SegConst, getContext()))
+      llvm_unreachable("Constant arg isn't actually constant?");
+    Value *Arg = llvm::ConstantInt::get(getLLVMContext(), SegConst);
+    Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_data_drop);
+    return Builder.CreateCall(Callee, {Arg});
+  }
   case WebAssembly::BI__builtin_wasm_throw: {
     Value *Tag = EmitScalarExpr(E->getArg(0));
     Value *Obj = EmitScalarExpr(E->getArg(1));
     Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_throw);
     return Builder.CreateCall(Callee, {Tag, Obj});
   }
   case WebAssembly::BI__builtin_wasm_rethrow: {
     Function *Callee = CGM.getIntrinsic(Intrinsic::wasm_rethrow);

clang/test/CodeGen/builtins-wasm.c

-// RUN: %clang_cc1 -triple wasm32-unknown-unknown -target-feature +unimplemented-simd128 -target-feature +nontrapping-fptoint -target-feature +exception-handling -fno-lax-vector-conversions -O3 -emit-llvm -o - %s | FileCheck %s -check-prefixes WEBASSEMBLY,WEBASSEMBLY32
-// RUN: %clang_cc1 -triple wasm64-unknown-unknown -target-feature +unimplemented-simd128 -target-feature +nontrapping-fptoint -target-feature +exception-handling -fno-lax-vector-conversions -O3 -emit-llvm -o - %s | FileCheck %s -check-prefixes WEBASSEMBLY,WEBASSEMBLY64
-// RUN: not %clang_cc1 -triple wasm64-unknown-unknown -target-feature +nontrapping-fptoint -target-feature +exception-handling -fno-lax-vector-conversions -O3 -emit-llvm -o - %s 2>&1 | FileCheck %s -check-prefixes MISSING-SIMD
+// RUN: %clang_cc1 -triple wasm32-unknown-unknown -target-feature +unimplemented-simd128 -target-feature +nontrapping-fptoint -target-feature +exception-handling -target-feature +bulk-memory -fno-lax-vector-conversions -O3 -emit-llvm -o - %s | FileCheck %s -check-prefixes WEBASSEMBLY,WEBASSEMBLY32
+// RUN: %clang_cc1 -triple wasm64-unknown-unknown -target-feature +unimplemented-simd128 -target-feature +nontrapping-fptoint -target-feature +exception-handling -target-feature +bulk-memory -fno-lax-vector-conversions -O3 -emit-llvm -o - %s | FileCheck %s -check-prefixes WEBASSEMBLY,WEBASSEMBLY64
+// RUN: not %clang_cc1 -triple wasm64-unknown-unknown -target-feature +nontrapping-fptoint -target-feature +exception-handling -target-feature +bulk-memory -fno-lax-vector-conversions -O3 -emit-llvm -o - %s 2>&1 | FileCheck %s -check-prefixes MISSING-SIMD
 
 // SIMD convenience types
 typedef char i8x16 __attribute((vector_size(16)));
 typedef short i16x8 __attribute((vector_size(16)));
 typedef int i32x4 __attribute((vector_size(16)));
 typedef long long i64x2 __attribute((vector_size(16)));
 typedef unsigned char u8x16 __attribute((vector_size(16)));
 typedef unsigned short u16x8 __attribute((vector_size(16)));

 }
 
 __SIZE_TYPE__ memory_grow(__SIZE_TYPE__ delta) {
   return __builtin_wasm_memory_grow(0, delta);
   // WEBASSEMBLY32: call i32 @llvm.wasm.memory.grow.i32(i32 0, i32 %{{.*}})
   // WEBASSEMBLY64: call i64 @llvm.wasm.memory.grow.i64(i32 0, i64 %{{.*}})
 }
 
+void memory_init(void *dest, int offset, int size) {
+  __builtin_wasm_memory_init(3, 0, dest, offset, size);
+  // WEBASSEMBLY32: call void @llvm.wasm.memory.init(i32 3, i32 0, i8* %{{.*}}, i32 %{{.*}}, i32 %{{.*}})
+  // WEBASSEMBLY64: call void @llvm.wasm.memory.init(i32 3, i32 0, i8* %{{.*}}, i32 %{{.*}}, i32 %{{.*}})
+}
+
+void data_drop() {
+  __builtin_wasm_data_drop(3);
+  // WEBASSEMBLY32: call void @llvm.wasm.data.drop(i32 3)
+  // WEBASSEMBLY64: call void @llvm.wasm.data.drop(i32 3)
+}
+
 void throw(unsigned int tag, void *obj) {
   return __builtin_wasm_throw(tag, obj);
   // WEBASSEMBLY32: call void @llvm.wasm.throw(i32 %{{.*}}, i8* %{{.*}})
   // WEBASSEMBLY64: call void @llvm.wasm.throw(i32 %{{.*}}, i8* %{{.*}})
 }
 
 void rethrow(void) {
   return __builtin_wasm_rethrow();

llvm/include/llvm/IR/IntrinsicsWebAssembly.td

def int_wasm_anytrue :
   Intrinsic<[llvm_i32_ty],
             [llvm_anyvector_ty],
             [IntrNoMem, IntrSpeculatable]>;
 def int_wasm_alltrue :
   Intrinsic<[llvm_i32_ty],
             [llvm_anyvector_ty],
             [IntrNoMem, IntrSpeculatable]>;
 
+//===----------------------------------------------------------------------===//
+// Bulk memory intrinsics
+//===----------------------------------------------------------------------===//
+
+def int_wasm_memory_init :
+  Intrinsic<[],
+            [llvm_i32_ty, llvm_i32_ty, llvm_ptr_ty, llvm_i32_ty, llvm_i32_ty],
+            [IntrWriteMem, IntrInaccessibleMemOrArgMemOnly, WriteOnly<2>,
+             IntrHasSideEffects]>;
+def int_wasm_data_drop :
+  Intrinsic<[],
+            [llvm_i32_ty],
+            [IntrNoDuplicate, IntrHasSideEffects]>;
+
 } // TargetPrefix = "wasm"

llvm/lib/Target/WebAssembly/WebAssemblyInstrBulkMemory.td

multiclass BULK_I<dag oops_r, dag iops_r, dag oops_s, dag iops_s,
                   list<dag> pattern_r, string asmstr_r = "",
                   string asmstr_s = "", bits<32> simdop = -1> {
   defm "" : I<oops_r, iops_r, oops_s, iops_s, pattern_r, asmstr_r, asmstr_s,
               !or(0xfc00, !and(0xff, simdop))>,
             Requires<[HasBulkMemory]>;
 }
 
 // Bespoke types and nodes for bulk memory ops
-def wasm_memcpy_t : SDTypeProfile<0, 3,
-  [SDTCisPtrTy<0>, SDTCisPtrTy<1>, SDTCisInt<2>]
+def wasm_memcpy_t : SDTypeProfile<0, 5,
+  [SDTCisInt<0>, SDTCisInt<1>, SDTCisPtrTy<2>, SDTCisPtrTy<3>, SDTCisInt<4>]
 >;
 def wasm_memcpy : SDNode<"WebAssemblyISD::MEMORY_COPY", wasm_memcpy_t,
                          [SDNPHasChain, SDNPMayLoad, SDNPMayStore]>;
 
 //===----------------------------------------------------------------------===//
+// memory.init
+//===----------------------------------------------------------------------===//
+
+let mayStore = 1 in
+defm MEMORY_INIT :
+  BULK_I<(outs),
+         (ins i32imm_op:$seg, i32imm_op:$idx, I32:$dest,
+              I32:$offset, I32:$size),
+         (outs), (ins i32imm_op:$seg, i32imm_op:$idx),
+         [(int_wasm_memory_init (i32 imm:$seg), (i32 imm:$idx), I32:$dest,
+            I32:$offset, I32:$size
+          )],
+         "memory.init\t$seg, $idx, $dest, $offset, $size",
+         "memory.init\t$seg, $idx", 0x08>;
+
+//===----------------------------------------------------------------------===//
+// data.drop
+//===----------------------------------------------------------------------===//
+
+defm DATA_DROP :
+  BULK_I<(outs), (ins i32imm_op:$seg), (outs), (ins i32imm_op:$seg),
+         [(int_wasm_data_drop (i32 imm:$seg))],
+         "data.drop\t$seg", "data.drop\t$seg", 0x09>;
+
+//===----------------------------------------------------------------------===//
 // memory.copy
 //===----------------------------------------------------------------------===//
 
 let mayLoad = 1, mayStore = 1 in
-defm MEMORY_COPY : BULK_I<(outs), (ins I32:$dst, I32:$src, I32:$len),
-                          (outs), (ins),
-                          [(wasm_memcpy I32:$dst, I32:$src, I32:$len)],
-                          "memory.copy\t$dst, $src, $len",
-                          "memory.copy", 0x0a>;
+defm MEMORY_COPY :
+  BULK_I<(outs), (ins i32imm_op:$src_idx, i32imm_op:$dst_idx,
+                      I32:$dst, I32:$src, I32:$len),
+         (outs), (ins i32imm_op:$src_idx, i32imm_op:$dst_idx),
+         [(wasm_memcpy (i32 imm:$src_idx), (i32 imm:$dst_idx),
+           I32:$dst, I32:$src, I32:$len
+         )],
+         "memory.copy\t$src_idx, $dst_idx, $dst, $src, $len",
+         "memory.copy\t$src_idx, $dst_idx", 0x0a>;

llvm/lib/Target/WebAssembly/WebAssemblySelectionDAGInfo.cpp

 #include "WebAssemblyTargetMachine.h"
 using namespace llvm;
 
 #define DEBUG_TYPE "wasm-selectiondag-info"
 
 WebAssemblySelectionDAGInfo::~WebAssemblySelectionDAGInfo() = default; // anchor
 
 SDValue WebAssemblySelectionDAGInfo::EmitTargetCodeForMemcpy(
-    SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Op1, SDValue Op2,
-    SDValue Op3, unsigned Align, bool IsVolatile, bool AlwaysInline,
+    SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Dst, SDValue Src,
+    SDValue Size, unsigned Align, bool IsVolatile, bool AlwaysInline,
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
   if (!DAG.getMachineFunction()
            .getSubtarget<WebAssemblySubtarget>()
            .hasBulkMemory())
     return SDValue();
 
-  return DAG.getNode(WebAssemblyISD::MEMORY_COPY, DL, MVT::Other, Chain, Op1,
-                     Op2, Op3);
+  SDValue MemIdx = DAG.getConstant(0, DL, MVT::i32);
+  return DAG.getNode(WebAssemblyISD::MEMORY_COPY, DL, MVT::Other,
+                     {Chain, MemIdx, MemIdx, Dst, Src,
+                      DAG.getZExtOrTrunc(Size, DL, MVT::i32)});
 }
 
 SDValue WebAssemblySelectionDAGInfo::EmitTargetCodeForMemmove(
     SelectionDAG &DAG, const SDLoc &DL, SDValue Chain, SDValue Op1, SDValue Op2,
     SDValue Op3, unsigned Align, bool IsVolatile,
     MachinePointerInfo DstPtrInfo, MachinePointerInfo SrcPtrInfo) const {
   return EmitTargetCodeForMemcpy(DAG, DL, Chain, Op1, Op2, Op3, Align,
                                  IsVolatile, false, DstPtrInfo,
                                  SrcPtrInfo);
 }

llvm/test/CodeGen/WebAssembly/bulk-memory-intrinsics.ll

+; RUN: llc < %s -asm-verbose=false -verify-machineinstrs -disable-wasm-fallthrough-return-opt -wasm-disable-explicit-locals -wasm-keep-registers -mattr=+bulk-memory | FileCheck %s
+
+; Test that bulk memory intrinsics lower correctly
+
+target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
+target triple = "wasm32-unknown-unknown"
+
+; CHECK-LABEL: memory_init:
+; CHECK-NEXT: .functype  memory_init (i32, i32, i32) -> ()
+; CHECK-NEXT: memory.init 3, 0, $0, $1, $2
+; CHECK-NEXT: return
+declare void @llvm.wasm.memory.init(i32, i32, i8*, i32, i32)
+define void @memory_init(i8* %dest, i32 %offset, i32 %size) {
+  call void @llvm.wasm.memory.init(
+    i32 3, i32 0, i8* %dest, i32 %offset, i32 %size
+  )
+  ret void
+}
+
+; CHECK-LABEL: data_drop:
+; CHECK-NEXT: .functype data_drop () -> ()
+; CHECK-NEXT: data.drop 3
+; CHECK-NEXT: return
+declare void @llvm.wasm.data.drop(i32)
+define void @data_drop() {
+  call void @llvm.wasm.data.drop(i32 3)
+  ret void
+}

llvm/test/CodeGen/WebAssembly/bulk-memory.ll

 ; RUN: llc < %s -asm-verbose=false -verify-machineinstrs -disable-wasm-fallthrough-return-opt -wasm-disable-explicit-locals -wasm-keep-registers -mattr=+bulk-memory | FileCheck %s --check-prefixes CHECK,BULK-MEM
 ; RUN: llc < %s -asm-verbose=false -verify-machineinstrs -disable-wasm-fallthrough-return-opt -wasm-disable-explicit-locals -wasm-keep-registers -mattr=-bulk-memory | FileCheck %s --check-prefixes CHECK,NO-BULK-MEM
 
 ; Test that basic bulk memory codegen works correctly
 
 target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown"
 
+declare void @llvm.memcpy.p0i8.p0i8.i32(i8*, i8*, i32, i1)
+declare void @llvm.memcpy.p0i32.p0i32.i32(i32*, i32*, i32, i1)
+
+declare void @llvm.memmove.p0i8.p0i8.i32(i8*, i8*, i32, i1)
+declare void @llvm.memmove.p0i32.p0i32.i32(i32*, i32*, i32, i1)
+
 ; CHECK-LABEL: memcpy_i8:
 ; NO-BULK-MEM-NOT: memory.copy
 ; BULK-MEM-NEXT: .functype memcpy_i8 (i32, i32, i32) -> ()
-; BULK-MEM-NEXT: memory.copy $0, $1, $2
+; BULK-MEM-NEXT: memory.copy 0, 0, $0, $1, $2
 ; BULK-MEM-NEXT: return
-declare void @llvm.memcpy.p0i8.p0i8.i32(
-  i8* %dest, i8* %src, i32 %len, i1 %volatile
-)
 define void @memcpy_i8(i8* %dest, i8* %src, i32 %len) {
   call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 %len, i1 0)
   ret void
 }
 
 ; CHECK-LABEL: memmove_i8:
 ; NO-BULK-MEM-NOT: memory.copy
 ; BULK-MEM-NEXT: .functype memmove_i8 (i32, i32, i32) -> ()
-; BULK-MEM-NEXT: memory.copy $0, $1, $2
+; BULK-MEM-NEXT: memory.copy 0, 0, $0, $1, $2
 ; BULK-MEM-NEXT: return
-declare void @llvm.memmove.p0i8.p0i8.i32(
-  i8* %dest, i8* %src, i32 %len, i1 %volatile
-)
 define void @memmove_i8(i8* %dest, i8* %src, i32 %len) {
   call void @llvm.memmove.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 %len, i1 0)
   ret void
 }
 
 ; CHECK-LABEL: memcpy_i32:
 ; NO-BULK-MEM-NOT: memory.copy
 ; BULK-MEM-NEXT: .functype memcpy_i32 (i32, i32, i32) -> ()
-; BULK-MEM-NEXT: memory.copy $0, $1, $2
+; BULK-MEM-NEXT: memory.copy 0, 0, $0, $1, $2
 ; BULK-MEM-NEXT: return
-declare void @llvm.memcpy.p0i32.p0i32.i32(
-  i32* %dest, i32* %src, i32 %len, i1 %volatile
-)
 define void @memcpy_i32(i32* %dest, i32* %src, i32 %len) {
   call void @llvm.memcpy.p0i32.p0i32.i32(i32* %dest, i32* %src, i32 %len, i1 0)
   ret void
 }
 
 ; CHECK-LABEL: memmove_i32:
 ; NO-BULK-MEM-NOT: memory.copy
 ; BULK-MEM-NEXT: .functype memmove_i32 (i32, i32, i32) -> ()
-; BULK-MEM-NEXT: memory.copy $0, $1, $2
+; BULK-MEM-NEXT: memory.copy 0, 0, $0, $1, $2
 ; BULK-MEM-NEXT: return
-declare void @llvm.memmove.p0i32.p0i32.i32(
-  i32* %dest, i32* %src, i32 %len, i1 %volatile
-)
 define void @memmove_i32(i32* %dest, i32* %src, i32 %len) {
   call void @llvm.memmove.p0i32.p0i32.i32(i32* %dest, i32* %src, i32 %len, i1 0)
   ret void
 }
 
 ; CHECK-LABEL: memcpy_1:
 ; CHECK-NEXT: .functype memcpy_1 (i32, i32) -> ()
 ; CHECK-NEXT: i32.load8_u $push[[L0:[0-9]+]]=, 0($1)
define void @memmove_1(i8* %dest, i8* %src) {
   call void @llvm.memmove.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 1, i1 0)
   ret void
 }
 
 ; CHECK-LABEL: memcpy_1024:
 ; NO-BULK-MEM-NOT: memory.copy
 ; BULK-MEM-NEXT: .functype memcpy_1024 (i32, i32) -> ()
 ; BULK-MEM-NEXT: i32.const $push[[L0:[0-9]+]]=, 1024
-; BULK-MEM-NEXT: memory.copy $0, $1, $pop[[L0]]
+; BULK-MEM-NEXT: memory.copy 0, 0, $0, $1, $pop[[L0]]
 ; BULK-MEM-NEXT: return
 define void @memcpy_1024(i8* %dest, i8* %src) {
   call void @llvm.memcpy.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 1024, i1 0)
   ret void
 }
 
 ; CHECK-LABEL: memmove_1024:
 ; NO-BULK-MEM-NOT: memory.copy
 ; BULK-MEM-NEXT: .functype memmove_1024 (i32, i32) -> ()
 ; BULK-MEM-NEXT: i32.const $push[[L0:[0-9]+]]=, 1024
-; BULK-MEM-NEXT: memory.copy $0, $1, $pop[[L0]]
+; BULK-MEM-NEXT: memory.copy 0, 0, $0, $1, $pop[[L0]]
 ; BULK-MEM-NEXT: return
 define void @memmove_1024(i8* %dest, i8* %src) {
   call void @llvm.memmove.p0i8.p0i8.i32(i8* %dest, i8* %src, i32 1024, i1 0)
   ret void
 }