[SelectionDAG] Generate scoped AA metadata when lowering memcpy.

[hliao]

Summary

• When memcpy is lowered in SelectionDAG, similar to inlining a function, scoped AA metadata should be generated and attached on the lowered loads/stores following 'noalias' arguments.

[hliao]

llvm/test/CodeGen/AArch64/arm64-2012-05-07-MemcpyAlignBug.ll

12

With that extra scoped AA info, this store is freely scheduled ahead to minimize register pressure as cyclone disables the latency heuristic.

llvm/test/CodeGen/AArch64/arm64-memcpy-inline.ll

33

Similar change due to 'cyclone' schedule favors register pressure over latency.

llvm/test/CodeGen/AArch64/arm64-misaligned-memcpy-inline.ll

36

functionality wise the same. The sequence of offset looks better in terms of the locality.

llvm/test/CodeGen/PowerPC/pr45301.ll

29

The same from the functionality wise but loads from that memcpy are scheduled ahead in the favor of latency hiding.

[nikic]

Do I understand correctly that this patch is trying to claim that the memcpy src and dst do not alias through scoped alias metadata? If so, I'm afraid this is incorrect. Our contract for llvm.memcpy requires that src/dst are either NoAlias or MustAlias (but not PartialAlias). From LangRef:

The ‘llvm.memcpy.*’ intrinsics copy a block of memory from the source location to the destination location, which must either be equal or non-overlapping.

[hliao]

In D102255#2751896, @nikic wrote:

Do I understand correctly that this patch is trying to claim that the memcpy src and dst do not alias through scoped alias metadata? If so, I'm afraid this is incorrect. Our contract for llvm.memcpy requires that src/dst are either NoAlias or MustAlias (but not PartialAlias). From LangRef:

The ‘llvm.memcpy.*’ intrinsics copy a block of memory from the source location to the destination location, which must either be equal or non-overlapping.

The langref says "The ‘llvm.memcpy.*’ intrinsics copy a block of memory from the source location to the destination location, which must either be equal or non-overlapping. It copies “len” bytes of memory over. If the argument is known to be aligned to some boundary, this can be specified as an attribute on the argument.". It says that src and dst are either equal or non-overlapping. For the equal case, they should be eliminated earlier. Even if there are remaining ones, claiming them 'NoAlias' doesn't have correctness issue when copying them. It won't cause correctness issue if we schedule those loads and stores freely.
In addition, the extra AA scope metadata here only has impact on this instance of memcpy.

[jeroen.dobbelaere]

It does look like a valid way to indicate that the individual loads and stores are independent, except for their value dependency.

Given the very late introduction of new !alias.scope and !noalias metadata, is their a way to have a testcase look at a machine ir dump, together with the metadata output at that phase ?

[hliao]

In D102255#2752039, @jeroen.dobbelaere wrote:

It does look like a valid way to indicate that the individual loads and stores are independent, except for their value dependency.

Given the very late introduction of new !alias.scope and !noalias metadata, is their a way to have a testcase look at a machine ir dump, together with the metadata output at that phase ?

That's on my plan to enable the dumping of scoped AA in MIR. These AA metadata are generated within the backend and need special support to dump them friendly.

[nikic]

In D102255#2751961, @hliao wrote:

In D102255#2751896, @nikic wrote:

Do I understand correctly that this patch is trying to claim that the memcpy src and dst do not alias through scoped alias metadata? If so, I'm afraid this is incorrect. Our contract for llvm.memcpy requires that src/dst are either NoAlias or MustAlias (but not PartialAlias). From LangRef:

The ‘llvm.memcpy.*’ intrinsics copy a block of memory from the source location to the destination location, which must either be equal or non-overlapping.

The langref says "The ‘llvm.memcpy.*’ intrinsics copy a block of memory from the source location to the destination location, which must either be equal or non-overlapping. It copies “len” bytes of memory over. If the argument is known to be aligned to some boundary, this can be specified as an attribute on the argument.". It says that src and dst are either equal or non-overlapping. For the equal case, they should be eliminated earlier.

Generally, you do not know whether they are equal or not.

Even if there are remaining ones, claiming them 'NoAlias' doesn't have correctness issue when copying them. It won't cause correctness issue if we schedule those loads and stores freely.

I agree that rescheduling the loads/stores is correct even if src and dst are equal. However, the metadata itself is still incorrect: It will claim that the loads/stores are NoAlias, even though they are actually MustAlias.

[jeroen.dobbelaere]

I propose to discuss this in tomorrow's aa tech call.

[jeroen.dobbelaere]

In D102255#2755372, @nikic wrote:

I agree that rescheduling the loads/stores is correct even if src and dst are equal. However, the metadata itself is still incorrect: It will claim that the loads/stores are NoAlias, even though they are actually MustAlias.

The proposed version introduces 2 scopes, assigning all store to one scope and all loads to another.

I order to avoid the MustAlias vs NoAlias problem, we could introduce a scope per load-store pair. Of course, that might be a lot of extra scopes..

[jeroen.dobbelaere]

In D102255#2765788, @jeroen.dobbelaere wrote:

In D102255#2755372, @nikic wrote:

I agree that rescheduling the loads/stores is correct even if src and dst are equal. However, the metadata itself is still incorrect: It will claim that the loads/stores are NoAlias, even though they are actually MustAlias.

The proposed version introduces 2 scopes, assigning all store to one scope and all loads to another.

I order to avoid the MustAlias vs NoAlias problem, we could introduce a scope per load-store pair. Of course, that might be a lot of extra scopes..

This was discussed earlier today in LLVM's AA Technical call. This method should work, but one possible gotcha came up: sometimes a memcpy lowering results in overlapping load/stores. Those overlapping load/stores must remain 'aliasing', so they should belong to the same scope.

possible example:

// memcpy(dst, src, 23) becomes:
store i64 dst+0, (load i64 src+0)    // scope 0
store i64 dst+8, (load i64, src+8)   // scope 1, not overlapping with previous pair
store i64 dst+15, (load i64, src+15) // also scope 1, overlapping with previous pair

[hliao]

Add MIR printer and parser support for scoped AA metadata generated in the backend.

[hliao]

In D102255#2766721, @jeroen.dobbelaere wrote:

In D102255#2765788, @jeroen.dobbelaere wrote:

In D102255#2755372, @nikic wrote:

I agree that rescheduling the loads/stores is correct even if src and dst are equal. However, the metadata itself is still incorrect: It will claim that the loads/stores are NoAlias, even though they are actually MustAlias.

The proposed version introduces 2 scopes, assigning all store to one scope and all loads to another.

I order to avoid the MustAlias vs NoAlias problem, we could introduce a scope per load-store pair. Of course, that might be a lot of extra scopes..

This was discussed earlier today in LLVM's AA Technical call. This method should work, but one possible gotcha came up: sometimes a memcpy lowering results in overlapping load/stores. Those overlapping load/stores must remain 'aliasing', so they should belong to the same scope.

possible example:

// memcpy(dst, src, 23) becomes:
store i64 dst+0, (load i64 src+0)    // scope 0
store i64 dst+8, (load i64, src+8)   // scope 1, not overlapping with previous pair
store i64 dst+15, (load i64, src+15) // also scope 1, overlapping with previous pair

Not being able to catch that meeting. I am not sure the current SDAG would generate that. Let me check. If we did that in the current SDAG implementation, we may remove scoped AA metadata for that trailing (or maybe the heading) loads as the conservative solution before we introduce new scopes.

[hliao]

Just read the relevant threads and bugs reported on the change of allowing exact-overlap on llvm.memcpy. See the reference list at the end. Personally, I think it's OK to assume NoAlias added here. By allowing exact-overlap in llvm.memcpy, the most significant change is on the basic-aa, which must consider the case where the source and destination of llvm.memcpy is the same. The make senses at the IR level, where llvm.memcpy is treated as a single op as the exact-overlap means the copy is a no-op and won't always overwrite the destination memory. But, where we lower that copy into loads/stores, we say that loads/stores won't alias. That's fine as the order between loads and stores on the same offset (or location for the exact-overlap case) is established through the data dependency. In addition, the no-alias established here is a scoped one, which only applies to loads/stored from this llvm.memcpy only. It won't affect the AA result between them to loads/stores out of the scope. (This patch depends on https://reviews.llvm.org/D102215, which propagates scoped AA on mem ops into loads/stores after lowering.)

But, there are definitely issues where the backend may split/merge/narrow/widen memory operations. For split and narrow, they seem fine. I started to address the merge issue in https://reviews.llvm.org/D102821 to ensure the correct scoped AA is used on the merged stores or loads. For widen, mostly load widening, we have a quite tricky issue as the widening result, in most cases, doesn't care about extra content loaded from memory. They want to choose the widened one to maximize performance but we have the risk that simply reusing the scoped AA metadata may result in undesired behavior.

https://lists.llvm.org/pipermail/cfe-dev/2020-August/066614.html

[hliao]

PING

[hliao]

Rebase.

[jeroen.dobbelaere]

In D102255#2769466, @hliao wrote:

Add MIR printer and parser support for scoped AA metadata generated in the backend.

Hi Michael, I think it would be better (easier to review) if you could put this part in a separate patch.

[nikic]

Can you please split off the metadata parsing part into a separate patch? It's not directly related to the memcpy lowering.

[hliao]

In D102255#2780327, @nikic wrote:

Can you please split off the metadata parsing part into a separate patch? It's not directly related to the memcpy lowering.

OK, I need to add unit tests to verify that as we won't be able to generate machine metadata within the current backend. I will prepare that this night.

[hliao]

In D102255#2780461, @hliao wrote:

In D102255#2780327, @nikic wrote:

Can you please split off the metadata parsing part into a separate patch? It's not directly related to the memcpy lowering.

OK, I need to add unit tests to verify that as we won't be able to generate machine metadata within the current backend. I will prepare that this night.

MIR printer change is split into https://reviews.llvm.org/D103205

[hliao]

Clean up after splitting MIR printer and parser changes.

[hliao]

In D102255#2785665, @hliao wrote:

Clean up after splitting MIR printer and parser changes.

here's the printer change (D103205) and parse change (D103282)

[hliao]

Rebase

[jeroen.dobbelaere]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6399–6422

As mentioned in D102255#2766721, we still would need to use separate scopes for ever load/store pair.
That's the only way to avoid a possible 'noalias' result when the source and destination happen to be identical.
(based on the AA Techcall discussion, such a situation was considered to be not acceptable).

[hliao]

Rebase to recent changes on the MIR printer (already committed) and parser (under review).

[hliao]

Rebase

[hliao]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

6399–6422

Separating scopes for each load/store pair make it impossible to schedule loads/stores freely generated from the same memcpy as they won't belong to the same scope. The exact overlap described in memcpy builtin refers to the situation where an optimizer should not assume distinct memory operands from the outside of memcpy.
In another perspective, such lowering of memcpy is eqivalent to inlining a memcpy function implementation at the LLVM IR level.
Could you explain more about the concern?

[hliao]

Rebase and resolve conflicts. Kindly PING for review.

BTW, the newly generated AA metadata only changes the alias checking results among loads/stores generated from this lowered memcpy. It won't change other alias checking result.

llvm/include/llvm/AsmParser/SlotMapping.h

 /// nodes, global values and types. It also contains the mapping for the named
 /// types.
 /// It can be used to save the parsing state of an LLVM IR module so that the
 /// textual references to the values in the module can be parsed outside of the
 /// module's source.
 struct SlotMapping {
   std::vector<GlobalValue *> GlobalValues;
   std::map<unsigned, TrackingMDNodeRef> MetadataNodes;
+  std::map<unsigned, TempMDTuple> ForwardRefMDNodes;
   StringMap<Type *> NamedTypes;
   std::map<unsigned, Type *> Types;
 };
 
 } // end namespace llvm
 
 #endif

llvm/include/llvm/CodeGen/MIRParser/MIParser.h

public:
 
   void setTarget(const TargetSubtargetInfo &NewSubtarget);
 };
 
 struct PerFunctionMIParsingState {
   BumpPtrAllocator Allocator;
   MachineFunction &MF;
   SourceMgr *SM;
-  const SlotMapping &IRSlots;
+  SlotMapping &IRSlots;
   PerTargetMIParsingState &Target;
 
   DenseMap<unsigned, MachineBasicBlock *> MBBSlots;
   DenseMap<Register, VRegInfo *> VRegInfos;
   StringMap<VRegInfo *> VRegInfosNamed;
   DenseMap<unsigned, int> FixedStackObjectSlots;
   DenseMap<unsigned, int> StackObjectSlots;
   DenseMap<unsigned, unsigned> ConstantPoolSlots;
   DenseMap<unsigned, unsigned> JumpTableSlots;
 
   /// Maps from slot numbers to function's unnamed values.
   DenseMap<unsigned, const Value *> Slots2Values;
 
   PerFunctionMIParsingState(MachineFunction &MF, SourceMgr &SM,
-                            const SlotMapping &IRSlots,
+                            SlotMapping &IRSlots,
                             PerTargetMIParsingState &Target);
 
   VRegInfo &getVRegInfo(Register Num);
   VRegInfo &getVRegInfoNamed(StringRef RegName);
   const Value *getIRValue(unsigned Slot);
 };
 
 /// Parse the machine basic block definitions, and skip the machine
bool parseVirtualRegisterReference(PerFunctionMIParsingState &PFS,
                                    SMDiagnostic &Error);
 
 bool parseStackObjectReference(PerFunctionMIParsingState &PFS, int &FI,
                                StringRef Src, SMDiagnostic &Error);
 
 bool parseMDNode(PerFunctionMIParsingState &PFS, MDNode *&Node, StringRef Src,
                  SMDiagnostic &Error);
 
+bool parseStandaloneMetadata(PerFunctionMIParsingState &PFS, StringRef Src,
+                             SMDiagnostic &Error);
+
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_MIRPARSER_MIPARSER_H

llvm/include/llvm/CodeGen/MIRYamlMapping.h

struct MachineFunction {
   MachineFrameInfo FrameInfo;
   std::vector<FixedMachineStackObject> FixedStackObjects;
   std::vector<MachineStackObject> StackObjects;
   std::vector<MachineConstantPoolValue> Constants; /// Constant pool.
   std::unique_ptr<MachineFunctionInfo> MachineFuncInfo;
   std::vector<CallSiteInfo> CallSitesInfo;
   std::vector<DebugValueSubstitution> DebugValueSubstitutions;
   MachineJumpTable JumpTableInfo;
+  std::vector<StringValue> MachineMetadataNodes;
   BlockStringValue Body;
 };
 
 template <> struct MappingTraits<MachineFunction> {
   static void mapping(IO &YamlIO, MachineFunction &MF) {
     YamlIO.mapRequired("name", MF.Name);
     YamlIO.mapOptional("alignment", MF.Alignment, None);
     YamlIO.mapOptional("exposesReturnsTwice", MF.ExposesReturnsTwice, false);
YamlIO.mapOptional("callSites", MF.CallSitesInfo,
                        std::vector<CallSiteInfo>());
     YamlIO.mapOptional("debugValueSubstitutions", MF.DebugValueSubstitutions,
                        std::vector<DebugValueSubstitution>());
     YamlIO.mapOptional("constants", MF.Constants,
                        std::vector<MachineConstantPoolValue>());
     YamlIO.mapOptional("machineFunctionInfo", MF.MachineFuncInfo);
     if (!YamlIO.outputting() || !MF.JumpTableInfo.Entries.empty())
       YamlIO.mapOptional("jumpTable", MF.JumpTableInfo, MachineJumpTable());
+    if (!YamlIO.outputting() || !MF.MachineMetadataNodes.empty())
+      YamlIO.mapOptional("machineMetadataNodes", MF.MachineMetadataNodes,
+                         std::vector<StringValue>());
     YamlIO.mapOptional("body", MF.Body, BlockStringValue());
   }
 };
 
 } // end namespace yaml
 } // end namespace llvm
 
 #endif // LLVM_CODEGEN_MIRYAMLMAPPING_H

llvm/include/llvm/CodeGen/MachineModuleSlotTracker.h

+#ifndef LLVM_CODEGEN_MACHINEMODULESLOTTRACKER_H
+#define LLVM_CODEGEN_MACHINEMODULESLOTTRACKER_H
+
+#include "llvm/IR/ModuleSlotTracker.h"
+
+namespace llvm {
+
+class AbstractSlotTrackerStorage;
+class Function;
+class MachineModuleInfo;
+class MachineFunction;
+class Module;
+
+class MachineModuleSlotTracker : public ModuleSlotTracker {
+  const MachineModuleInfo &MMI;
+  unsigned MDNStartSlot, MDNEndSlot;
+
+  void processMachineFunctionMetadata(AbstractSlotTrackerStorage *AST,
+                                      const MachineFunction &MF);
+  void processMachineModule(AbstractSlotTrackerStorage *AST, const Module *M,
+                            bool ShouldInitializeAllMetadata);
+  void processMachineFunction(AbstractSlotTrackerStorage *AST,
+                              const Function *F,
+                              bool ShouldInitializeAllMetadata);
+
+public:
+  MachineModuleSlotTracker(const MachineModuleInfo &MMI, const Module *M,
+                           bool ShouldInitializeAllMetadata = true);
+  MachineModuleSlotTracker(const MachineFunction *MF,
+                           bool ShouldInitializeAllMetadata = true);
+  ~MachineModuleSlotTracker();
+
+  void collectMachineMDNodes(MachineMDNodeListType &L) const;
+};
+
+} // namespace llvm
+
+#endif // LLVM_CODEGEN_MACHINEMODULESLOTTRACKER_H

llvm/include/llvm/IR/ModuleSlotTracker.h

 //===-- llvm/IR/ModuleSlotTracker.h -----------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_IR_MODULESLOTTRACKER_H
 #define LLVM_IR_MODULESLOTTRACKER_H
 
+#include <functional>
 #include <memory>
+#include <utility>
+#include <vector>
 
 namespace llvm {
 
 class Module;
 class Function;
 class SlotTracker;
 class Value;
+class MDNode;
+
+/// Abstract interface of slot tracker storage.
+class AbstractSlotTrackerStorage {
+public:
+  virtual ~AbstractSlotTrackerStorage();
+
+  virtual unsigned getNextMetadataSlot() = 0;
+
+  virtual void createMetadataSlot(const MDNode *) = 0;
+  virtual int getMetadataSlot(const MDNode *) = 0;
+};
 
 /// Manage lifetime of a slot tracker for printing IR.
 ///
 /// Wrapper around the \a SlotTracker used internally by \a AsmWriter.  This
 /// class allows callers to share the cost of incorporating the metadata in a
 /// module or a function.
 ///
 /// If the IR changes from underneath \a ModuleSlotTracker, strings like
 /// "<badref>" will be printed, or, worse, the wrong slots entirely.
 class ModuleSlotTracker {
   /// Storage for a slot tracker.
   std::unique_ptr<SlotTracker> MachineStorage;
   bool ShouldCreateStorage = false;
   bool ShouldInitializeAllMetadata = false;
 
   const Module *M = nullptr;
   const Function *F = nullptr;
   SlotTracker *Machine = nullptr;
 
+  std::function<void(AbstractSlotTrackerStorage *, const Module *, bool)>
+      processModuleHookFn;
+  std::function<void(AbstractSlotTrackerStorage *, const Function *, bool)>
+      processFunctionHookFn;
+
 public:
   /// Wrap a preinitialized SlotTracker.
   ModuleSlotTracker(SlotTracker &Machine, const Module *M,
                     const Function *F = nullptr);
 
   /// Construct a slot tracker from a module.
   ///
   /// If \a M is \c nullptr, uses a null slot tracker.  Otherwise, initializes
   /// a slot tracker, and initializes all metadata slots.  \c
   /// ShouldInitializeAllMetadata defaults to true because this is expected to
   /// be shared between multiple callers, and otherwise MDNode references will
   /// not match up.
   explicit ModuleSlotTracker(const Module *M,
                              bool ShouldInitializeAllMetadata = true);
 
   /// Destructor to clean up storage.
-  ~ModuleSlotTracker();
+  virtual ~ModuleSlotTracker();
 
   /// Lazily creates a slot tracker.
   SlotTracker *getMachine();
 
   const Module *getModule() const { return M; }
   const Function *getCurrentFunction() const { return F; }
 
   /// Incorporate the given function.
   ///
   /// Purge the currently incorporated function and incorporate \c F.  If \c F
   /// is currently incorporated, this is a no-op.
   void incorporateFunction(const Function &F);
 
   /// Return the slot number of the specified local value.
   ///
   /// A function that defines this value should be incorporated prior to calling
   /// this method.
   /// Return -1 if the value is not in the function's SlotTracker.
   int getLocalSlot(const Value *V);
+
+  void setProcessHook(
+      std::function<void(AbstractSlotTrackerStorage *, const Module *, bool)>);
+  void setProcessHook(std::function<void(AbstractSlotTrackerStorage *,
+                                         const Function *, bool)>);
+
+  using MachineMDNodeListType =
+      std::vector<std::pair<unsigned, const MDNode *>>;
+
+  void collectMDNodes(MachineMDNodeListType &L, unsigned LB, unsigned UB) const;
 };
 
 } // end namespace llvm
 
 #endif

llvm/lib/CodeGen/CMakeLists.txt

add_llvm_component_library(LLVMCodeGen
   MachineFunctionSplitter.cpp
   MachineInstrBundle.cpp
   MachineInstr.cpp
   MachineLICM.cpp
   MachineLoopInfo.cpp
   MachineLoopUtils.cpp
   MachineModuleInfo.cpp
   MachineModuleInfoImpls.cpp
+  MachineModuleSlotTracker.cpp
   MachineOperand.cpp
   MachineOptimizationRemarkEmitter.cpp
   MachineOutliner.cpp
   MachinePassManager.cpp
   MachinePipeliner.cpp
   MachinePostDominators.cpp
   MachineRegionInfo.cpp
   MachineRegisterInfo.cpp

llvm/lib/CodeGen/MIRParser/MILexer.h

enum TokenKind {
     kw_shufflemask,
     kw_pre_instr_symbol,
     kw_post_instr_symbol,
     kw_heap_alloc_marker,
     kw_bbsections,
     kw_unknown_size,
     kw_unknown_address,
 
+    // Metadata types.
+    kw_distinct,
+    kw_null,
+
     // Named metadata keywords
     md_tbaa,
     md_alias_scope,
     md_noalias,
     md_range,
     md_diexpr,
     md_dilocation,
 

llvm/lib/CodeGen/MIRParser/MILexer.cpp

return StringSwitch<MIToken::TokenKind>(Identifier)
       .Case("intpred", MIToken::kw_intpred)
       .Case("shufflemask", MIToken::kw_shufflemask)
       .Case("pre-instr-symbol", MIToken::kw_pre_instr_symbol)
       .Case("post-instr-symbol", MIToken::kw_post_instr_symbol)
       .Case("heap-alloc-marker", MIToken::kw_heap_alloc_marker)
       .Case("bbsections", MIToken::kw_bbsections)
       .Case("unknown-size", MIToken::kw_unknown_size)
       .Case("unknown-address", MIToken::kw_unknown_address)
+      .Case("distinct", MIToken::kw_distinct)
+      .Case("null", MIToken::kw_null)
       .Default(MIToken::Identifier);
 }
 
 static Cursor maybeLexIdentifier(Cursor C, MIToken &Token) {
   if (!isalpha(C.peek()) && C.peek() != '_')
     return None;
   auto Range = C;
   while (isIdentifierChar(C.peek()))

llvm/lib/CodeGen/MIRParser/MIParser.cpp

 const RegisterBank *PerTargetMIParsingState::getRegBank(StringRef Name) {
   auto RegBankInfo = Names2RegBanks.find(Name);
   if (RegBankInfo == Names2RegBanks.end())
     return nullptr;
   return RegBankInfo->getValue();
 }
 
 PerFunctionMIParsingState::PerFunctionMIParsingState(MachineFunction &MF,
-    SourceMgr &SM, const SlotMapping &IRSlots, PerTargetMIParsingState &T)
-  : MF(MF), SM(&SM), IRSlots(IRSlots), Target(T) {
-}
+                                                     SourceMgr &SM,
+                                                     SlotMapping &IRSlots,
+                                                     PerTargetMIParsingState &T)
+    : MF(MF), SM(&SM), IRSlots(IRSlots), Target(T) {}
 
 VRegInfo &PerFunctionMIParsingState::getVRegInfo(Register Num) {
   auto I = VRegInfos.insert(std::make_pair(Num, nullptr));
   if (I.second) {
     MachineRegisterInfo &MRI = MF.getRegInfo();
     VRegInfo *Info = new (Allocator) VRegInfo;
     Info->VReg = MRI.createIncompleteVirtualRegister();
     I.first->second = Info;
public:
   bool parseBasicBlocks();
   bool parse(MachineInstr *&MI);
   bool parseStandaloneMBB(MachineBasicBlock *&MBB);
   bool parseStandaloneNamedRegister(Register &Reg);
   bool parseStandaloneVirtualRegister(VRegInfo *&Info);
   bool parseStandaloneRegister(Register &Reg);
   bool parseStandaloneStackObject(int &FI);
   bool parseStandaloneMDNode(MDNode *&Node);
+  bool parseStandaloneMetadata();
+  bool parseMDTuple(MDNode *&MD, bool IsDistinct);
+  bool parseMDNodeVector(SmallVectorImpl<Metadata *> &Elts);
+  bool parseMetadata(Metadata *&MD);
 
   bool
   parseBasicBlockDefinition(DenseMap<unsigned, MachineBasicBlock *> &MBBSlots);
   bool parseBasicBlock(MachineBasicBlock &MBB,
                        MachineBasicBlock *&AddFalthroughFrom);
   bool parseBasicBlockLiveins(MachineBasicBlock &MBB);
   bool parseBasicBlockSuccessors(MachineBasicBlock &MBB);
 
if (parseDILocation(Node))
       return true;
   } else
     return error("expected a metadata node");
   if (Token.isNot(MIToken::Eof))
     return error("expected end of string after the metadata node");
   return false;
 }
 
+bool MIParser::parseStandaloneMetadata() {
+  lex();
+  if (Token.isNot(MIToken::exclaim))
+    return error("expected a metadata node");
+
+  lex();
+  if (Token.isNot(MIToken::IntegerLiteral) || Token.integerValue().isSigned())
+    return error("expected metadata id after '!'");
+  unsigned ID = 0;
+  if (getUnsigned(ID))
+    return true;
+  lex();
+  if (expectAndConsume(MIToken::equal))
+    return true;
+  bool IsDistinct = Token.is(MIToken::kw_distinct);
+  if (IsDistinct)
+    lex();
+  if (Token.isNot(MIToken::exclaim))
+    return error("expected a metadata node");
+  lex();
+
+  MDNode *MD;
+  if (parseMDTuple(MD, IsDistinct))
+    return true;
+
+  auto FI = PFS.IRSlots.ForwardRefMDNodes.find(ID);
+  if (FI != PFS.IRSlots.ForwardRefMDNodes.end()) {
+    FI->second->replaceAllUsesWith(MD);
+    PFS.IRSlots.ForwardRefMDNodes.erase(FI);
+
+    assert(PFS.IRSlots.MetadataNodes[ID] == MD && "Tracking VH didn't work");
+  } else {
+    if (PFS.IRSlots.MetadataNodes.count(ID))
+      return error("Metadata id is already used");
+    PFS.IRSlots.MetadataNodes[ID].reset(MD);
+  }
+
+  return false;
+}
+
+bool MIParser::parseMDTuple(MDNode *&MD, bool IsDistinct) {
+  SmallVector<Metadata *, 16> Elts;
+  if (parseMDNodeVector(Elts))
+    return true;
+  MD = (IsDistinct ? MDTuple::getDistinct
+                   : MDTuple::get)(MF.getFunction().getContext(), Elts);
+  return false;
+}
+
+bool MIParser::parseMDNodeVector(SmallVectorImpl<Metadata *> &Elts) {
+  if (Token.isNot(MIToken::lbrace))
+    return error("expected '{' here");
+  lex();
+
+  if (Token.is(MIToken::rbrace)) {
+    lex();
+    return false;
+  }
+
+  do {
+    if (Token.is(MIToken::kw_null)) {
+      lex();
+      Elts.push_back(nullptr);
+      continue;
+    }
+
+    Metadata *MD;
+    if (parseMetadata(MD))
+      return true;
+
+    Elts.push_back(MD);
+
+    if (Token.isNot(MIToken::comma))
+      break;
+    lex();
+  } while (true);
+
+  if (Token.isNot(MIToken::rbrace))
+    return error("expected end of metadata node");
+  lex();
+
+  return false;
+}
+
+// ::= !42
+// ::= !"string"
+bool MIParser::parseMetadata(Metadata *&MD) {
+  if (Token.isNot(MIToken::exclaim))
+    return error("expected '!' here");
+  lex();
+
+  if (Token.is(MIToken::StringConstant)) {
+    std::string Str;
+    if (parseStringConstant(Str))
+      return true;
+    MD = MDString::get(MF.getFunction().getContext(), Str);
+    return false;
+  }
+
+  if (Token.isNot(MIToken::IntegerLiteral) || Token.integerValue().isSigned())
+    return error("expected metadata id after '!'");
+
+  unsigned ID = 0;
+  if (getUnsigned(ID))
+    return true;
+  lex();
+
+  auto NodeInfo = PFS.IRSlots.MetadataNodes.find(ID);
+  if (NodeInfo != PFS.IRSlots.MetadataNodes.end()) {
+    MD = NodeInfo->second.get();
+    return false;
+  }
+  // Forward reference.
+  auto &FwdRef = PFS.IRSlots.ForwardRefMDNodes[ID];
+  FwdRef = MDTuple::getTemporary(MF.getFunction().getContext(), None);
+  PFS.IRSlots.MetadataNodes[ID].reset(FwdRef.get());
+  MD = FwdRef.get();
+
+  return false;
+}
+
 static const char *printImplicitRegisterFlag(const MachineOperand &MO) {
   assert(MO.isImplicit());
   return MO.isDef() ? "implicit-def" : "implicit";
 }
 
 static std::string getRegisterName(const TargetRegisterInfo *TRI,
                                    Register Reg) {
   assert(Register::isPhysicalRegister(Reg) && "expected phys reg");
bool llvm::parseStackObjectReference(PerFunctionMIParsingState &PFS,
   return MIParser(PFS, Error, Src).parseStandaloneStackObject(FI);
 }
 
 bool llvm::parseMDNode(PerFunctionMIParsingState &PFS,
                        MDNode *&Node, StringRef Src, SMDiagnostic &Error) {
   return MIParser(PFS, Error, Src).parseStandaloneMDNode(Node);
 }
 
+bool llvm::parseStandaloneMetadata(PerFunctionMIParsingState &PFS,
+                                   StringRef Src, SMDiagnostic &Error) {
+  return MIParser(PFS, Error, Src).parseStandaloneMetadata();
+}
+
 bool MIRFormatter::parseIRValue(StringRef Src, MachineFunction &MF,
                                 PerFunctionMIParsingState &PFS, const Value *&V,
                                 ErrorCallbackType ErrorCallback) {
   MIToken Token;
   Src = lexMIToken(Src, Token, [&](StringRef::iterator Loc, const Twine &Msg) {
     ErrorCallback(Loc, Msg);
   });
   V = nullptr;
 
   return ::parseIRValue(Token, PFS, V, ErrorCallback);
 }

llvm/lib/CodeGen/MIRParser/MIRParser.cpp

public:
 
   bool initializeConstantPool(PerFunctionMIParsingState &PFS,
                               MachineConstantPool &ConstantPool,
                               const yaml::MachineFunction &YamlMF);
 
   bool initializeJumpTableInfo(PerFunctionMIParsingState &PFS,
                                const yaml::MachineJumpTable &YamlJTI);
 
+  bool parseMachineMetadataNodes(PerFunctionMIParsingState &PFS,
+                                 MachineFunction &MF,
+                                 const yaml::MachineFunction &YMF);
+
 private:
   bool parseMDNode(PerFunctionMIParsingState &PFS, MDNode *&Node,
                    const yaml::StringValue &Source);
 
   bool parseMBBReference(PerFunctionMIParsingState &PFS,
                          MachineBasicBlock *&MBB,
                          const yaml::StringValue &Source);
 
+  bool parseStandaloneMetadata(PerFunctionMIParsingState &PFS,
+                               const yaml::StringValue &Source);
+
   /// Return a MIR diagnostic converted from an MI string diagnostic.
   SMDiagnostic diagFromMIStringDiag(const SMDiagnostic &Error,
                                     SMRange SourceRange);
 
   /// Return a MIR diagnostic converted from a diagnostic located in a YAML
   /// block scalar string.
   SMDiagnostic diagFromBlockStringDiag(const SMDiagnostic &Error,
                                        SMRange SourceRange);
MIRParserImpl::initializeMachineFunction(const yaml::MachineFunction &YamlMF,
   if (parseRegisterInfo(PFS, YamlMF))
     return true;
   if (!YamlMF.Constants.empty()) {
     auto *ConstantPool = MF.getConstantPool();
     assert(ConstantPool && "Constant pool must be created");
     if (initializeConstantPool(PFS, *ConstantPool, YamlMF))
       return true;
   }
+  if (!YamlMF.MachineMetadataNodes.empty() &&
+      parseMachineMetadataNodes(PFS, MF, YamlMF))
+    return true;
 
   StringRef BlockStr = YamlMF.Body.Value.Value;
   SMDiagnostic Error;
   SourceMgr BlockSM;
   BlockSM.AddNewSourceBuffer(
       MemoryBuffer::getMemBuffer(BlockStr, "",/*RequiresNullTerminator=*/false),
       SMLoc());
   PFS.SM = &BlockSM;
bool MIRParserImpl::parseMBBReference(PerFunctionMIParsingState &PFS,
                                       MachineBasicBlock *&MBB,
                                       const yaml::StringValue &Source) {
   SMDiagnostic Error;
   if (llvm::parseMBBReference(PFS, MBB, Source.Value, Error))
     return error(Error, Source.SourceRange);
   return false;
 }
 
+bool MIRParserImpl::parseStandaloneMetadata(PerFunctionMIParsingState &PFS,
+                                            const yaml::StringValue &Source) {
+  SMDiagnostic Error;
+  if (llvm::parseStandaloneMetadata(PFS, Source.Value, Error))
+    return error(Error, Source.SourceRange);
+  return false;
+}
+
+bool MIRParserImpl::parseMachineMetadataNodes(
+    PerFunctionMIParsingState &PFS, MachineFunction &MF,
+    const yaml::MachineFunction &YMF) {
+  for (auto &MDS : YMF.MachineMetadataNodes) {
+    if (parseStandaloneMetadata(PFS, MDS))
+      return true;
+  }
+  return false;
+}
+
 SMDiagnostic MIRParserImpl::diagFromMIStringDiag(const SMDiagnostic &Error,
                                                  SMRange SourceRange) {
   assert(SourceRange.isValid() && "Invalid source range");
   SMLoc Loc = SourceRange.Start;
   bool HasQuote = Loc.getPointer() < SourceRange.End.getPointer() &&
                   *Loc.getPointer() == '\'';
   // Translate the location of the error from the location in the MI string to
   // the corresponding location in the MIR file.

llvm/lib/CodeGen/MIRPrinter.cpp

 #include "llvm/CodeGen/MIRYamlMapping.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
+#include "llvm/CodeGen/MachineModuleSlotTracker.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/IRPrintingPasses.h"
 #include "llvm/IR/InstrTypes.h"
void convert(yaml::MachineFunction &MF,
                const MachineConstantPool &ConstantPool);
   void convert(ModuleSlotTracker &MST, yaml::MachineJumpTable &YamlJTI,
                const MachineJumpTableInfo &JTI);
   void convertStackObjects(yaml::MachineFunction &YMF,
                            const MachineFunction &MF, ModuleSlotTracker &MST);
   void convertCallSiteObjects(yaml::MachineFunction &YMF,
                               const MachineFunction &MF,
                               ModuleSlotTracker &MST);
+  void convertMachineMetadataNodes(yaml::MachineFunction &YMF,
+                                   const MachineFunction &MF,
+                                   MachineModuleSlotTracker &MST);
 
 private:
   void initRegisterMaskIds(const MachineFunction &MF);
 };
 
 /// This class prints out the machine instructions using the MIR serialization
 /// format.
 class MIPrinter {
void MIRPrinter::print(const MachineFunction &MF) {
   YamlMF.RegBankSelected = MF.getProperties().hasProperty(
       MachineFunctionProperties::Property::RegBankSelected);
   YamlMF.Selected = MF.getProperties().hasProperty(
       MachineFunctionProperties::Property::Selected);
   YamlMF.FailedISel = MF.getProperties().hasProperty(
       MachineFunctionProperties::Property::FailedISel);
 
   convert(YamlMF, MF.getRegInfo(), MF.getSubtarget().getRegisterInfo());
-  ModuleSlotTracker MST(MF.getFunction().getParent());
+  MachineModuleSlotTracker MST(&MF);
   MST.incorporateFunction(MF.getFunction());
   convert(MST, YamlMF.FrameInfo, MF.getFrameInfo());
   convertStackObjects(YamlMF, MF, MST);
   convertCallSiteObjects(YamlMF, MF, MST);
   for (auto &Sub : MF.DebugValueSubstitutions)
     YamlMF.DebugValueSubstitutions.push_back({Sub.first.first, Sub.first.second,
                                               Sub.second.first,
                                               Sub.second.second});
void MIRPrinter::print(const MachineFunction &MF) {
   for (const auto &MBB : MF) {
     if (IsNewlineNeeded)
       StrOS << "\n";
     MIPrinter(StrOS, MST, RegisterMaskIds, StackObjectOperandMapping)
         .print(MBB);
     IsNewlineNeeded = true;
   }
   StrOS.flush();
+  // Convert machine metadata collected during the print of the machine
+  // function.
+  convertMachineMetadataNodes(YamlMF, MF, MST);
+
   yaml::Output Out(OS);
   if (!SimplifyMIR)
       Out.setWriteDefaultValues(true);
   Out << YamlMF;
 }
 
 static void printCustomRegMask(const uint32_t *RegMask, raw_ostream &OS,
                                const TargetRegisterInfo *TRI) {
void MIRPrinter::convertCallSiteObjects(yaml::MachineFunction &YMF,
   llvm::sort(YMF.CallSitesInfo.begin(), YMF.CallSitesInfo.end(),
              [](yaml::CallSiteInfo A, yaml::CallSiteInfo B) {
                if (A.CallLocation.BlockNum == B.CallLocation.BlockNum)
                  return A.CallLocation.Offset < B.CallLocation.Offset;
                return A.CallLocation.BlockNum < B.CallLocation.BlockNum;
              });
 }
 
+void MIRPrinter::convertMachineMetadataNodes(yaml::MachineFunction &YMF,
+                                             const MachineFunction &MF,
+                                             MachineModuleSlotTracker &MST) {
+  MachineModuleSlotTracker::MachineMDNodeListType MDList;
+  MST.collectMachineMDNodes(MDList);
+  for (auto &MD : MDList) {
+    std::string NS;
+    raw_string_ostream StrOS(NS);
+    MD.second->print(StrOS, MST, MF.getFunction().getParent());
+    YMF.MachineMetadataNodes.push_back(StrOS.str());
+  }
+}
+
 void MIRPrinter::convert(yaml::MachineFunction &MF,
                          const MachineConstantPool &ConstantPool) {
   unsigned ID = 0;
   for (const MachineConstantPoolEntry &Constant : ConstantPool.getConstants()) {
     std::string Str;
     raw_string_ostream StrOS(Str);
     if (Constant.isMachineConstantPoolEntry()) {
       Constant.Val.MachineCPVal->print(StrOS);

llvm/lib/CodeGen/MachineFunction.cpp

 #include "llvm/Analysis/EHPersonalities.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineJumpTableInfo.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
+#include "llvm/CodeGen/MachineModuleSlotTracker.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetLowering.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/CodeGen/WasmEHFuncInfo.h"
 #include "llvm/CodeGen/WinEHFuncInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/Module.h"
-#include "llvm/IR/ModuleSlotTracker.h"
 #include "llvm/IR/Value.h"
 #include "llvm/MC/MCContext.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/MC/SectionKind.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/DOTGraphTraits.h"
for (MachineRegisterInfo::livein_iterator
       if (I->second)
         OS << " in " << printReg(I->second, TRI);
       if (std::next(I) != E)
         OS << ", ";
     }
     OS << '\n';
   }
 
-  ModuleSlotTracker MST(getFunction().getParent());
+  MachineModuleSlotTracker MST(this);
   MST.incorporateFunction(getFunction());
   for (const auto &BB : *this) {
     OS << '\n';
     // If we print the whole function, print it at its most verbose level.
     BB.print(OS, MST, Indexes, /*IsStandalone=*/true);
   }
 
   OS << "\n# End machine code for function " << getName() << ".\n\n";

llvm/lib/CodeGen/MachineModuleSlotTracker.cpp

+#include "llvm/CodeGen/MachineModuleSlotTracker.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/CodeGen/MachineModuleInfo.h"
+
+using namespace llvm;
+
+void MachineModuleSlotTracker::processMachineFunctionMetadata(
+    AbstractSlotTrackerStorage *AST, const MachineFunction &MF) {
+  // Create metadata created within the backend.
+  for (const MachineBasicBlock &MBB : MF)
+    for (const MachineInstr &MI : MBB.instrs())
+      for (const MachineMemOperand *MMO : MI.memoperands()) {
+        AAMDNodes AAInfo = MMO->getAAInfo();
+        if (AAInfo.TBAA)
+          AST->createMetadataSlot(AAInfo.TBAA);
+        if (AAInfo.TBAAStruct)
+          AST->createMetadataSlot(AAInfo.TBAAStruct);
+        if (AAInfo.Scope)
+          AST->createMetadataSlot(AAInfo.Scope);
+        if (AAInfo.NoAlias)
+          AST->createMetadataSlot(AAInfo.NoAlias);
+      }
+}
+
+void MachineModuleSlotTracker::processMachineModule(
+    AbstractSlotTrackerStorage *AST, const Module *M,
+    bool ShouldInitializeAllMetadata) {
+  if (ShouldInitializeAllMetadata) {
+    MDNStartSlot = AST->getNextMetadataSlot();
+    for (const Function &F : *M) {
+      if (auto MF = MMI.getMachineFunction(F))
+        processMachineFunctionMetadata(AST, *MF);
+    }
+    MDNEndSlot = AST->getNextMetadataSlot();
+  }
+}
+
+void MachineModuleSlotTracker::processMachineFunction(
+    AbstractSlotTrackerStorage *AST, const Function *F,
+    bool ShouldInitializeAllMetadata) {
+  if (!ShouldInitializeAllMetadata) {
+    MDNStartSlot = AST->getNextMetadataSlot();
+    if (auto MF = MMI.getMachineFunction(*F))
+      processMachineFunctionMetadata(AST, *MF);
+    MDNEndSlot = AST->getNextMetadataSlot();
+  }
+}
+
+void MachineModuleSlotTracker::collectMachineMDNodes(
+    MachineMDNodeListType &L) const {
+  collectMDNodes(L, MDNStartSlot, MDNEndSlot);
+}
+
+MachineModuleSlotTracker::MachineModuleSlotTracker(
+    const MachineModuleInfo &MMI, const Module *M,
+    bool ShouldInitializeAllMetadata)
+    : ModuleSlotTracker(M, ShouldInitializeAllMetadata), MMI(MMI),
+      MDNStartSlot(0), MDNEndSlot(0) {
+  setProcessHook([this](AbstractSlotTrackerStorage *AST, const Module *M,
+                        bool ShouldInitializeAllMetadata) {
+    this->processMachineModule(AST, M, ShouldInitializeAllMetadata);
+  });
+  setProcessHook([this](AbstractSlotTrackerStorage *AST, const Function *F,
+                        bool ShouldInitializeAllMetadata) {
+    this->processMachineFunction(AST, F, ShouldInitializeAllMetadata);
+  });
+}
+
+MachineModuleSlotTracker::MachineModuleSlotTracker(
+    const MachineFunction *MF, bool ShouldInitializeAllMetadata)
+    : MachineModuleSlotTracker(MF->getMMI(), MF->getFunction().getParent(),
+                               ShouldInitializeAllMetadata) {}
+
+MachineModuleSlotTracker::~MachineModuleSlotTracker() = default;

llvm/lib/CodeGen/MachineStableHash.cpp

 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/StableHashing.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/IRPrintingPasses.h"
 #include "llvm/IR/Instructions.h"
-#include "llvm/IR/ModuleSlotTracker.h"
 #include "llvm/MC/MCDwarf.h"
 #include "llvm/Target/TargetIntrinsicInfo.h"
 #include "llvm/Target/TargetMachine.h"
 
 #define DEBUG_TYPE "machine-stable-hash"
 
 using namespace llvm;
 

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalValue.h"
+#include "llvm/IR/MDBuilder.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/Type.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"

 void SelectionDAG::DAGUpdateListener::NodeDeleted(SDNode*, SDNode*) {}
 void SelectionDAG::DAGUpdateListener::NodeUpdated(SDNode*) {}
 void SelectionDAG::DAGUpdateListener::NodeInserted(SDNode *) {}
 
 void SelectionDAG::DAGNodeDeletedListener::anchor() {}
 
 #define DEBUG_TYPE "selectiondag"
 
+static cl::opt<bool> EnableMemCpyScopedNoAlias(
+    "enable-memcpy-scoped-noalias", cl::Hidden, cl::init(true),
+    cl::desc("Enable scoped no-alias support during memcpy lowering"));
+
 static cl::opt<bool> EnableMemCpyDAGOpt("enable-memcpy-dag-opt",
        cl::Hidden, cl::init(true),
        cl::desc("Gang up loads and stores generated by inlining of memcpy"));
 
 static cl::opt<int> MaxLdStGlue("ldstmemcpy-glue-max",
        cl::desc("Number limit for gluing ld/st of memcpy."),
        cl::Hidden, cl::init(0));
 
if (NewAlign > Alignment) {
       Alignment = NewAlign;
     }
   }
 
   // Prepare AAInfo for loads/stores after lowering this memcpy.
   AAMDNodes NewAAInfo = AAInfo;
   NewAAInfo.TBAA = NewAAInfo.TBAAStruct = nullptr;
 
+  AAMDNodes DstAAInfo, SrcAAInfo;
+  DstAAInfo = SrcAAInfo = NewAAInfo;
+  // Generate new scoped AA metadata for this memcpy instance if enabled.
+  if (EnableMemCpyScopedNoAlias) {
+    MDBuilder MDB(*DAG.getContext());
+    MDNode *Domain =
+        MDB.createAnonymousAliasScopeDomain("MemcpyLoweringDomain");
+    MDNode *DstScope = MDB.createAnonymousAliasScope(Domain, "Dst");
+    MDNode *SrcScope = MDB.createAnonymousAliasScope(Domain, "Src");
+    MDNode *DstAliasScope = MDNode::concatenate(
+        NewAAInfo.Scope, MDNode::get(*DAG.getContext(), {DstScope}));
+    MDNode *DstNoAliase = MDNode::concatenate(
+        NewAAInfo.NoAlias, MDNode::get(*DAG.getContext(), {SrcScope}));
+    MDNode *SrcAliasScope = MDNode::concatenate(
+        NewAAInfo.Scope, MDNode::get(*DAG.getContext(), {SrcScope}));
+    MDNode *SrcNoAliase = MDNode::concatenate(
+        NewAAInfo.NoAlias, MDNode::get(*DAG.getContext(), {DstScope}));
+
+    DstAAInfo.Scope = DstAliasScope;
+    DstAAInfo.NoAlias = DstNoAliase;
+    SrcAAInfo.Scope = SrcAliasScope;
+    SrcAAInfo.NoAlias = SrcNoAliase;
+  }
+
   MachineMemOperand::Flags MMOFlags =
       isVol ? MachineMemOperand::MOVolatile : MachineMemOperand::MONone;
   SmallVector<SDValue, 16> OutLoadChains;
   SmallVector<SDValue, 16> OutStoreChains;
   SmallVector<SDValue, 32> OutChains;
   unsigned NumMemOps = MemOps.size();
   uint64_t SrcOff = 0, DstOff = 0;
   for (unsigned i = 0; i != NumMemOps; ++i) {
if (CopyFromConstant &&
         SubSlice.Offset = 0;
         SubSlice.Length = VTSize;
       }
       Value = getMemsetStringVal(VT, dl, DAG, TLI, SubSlice);
       if (Value.getNode()) {
         Store = DAG.getStore(
             Chain, dl, Value,
             DAG.getMemBasePlusOffset(Dst, TypeSize::Fixed(DstOff), dl),
-            DstPtrInfo.getWithOffset(DstOff), Alignment, MMOFlags, NewAAInfo);
+            DstPtrInfo.getWithOffset(DstOff), Alignment, MMOFlags, DstAAInfo);
         OutChains.push_back(Store);
       }
     }
 
     if (!Store.getNode()) {
       // The type might not be legal for the target.  This should only happen
       // if the type is smaller than a legal type, as on PPC, so the right
       // thing to do is generate a LoadExt/StoreTrunc pair.  These simplify
       // to Load/Store if NVT==VT.
       // FIXME does the case above also need this?
       EVT NVT = TLI.getTypeToTransformTo(C, VT);
       assert(NVT.bitsGE(VT));
 
       bool isDereferenceable =
         SrcPtrInfo.getWithOffset(SrcOff).isDereferenceable(VTSize, C, DL);
       MachineMemOperand::Flags SrcMMOFlags = MMOFlags;
       if (isDereferenceable)
         SrcMMOFlags |= MachineMemOperand::MODereferenceable;
 
       Value = DAG.getExtLoad(
           ISD::EXTLOAD, dl, NVT, Chain,
           DAG.getMemBasePlusOffset(Src, TypeSize::Fixed(SrcOff), dl),
           SrcPtrInfo.getWithOffset(SrcOff), VT,
-          commonAlignment(*SrcAlign, SrcOff), SrcMMOFlags, NewAAInfo);
+          commonAlignment(*SrcAlign, SrcOff), SrcMMOFlags, SrcAAInfo);
       OutLoadChains.push_back(Value.getValue(1));
 
       Store = DAG.getTruncStore(
           Chain, dl, Value,
           DAG.getMemBasePlusOffset(Dst, TypeSize::Fixed(DstOff), dl),
-          DstPtrInfo.getWithOffset(DstOff), VT, Alignment, MMOFlags, NewAAInfo);
+          DstPtrInfo.getWithOffset(DstOff), VT, Alignment, MMOFlags, DstAAInfo);
       OutStoreChains.push_back(Store);
     }
     SrcOff += VTSize;
     DstOff += VTSize;
     Size -= VTSize;
   }
 
   unsigned GluedLdStLimit = MaxLdStGlue == 0 ?

llvm/lib/IR/AsmWriter.cpp

if (STy->getNumElements() == 0) {
     }
 
     OS << " }";
   }
   if (STy->isPacked())
     OS << '>';
 }
 
+AbstractSlotTrackerStorage::~AbstractSlotTrackerStorage() {}
+
 namespace llvm {
 
 //===----------------------------------------------------------------------===//
 // SlotTracker Class: Enumerate slot numbers for unnamed values
 //===----------------------------------------------------------------------===//
 /// This class provides computation of slot numbers for LLVM Assembly writing.
 ///
-class SlotTracker {
+class SlotTracker : public AbstractSlotTrackerStorage {
 public:
   /// ValueMap - A mapping of Values to slot numbers.
   using ValueMap = DenseMap<const Value *, unsigned>;
 
 private:
   /// TheModule - The module for which we are holding slot numbers.
   const Module* TheModule;
 
   /// TheFunction - The function for which we are holding slot numbers.
   const Function* TheFunction = nullptr;
   bool FunctionProcessed = false;
   bool ShouldInitializeAllMetadata;
 
+  std::function<void(AbstractSlotTrackerStorage *, const Module *, bool)>
+      processModuleHookFn;
+  std::function<void(AbstractSlotTrackerStorage *, const Function *, bool)>
+      processFunctionHookFn;
+
   /// The summary index for which we are holding slot numbers.
   const ModuleSummaryIndex *TheIndex = nullptr;
 
   /// mMap - The slot map for the module level data.
   ValueMap mMap;
   unsigned mNext = 0;
 
   /// fMap - The slot map for the function level data.
explicit SlotTracker(const Function *F,
                        bool ShouldInitializeAllMetadata = false);
 
   /// Construct from a module summary index.
   explicit SlotTracker(const ModuleSummaryIndex *Index);
 
   SlotTracker(const SlotTracker &) = delete;
   SlotTracker &operator=(const SlotTracker &) = delete;
 
+  ~SlotTracker() = default;
+
+  void setProcessHook(
+      std::function<void(AbstractSlotTrackerStorage *, const Module *, bool)>);
+  void setProcessHook(std::function<void(AbstractSlotTrackerStorage *,
+                                         const Function *, bool)>);
+
+  unsigned getNextMetadataSlot() override { return mdnNext; }
+
+  void createMetadataSlot(const MDNode *N) override;
+
   /// Return the slot number of the specified value in it's type
   /// plane.  If something is not in the SlotTracker, return -1.
   int getLocalSlot(const Value *V);
   int getGlobalSlot(const GlobalValue *V);
-  int getMetadataSlot(const MDNode *N);
+  int getMetadataSlot(const MDNode *N) override;
   int getAttributeGroupSlot(AttributeSet AS);
   int getModulePathSlot(StringRef Path);
   int getGUIDSlot(GlobalValue::GUID GUID);
   int getTypeIdSlot(StringRef Id);
 
   /// If you'd like to deal with a function instead of just a module, use
   /// this method to get its data into the SlotTracker.
   void incorporateFunction(const Function *F) {

 SlotTracker *ModuleSlotTracker::getMachine() {
   if (!ShouldCreateStorage)
     return Machine;
 
   ShouldCreateStorage = false;
   MachineStorage =
       std::make_unique<SlotTracker>(M, ShouldInitializeAllMetadata);
   Machine = MachineStorage.get();
+  if (processModuleHookFn)
+    Machine->setProcessHook(processModuleHookFn);
+  if (processFunctionHookFn)
+    Machine->setProcessHook(processFunctionHookFn);
   return Machine;
 }
 
 void ModuleSlotTracker::incorporateFunction(const Function &F) {
   // Using getMachine() may lazily create the slot tracker.
   if (!getMachine())
     return;
 
   // Nothing to do if this is the right function already.
   if (this->F == &F)
     return;
   if (this->F)
     Machine->purgeFunction();
   Machine->incorporateFunction(&F);
   this->F = &F;
 }
 
 int ModuleSlotTracker::getLocalSlot(const Value *V) {
   assert(F && "No function incorporated");
   return Machine->getLocalSlot(V);
 }
 
+void ModuleSlotTracker::setProcessHook(
+    std::function<void(AbstractSlotTrackerStorage *, const Module *, bool)>
+        Fn) {
+  processModuleHookFn = Fn;
+}
+
+void ModuleSlotTracker::setProcessHook(
+    std::function<void(AbstractSlotTrackerStorage *, const Function *, bool)>
+        Fn) {
+  processFunctionHookFn = Fn;
+}
+
 static SlotTracker *createSlotTracker(const Value *V) {
   if (const Argument *FA = dyn_cast<Argument>(V))
     return new SlotTracker(FA->getParent());
 
   if (const Instruction *I = dyn_cast<Instruction>(V))
     if (I->getParent())
       return new SlotTracker(I->getParent()->getParent());
 
for (const Function &F : *TheModule) {
 
     // Add all the function attributes to the table.
     // FIXME: Add attributes of other objects?
     AttributeSet FnAttrs = F.getAttributes().getFnAttributes();
     if (FnAttrs.hasAttributes())
       CreateAttributeSetSlot(FnAttrs);
   }
 
+  if (processModuleHookFn)
+    processModuleHookFn(this, TheModule, ShouldInitializeAllMetadata);
+
   ST_DEBUG("end processModule!\n");
 }
 
 // Process the arguments, basic blocks, and instructions  of a function.
 void SlotTracker::processFunction() {
   ST_DEBUG("begin processFunction!\n");
   fNext = 0;
 
for (auto &I : BB) {
         // Add all the call attributes to the table.
         AttributeSet Attrs = Call->getAttributes().getFnAttributes();
         if (Attrs.hasAttributes())
           CreateAttributeSetSlot(Attrs);
       }
     }
   }
 
+  if (processFunctionHookFn)
+    processFunctionHookFn(this, TheFunction, ShouldInitializeAllMetadata);
+
   FunctionProcessed = true;
 
   ST_DEBUG("end processFunction!\n");
 }
 
 // Iterate through all the GUID in the index and create slots for them.
 int SlotTracker::processIndex() {
   ST_DEBUG("begin processIndex!\n");
int SlotTracker::getGlobalSlot(const GlobalValue *V) {
   // Check for uninitialized state and do lazy initialization.
   initializeIfNeeded();
 
   // Find the value in the module map
   ValueMap::iterator MI = mMap.find(V);
   return MI == mMap.end() ? -1 : (int)MI->second;
 }
 
+void SlotTracker::setProcessHook(
+    std::function<void(AbstractSlotTrackerStorage *, const Module *, bool)>
+        Fn) {
+  processModuleHookFn = Fn;
+}
+
+void SlotTracker::setProcessHook(
+    std::function<void(AbstractSlotTrackerStorage *, const Function *, bool)>
+        Fn) {
+  processFunctionHookFn = Fn;
+}
+
+/// getMetadataSlot - Get the slot number of a MDNode.
+void SlotTracker::createMetadataSlot(const MDNode *N) {
+  CreateMetadataSlot(N);
+}
+
 /// getMetadataSlot - Get the slot number of a MDNode.
 int SlotTracker::getMetadataSlot(const MDNode *N) {
   // Check for uninitialized state and do lazy initialization.
   initializeIfNeeded();
 
   // Find the MDNode in the module map
   mdn_iterator MI = mdnMap.find(N);
   return MI == mdnMap.end() ? -1 : (int)MI->second;

 
 void ModuleSummaryIndex::print(raw_ostream &ROS, bool IsForDebug) const {
   SlotTracker SlotTable(this);
   formatted_raw_ostream OS(ROS);
   AssemblyWriter W(OS, SlotTable, this, IsForDebug);
   W.printModuleSummaryIndex();
 }
 
+void ModuleSlotTracker::collectMDNodes(MachineMDNodeListType &L, unsigned LB,
+                                       unsigned UB) const {
+  SlotTracker *ST = MachineStorage.get();
+  if (!ST)
+    return;
+
+  for (auto &I : llvm::make_range(ST->mdn_begin(), ST->mdn_end()))
+    if (I.second >= LB && I.second < UB)
+      L.push_back(std::make_pair(I.second, I.first));
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 // Value::dump - allow easy printing of Values from the debugger.
 LLVM_DUMP_METHOD
 void Value::dump() const { print(dbgs(), /*IsForDebug=*/true); dbgs() << '\n'; }
 
 // Type::dump - allow easy printing of Types from the debugger.
 LLVM_DUMP_METHOD
 void Type::dump() const { print(dbgs(), /*IsForDebug=*/true); dbgs() << '\n'; }

llvm/test/CodeGen/AArch64/arm64-2012-05-07-MemcpyAlignBug.ll

 ; RUN: llc < %s -mtriple=arm64-eabi -mcpu=cyclone | FileCheck %s
 ; <rdar://problem/11294426>
 
 @b = private unnamed_addr constant [3 x i32] [i32 1768775988, i32 1685481784, i32 1836253201], align 4
 
 ; The important thing for this test is that we need an unaligned load of `l_b'
 ; ("ldr w2, [x1, #8]" in this case).
 
 ; CHECK:      adrp x[[PAGE:[0-9]+]], {{l_b@PAGE|.Lb}}
 ; CHECK: add  x[[ADDR:[0-9]+]], x[[PAGE]], {{l_b@PAGEOFF|:lo12:.Lb}}
 ; CHECK-NEXT: ldr  [[VAL2:x[0-9]+]], [x[[ADDR]]]
+; CHECK-NEXT: str  [[VAL2]], [x0]
 ; CHECK-NEXT: ldr  [[VAL:w[0-9]+]], [x[[ADDR]], #8]
 ; CHECK-NEXT: str  [[VAL]], [x0, #8]
-; CHECK-NEXT: str  [[VAL2]], [x0]
 
 define void @foo(i8* %a) {
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 4 %a, i8* align 4 bitcast ([3 x i32]* @b to i8*), i64 12, i1 false)
   ret void
 }
 
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture, i64, i1) nounwind

llvm/test/CodeGen/AArch64/arm64-memcpy-inline.ll

 ; CHECK-DAG: str [[REG2]],
   call void @llvm.memcpy.p0i8.p0i8.i32(i8* align 8 getelementptr inbounds (%struct.x, %struct.x* @dst, i32 0, i32 0), i8* align 8 getelementptr inbounds (%struct.x, %struct.x* @src, i32 0, i32 0), i32 11, i1 false)
   ret i32 0
 }
 
 define void @t1(i8* nocapture %C) nounwind {
 entry:
 ; CHECK-LABEL: t1:
-; CHECK: ldr [[DEST:q[0-9]+]], [x[[BASEREG]]]
-; CHECK: ldur [[DEST:q[0-9]+]], [x[[BASEREG:[0-9]+]], #15]
-; CHECK: stur [[DEST:q[0-9]+]], [x0, #15]
-; CHECK: str [[DEST:q[0-9]+]], [x0]
+; CHECK: ldr [[REG0:q[0-9]+]], [x[[BASEREG:[0-9]+]]]
+; CHECK: str [[REG0]], [x0]
+; CHECK: ldur [[REG1:q[0-9]+]], [x[[BASEREG]], #15]
+; CHECK: stur [[REG1]], [x0, #15]
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %C, i8* getelementptr inbounds ([31 x i8], [31 x i8]* @.str1, i64 0, i64 0), i64 31, i1 false)
   ret void
 }
 
 define void @t2(i8* nocapture %C) nounwind {
 entry:
 ; CHECK-LABEL: t2:
 ; CHECK: mov [[REG3:w[0-9]+]]
 ; CHECK: movk [[REG3]],
 ; CHECK: str [[REG3]], [x0, #32]
 ; CHECK: ldp [[DEST1:q[0-9]+]], [[DEST2:q[0-9]+]], [x{{[0-9]+}}]
 ; CHECK: stp [[DEST1]], [[DEST2]], [x0]
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %C, i8* getelementptr inbounds ([36 x i8], [36 x i8]* @.str2, i64 0, i64 0), i64 36, i1 false)
   ret void
 }
 
 define void @t3(i8* nocapture %C) nounwind {
 entry:
 ; CHECK-LABEL: t3:
-; CHECK: ldr [[DEST:q[0-9]+]], [x[[BASEREG]]]
-; CHECK: ldr [[REG4:x[0-9]+]], [x[[BASEREG:[0-9]+]], #16]
-; CHECK: str [[REG4]], [x0, #16]
+; CHECK: ldr [[DEST:q[0-9]+]], [x[[BASEREG:[0-9]+]]]
 ; CHECK: str [[DEST]], [x0]
+; CHECK: ldr [[REG4:x[0-9]+]], [x[[BASEREG]], #16]
+; CHECK: str [[REG4]], [x0, #16]
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* %C, i8* getelementptr inbounds ([24 x i8], [24 x i8]* @.str3, i64 0, i64 0), i64 24, i1 false)
   ret void
 }
 
 define void @t4(i8* nocapture %C) nounwind {
 entry:
 ; CHECK-LABEL: t4:
 ; CHECK: mov [[REG5:w[0-9]+]], #32

llvm/test/CodeGen/AArch64/arm64-misaligned-memcpy-inline.ll

entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* align 8 %out, i8* align 8 %in, i64 16, i1 false)
   ret void
 }
 
 ; Tiny (4 bytes here) unaligned memcpy() should be inlined with byte sized
 ; loads and stores if strict-alignment is turned on.
 define void @t2(i8* %out, i8* %in) {
 ; CHECK-LABEL: t2:
-; CHECK:      ldrb w{{[0-9]+}}, [x1, #3]
-; CHECK-NEXT: ldrb w{{[0-9]+}}, [x1, #2]
-; CHECK-NEXT: ldrb w{{[0-9]+}}, [x1, #1]
-; CHECK-NEXT: ldrb w{{[0-9]+}}, [x1]
-; CHECK-NEXT: strb w{{[0-9]+}}, [x0, #3]
-; CHECK-NEXT: strb w{{[0-9]+}}, [x0, #2]
-; CHECK-NEXT: strb w{{[0-9]+}}, [x0, #1]
-; CHECK-NEXT: strb w{{[0-9]+}}, [x0]
+; CHECK:      ldrb w[[V0:[0-9]+]], [x1]
+; CHECK-NEXT: ldrb w[[V1:[0-9]+]], [x1, #1]
+; CHECK-NEXT: ldrb w[[V2:[0-9]+]], [x1, #2]
+; CHECK-NEXT: ldrb w[[V3:[0-9]+]], [x1, #3]
+; CHECK-NEXT: strb w[[V0]], [x0]
+; CHECK-NEXT: strb w[[V1]], [x0, #1]
+; CHECK-NEXT: strb w[[V2]], [x0, #2]
+; CHECK-NEXT: strb w[[V3]], [x0, #3]
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* %out, i8* %in, i64 4, i1 false)
   ret void
 }
 
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8* nocapture, i8* nocapture readonly, i64, i1)

llvm/test/CodeGen/AArch64/memcpy-scoped-aa.ll

 ; RUN: llc -mtriple=aarch64-linux-gnu -o - %s | FileCheck %s
 ; RUN: llc -mtriple=aarch64-linux-gnu -stop-after=finalize-isel -o - %s | FileCheck --check-prefix=MIR %s
 
 ; MIR-DAG: ![[DOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"bax"}
 ; MIR-DAG: ![[SCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[DOMAIN]], !"bax: %p"}
 ; MIR-DAG: ![[SCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[DOMAIN]], !"bax: %q"}
 ; MIR-DAG: ![[SET0:[0-9]+]] = !{![[SCOPE0]]}
 ; MIR-DAG: ![[SET1:[0-9]+]] = !{![[SCOPE1]]}
 
 ; MIR-LABEL: name: test_memcpy
-; MIR:      %2:fpr128 = LDRQui %0, 1 :: (load 16 from %ir.p1, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: STRQui killed %2, %0, 0 :: (store 16 into %ir.p0, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR:      %2:fpr128 = LDRQui %0, 1 :: (load 16 from %ir.p1, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: STRQui killed %2, %0, 0 :: (store 16 into %ir.p0, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
 define i32 @test_memcpy(i32* nocapture %p, i32* nocapture readonly %q) {
 ; CHECK-LABEL: test_memcpy:
 ; CHECK-DAG:    ldp [[Q0:w[0-9]+]], [[Q1:w[0-9]+]], [x1]
 ; CHECK-DAG:    ldr [[PVAL:q[0-9]+]], [x0, #16]
 ; CHECK-DAG:    add w8, [[Q0]], [[Q1]]
 ; CHECK:        str [[PVAL]], [x0]
 ; CHECK:        mov w0, w8
 ; CHECK:        ret
   %p0 = bitcast i32* %p to i8*
   %add.ptr = getelementptr inbounds i32, i32* %p, i64 4
   %p1 = bitcast i32* %add.ptr to i8*
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* noundef nonnull align 4 dereferenceable(16) %p0, i8* noundef nonnull align 4 dereferenceable(16) %p1, i64 16, i1 false), !alias.scope !2, !noalias !4
   %v0 = load i32, i32* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32* %q, i64 1
   %v1 = load i32, i32* %q1, align 4, !alias.scope !4, !noalias !2
   %add = add i32 %v0, %v1
   ret i32 %add
 }
 
 ; MIR-LABEL: name: test_memcpy_inline
-; MIR:      %2:fpr128 = LDRQui %0, 1 :: (load 16 from %ir.p1, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: STRQui killed %2, %0, 0 :: (store 16 into %ir.p0, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR:      %2:fpr128 = LDRQui %0, 1 :: (load 16 from %ir.p1, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: STRQui killed %2, %0, 0 :: (store 16 into %ir.p0, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
 define i32 @test_memcpy_inline(i32* nocapture %p, i32* nocapture readonly %q) {
 ; CHECK-LABEL: test_memcpy_inline:
 ; CHECK-DAG:    ldp [[Q0:w[0-9]+]], [[Q1:w[0-9]+]], [x1]
 ; CHECK-DAG:    ldr [[PVAL:q[0-9]+]], [x0, #16]
 ; CHECK-DAG:    add w8, [[Q0]], [[Q1]]
 ; CHECK:        str [[PVAL]], [x0]
 ; CHECK:        mov w0, w8
 ; CHECK:        ret
; CHECK:        ret
   %v0 = load i32, i32* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32* %q, i64 1
   %v1 = load i32, i32* %q1, align 4, !alias.scope !4, !noalias !2
   %add = add i32 %v0, %v1
   ret i32 %add
 }
 
 ; MIR-LABEL: name: test_mempcpy
-; MIR:      %2:fpr128 = LDRQui %0, 1 :: (load 16 from %ir.p1, align 1, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: STRQui killed %2, %0, 0 :: (store 16 into %ir.p0, align 1, !alias.scope ![[SET0]], !noalias ![[SET1]])
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR:      %2:fpr128 = LDRQui %0, 1 :: (load 16 from %ir.p1, align 1, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: STRQui killed %2, %0, 0 :: (store 16 into %ir.p0, align 1, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
 define i32 @test_mempcpy(i32* nocapture %p, i32* nocapture readonly %q) {
 ; CHECK-LABEL: test_mempcpy:
 ; CHECK-DAG:    ldp [[Q0:w[0-9]+]], [[Q1:w[0-9]+]], [x1]
 ; CHECK-DAG:    ldr [[PVAL:q[0-9]+]], [x0, #16]
 ; CHECK-DAG:    add w8, [[Q0]], [[Q1]]
 ; CHECK:        str [[PVAL]], [x0]
 ; CHECK:        mov w0, w8
 ; CHECK:        ret

llvm/test/CodeGen/AMDGPU/memcpy-scoped-aa.ll

 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -o - %s | FileCheck %s
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1010 -stop-after=finalize-isel -o - %s | FileCheck --check-prefix=MIR %s
 
 ; Ensure that the scoped AA is attached on loads/stores lowered from mem ops.
 
 ; Re-evaluate the slot numbers of scopes as that numbering could be changed run-by-run.
 
 ; MIR-DAG: ![[DOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"bax"}
 ; MIR-DAG: ![[SCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[DOMAIN]], !"bax: %p"}
 ; MIR-DAG: ![[SCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[DOMAIN]], !"bax: %q"}
 ; MIR-DAG: ![[SET0:[0-9]+]] = !{![[SCOPE0]]}
 ; MIR-DAG: ![[SET1:[0-9]+]] = !{![[SCOPE1]]}
 
 ; MIR-LABEL: name: test_memcpy
-; MIR: %8:vreg_128 = GLOBAL_LOAD_DWORDX4 %9, 16, 0, implicit $exec :: (load 16 from %ir.p1, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]], addrspace 1)
-; MIR: GLOBAL_STORE_DWORDX4 %10, killed %8, 0, 0, implicit $exec :: (store 16 into %ir.p0, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]], addrspace 1)
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR: %8:vreg_128 = GLOBAL_LOAD_DWORDX4 %9, 16, 0, implicit $exec :: (load 16 from %ir.p1, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]], addrspace 1)
+; MIR: GLOBAL_STORE_DWORDX4 %10, killed %8, 0, 0, implicit $exec :: (store 16 into %ir.p0, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]], addrspace 1)
 define i32 @test_memcpy(i32 addrspace(1)* nocapture %p, i32 addrspace(1)* nocapture readonly %q) {
 ; Check loads of %q are scheduled ahead of that store of the memcpy on %p.
 ; CHECK-LABEL: test_memcpy:
 ; CHECK-DAG:    global_load_dwordx2 v{{\[}}[[Q0:[0-9]+]]:[[Q1:[0-9]+]]{{\]}}, v[2:3], off
 ; CHECK-DAG:    global_load_dwordx4 [[PVAL:v\[[0-9]+:[0-9]+\]]], v[0:1], off offset:16
 ; CHECK-DAG:    v_add_nc_u32_e32 v{{[0-9]+}}, v[[Q0]], v[[Q1]]
 ; CHECK:        global_store_dwordx4 v[0:1], [[PVAL]], off
 ; CHECK:        s_setpc_b64 s[30:31]
   %p0 = bitcast i32 addrspace(1)* %p to i8 addrspace(1)*
   %add.ptr = getelementptr inbounds i32, i32 addrspace(1)* %p, i64 4
   %p1 = bitcast i32 addrspace(1)* %add.ptr to i8 addrspace(1)*
   tail call void @llvm.memcpy.p1i8.p1i8.i64(i8 addrspace(1)* noundef nonnull align 4 dereferenceable(16) %p0, i8 addrspace(1)* noundef nonnull align 4 dereferenceable(16) %p1, i64 16, i1 false), !alias.scope !2, !noalias !4
   %v0 = load i32, i32 addrspace(1)* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32 addrspace(1)* %q, i64 1
   %v1 = load i32, i32 addrspace(1)* %q1, align 4, !alias.scope !4, !noalias !2
   %add = add i32 %v0, %v1
   ret i32 %add
 }
 
 ; MIR-LABEL: name: test_memcpy_inline
-; MIR: %8:vreg_128 = GLOBAL_LOAD_DWORDX4 %9, 16, 0, implicit $exec :: (load 16 from %ir.p1, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]], addrspace 1)
-; MIR: GLOBAL_STORE_DWORDX4 %10, killed %8, 0, 0, implicit $exec :: (store 16 into %ir.p0, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]], addrspace 1)
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR: %8:vreg_128 = GLOBAL_LOAD_DWORDX4 %9, 16, 0, implicit $exec :: (load 16 from %ir.p1, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]], addrspace 1)
+; MIR: GLOBAL_STORE_DWORDX4 %10, killed %8, 0, 0, implicit $exec :: (store 16 into %ir.p0, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]], addrspace 1)
 define i32 @test_memcpy_inline(i32 addrspace(1)* nocapture %p, i32 addrspace(1)* nocapture readonly %q) {
 ; Check loads of %q are scheduled ahead of that store of the memcpy on %p.
 ; CHECK-LABEL: test_memcpy_inline:
 ; CHECK-DAG:    global_load_dwordx2 v{{\[}}[[Q0:[0-9]+]]:[[Q1:[0-9]+]]{{\]}}, v[2:3], off
 ; CHECK-DAG:    global_load_dwordx4 [[PVAL:v\[[0-9]+:[0-9]+\]]], v[0:1], off offset:16
 ; CHECK-DAG:    v_add_nc_u32_e32 v{{[0-9]+}}, v[[Q0]], v[[Q1]]
 ; CHECK:        global_store_dwordx4 v[0:1], [[PVAL]], off
 ; CHECK:        s_setpc_b64 s[30:31]

llvm/test/CodeGen/PowerPC/pr45301.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=powerpc64-- -verify-machineinstrs \
 ; RUN:   -ppc-asm-full-reg-names < %s | FileCheck %s
 %struct.e.0.1.2.3.12.29 = type { [10 x i32] }
 
 define dso_local void @g(%struct.e.0.1.2.3.12.29* %agg.result) local_unnamed_addr #0 {
 ; CHECK-LABEL: g:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    mflr r0
 ; CHECK-NEXT:    std r0, 16(r1)
 ; CHECK-NEXT:    stdu r1, -112(r1)
 ; CHECK-NEXT:    bl i
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    addis r4, r2, g@toc@ha
 ; CHECK-NEXT:    addi r4, r4, g@toc@l
 ; CHECK-NEXT:    ld r5, 0(r4)
-; CHECK-NEXT:    std r5, 0(r3)
-; CHECK-NEXT:    ld r5, 16(r4)
-; CHECK-NEXT:    std r5, 16(r3)
-; CHECK-NEXT:    ld r6, 8(r4)
-; CHECK-NEXT:    std r6, 8(r3)
-; CHECK-NEXT:    ld r6, 24(r4)
-; CHECK-NEXT:    std r6, 24(r3)
-; CHECK-NEXT:    lwz r6, 0(r3)
+; CHECK-NEXT:    ld r6, 16(r4)
+; CHECK-NEXT:    ld r7, 8(r4)
+; CHECK-NEXT:    ld r8, 24(r4)
 ; CHECK-NEXT:    ld r4, 32(r4)
+; CHECK-NEXT:    std r5, 0(r3)
 ; CHECK-NEXT:    std r4, 32(r3)
 ; CHECK-NEXT:    li r4, 20
-; CHECK-NEXT:    stwbrx r6, 0, r3
-; CHECK-NEXT:    stwbrx r5, r3, r4
+; CHECK-NEXT:    lwz r5, 0(r3)
+; CHECK-NEXT:    std r7, 8(r3)
+; CHECK-NEXT:    std r8, 24(r3)
+; CHECK-NEXT:    std r6, 16(r3)
+; CHECK-NEXT:    stwbrx r5, 0, r3
+; CHECK-NEXT:    stwbrx r6, r3, r4
 ; CHECK-NEXT:    addi r1, r1, 112
 ; CHECK-NEXT:    ld r0, 16(r1)
 ; CHECK-NEXT:    mtlr r0
 ; CHECK-NEXT:    blr
 entry:
   %call = tail call signext i32 bitcast (i32 (...)* @i to i32 ()*)()
   %conv = sext i32 %call to i64
   %0 = inttoptr i64 %conv to i8*

llvm/test/CodeGen/X86/memcpy-scoped-aa.ll

 ; RUN: llc -mtriple=x86_64-linux-gnu -stop-after=finalize-isel -o - %s | FileCheck --check-prefix=MIR %s
 
 ; Ensure that the scoped AA is attached on loads/stores lowered from mem ops.
 
 ; Re-evaluate the slot numbers of scopes as that numbering could be changed run-by-run.
 
 ; MIR-DAG: ![[DOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"bax"}
 ; MIR-DAG: ![[SCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[DOMAIN]], !"bax: %p"}
 ; MIR-DAG: ![[SCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[DOMAIN]], !"bax: %q"}
 ; MIR-DAG: ![[SET0:[0-9]+]] = !{![[SCOPE0]]}
 ; MIR-DAG: ![[SET1:[0-9]+]] = !{![[SCOPE1]]}
 
 ; MIR-LABEL: name: test_memcpy
-; MIR:      %2:gr64 = MOV64rm %0, 1, $noreg, 16, $noreg :: (load 8 from %ir.p1, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: %3:gr64 = MOV64rm %0, 1, $noreg, 24, $noreg :: (load 8 from %ir.p1 + 8, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: MOV64mr %0, 1, $noreg, 8, $noreg, killed %3 :: (store 8 into %ir.p0 + 8, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: MOV64mr %0, 1, $noreg, 0, $noreg, killed %2 :: (store 8 into %ir.p0, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR:      %2:gr64 = MOV64rm %0, 1, $noreg, 16, $noreg :: (load 8 from %ir.p1, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: %3:gr64 = MOV64rm %0, 1, $noreg, 24, $noreg :: (load 8 from %ir.p1 + 8, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: MOV64mr %0, 1, $noreg, 8, $noreg, killed %3 :: (store 8 into %ir.p0 + 8, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
+; MIR-NEXT: MOV64mr %0, 1, $noreg, 0, $noreg, killed %2 :: (store 8 into %ir.p0, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
 define i32 @test_memcpy(i32* nocapture %p, i32* nocapture readonly %q) {
   %p0 = bitcast i32* %p to i8*
   %add.ptr = getelementptr inbounds i32, i32* %p, i64 4
   %p1 = bitcast i32* %add.ptr to i8*
   tail call void @llvm.memcpy.p0i8.p0i8.i64(i8* noundef nonnull align 4 dereferenceable(16) %p0, i8* noundef nonnull align 4 dereferenceable(16) %p1, i64 16, i1 false), !alias.scope !2, !noalias !4
   %v0 = load i32, i32* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32* %q, i64 1
   %v1 = load i32, i32* %q1, align 4, !alias.scope !4, !noalias !2
   %add = add i32 %v0, %v1
   ret i32 %add
 }
 
 ; MIR-LABEL: name: test_memcpy_inline
-; MIR:      %2:gr64 = MOV64rm %0, 1, $noreg, 16, $noreg :: (load 8 from %ir.p1, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: %3:gr64 = MOV64rm %0, 1, $noreg, 24, $noreg :: (load 8 from %ir.p1 + 8, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: MOV64mr %0, 1, $noreg, 8, $noreg, killed %3 :: (store 8 into %ir.p0 + 8, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: MOV64mr %0, 1, $noreg, 0, $noreg, killed %2 :: (store 8 into %ir.p0, align 4, !alias.scope ![[SET0]], !noalias ![[SET1]])
+; MIR: machineMetadataNodes:
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR:      %2:gr64 = MOV64rm %0, 1, $noreg, 16, $noreg :: (load 8 from %ir.p1, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: %3:gr64 = MOV64rm %0, 1, $noreg, 24, $noreg :: (load 8 from %ir.p1 + 8, align 4, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: MOV64mr %0, 1, $noreg, 8, $noreg, killed %3 :: (store 8 into %ir.p0 + 8, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
+; MIR-NEXT: MOV64mr %0, 1, $noreg, 0, $noreg, killed %2 :: (store 8 into %ir.p0, align 4, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
 define i32 @test_memcpy_inline(i32* nocapture %p, i32* nocapture readonly %q) {
   %p0 = bitcast i32* %p to i8*
   %add.ptr = getelementptr inbounds i32, i32* %p, i64 4
   %p1 = bitcast i32* %add.ptr to i8*
   tail call void @llvm.memcpy.inline.p0i8.p0i8.i64(i8* noundef nonnull align 4 dereferenceable(16) %p0, i8* noundef nonnull align 4 dereferenceable(16) %p1, i64 16, i1 false), !alias.scope !2, !noalias !4
   %v0 = load i32, i32* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32* %q, i64 1
   %v1 = load i32, i32* %q1, align 4, !alias.scope !4, !noalias !2
define i32 @test_memset(i32* nocapture %p, i32* nocapture readonly %q) {
   %v0 = load i32, i32* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32* %q, i64 1
   %v1 = load i32, i32* %q1, align 4, !alias.scope !4, !noalias !2
   %add = add i32 %v0, %v1
   ret i32 %add
 }
 
 ; MIR-LABEL: name: test_mempcpy
-; MIR:      %2:gr64 = MOV64rm %0, 1, $noreg, 16, $noreg :: (load 8 from %ir.p1, align 1, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: %3:gr64 = MOV64rm %0, 1, $noreg, 24, $noreg :: (load 8 from %ir.p1 + 8, align 1, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: MOV64mr %0, 1, $noreg, 8, $noreg, killed %3 :: (store 8 into %ir.p0 + 8, align 1, !alias.scope ![[SET0]], !noalias ![[SET1]])
-; MIR-NEXT: MOV64mr %0, 1, $noreg, 0, $noreg, killed %2 :: (store 8 into %ir.p0, align 1, !alias.scope ![[SET0]], !noalias ![[SET1]])
+; MIR-DAG: ![[MMDOMAIN:[0-9]+]] = distinct !{!{{[0-9]+}}, !"MemcpyLoweringDomain"}
+; MIR-DAG: ![[MMSCOPE0:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Src"}
+; MIR-DAG: ![[MMSCOPE1:[0-9]+]] = distinct !{!{{[0-9]+}}, ![[MMDOMAIN]], !"Dst"}
+; MIR-DAG: ![[MMSET0:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE0]]}
+; MIR-DAG: ![[MMSET1:[0-9]+]] = !{![[SCOPE0]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET2:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE1]]}
+; MIR-DAG: ![[MMSET3:[0-9]+]] = !{![[SCOPE1]], ![[MMSCOPE0]]}
+; MIR: body:
+; MIR:      %2:gr64 = MOV64rm %0, 1, $noreg, 16, $noreg :: (load 8 from %ir.p1, align 1, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: %3:gr64 = MOV64rm %0, 1, $noreg, 24, $noreg :: (load 8 from %ir.p1 + 8, align 1, !alias.scope ![[MMSET0]], !noalias ![[MMSET2]])
+; MIR-NEXT: MOV64mr %0, 1, $noreg, 8, $noreg, killed %3 :: (store 8 into %ir.p0 + 8, align 1, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
+; MIR-NEXT: MOV64mr %0, 1, $noreg, 0, $noreg, killed %2 :: (store 8 into %ir.p0, align 1, !alias.scope ![[MMSET1]], !noalias ![[MMSET3]])
 define i32 @test_mempcpy(i32* nocapture %p, i32* nocapture readonly %q) {
   %p0 = bitcast i32* %p to i8*
   %add.ptr = getelementptr inbounds i32, i32* %p, i64 4
   %p1 = bitcast i32* %add.ptr to i8*
   %call = tail call i8* @mempcpy(i8* noundef nonnull align 4 dereferenceable(16) %p0, i8* noundef nonnull align 4 dereferenceable(16) %p1, i64 16), !alias.scope !2, !noalias !4
   %v0 = load i32, i32* %q, align 4, !alias.scope !4, !noalias !2
   %q1 = getelementptr inbounds i32, i32* %q, i64 1
   %v1 = load i32, i32* %q1, align 4, !alias.scope !4, !noalias !2