[FunctionAttrs] Infer argmemonly .

[fhahn]

Summary

This patch adds initial argmemonly inference, by checking the underlying
objects of locations returned by MemoryLocation.

I think this should cover most cases, except function calls to other
argmemonly functions.

I'm not sure if there's a reason why we don't infer those yet.

Additional argmemonly can improve codegen in some cases. It also makes
it easier to come up with a C reproducer for 7662d1687b09 (already fixed,
but I'm trying to see if C/C++ fuzzing could help to uncover similar
issues.)

Compile-time impact:
NewPM-O3: +0.01%
NewPM-ReleaseThinLTO: +0.03%
NewPM-ReleaseLTO+g: +0.05%

https://llvm-compile-time-tracker.com/compare.php?from=067c035012fc061ad6378458774ac2df117283c6&to=fe209d4aab5b593bd62d18c0876732ddcca1614d&stat=instructions

[nikic]

Just wondering whether integrating this into checkFunctionMemoryAccess() would make more sense? It already does all the ModRefBehavior queries, so I'd expect it to fit naturally.

[jdoerfert]

FWIW, D50107. Though, I'm happily abandon that one.

[reames]

Working backwards from each memory instruction seems potentially extremely expensive. We could clearly do a forward walk to identify an OK set, then return false if any memory access is in that set. Not sure the complexity is worth it though, have you run compile time measurements with this patch?

The other benefit to the forward walk is that you could process the entire SCC at once, and have speculative membership in that set.

[fhahn]

In D121415#3373870, @nikic wrote:

Just wondering whether integrating this into checkFunctionMemoryAccess() would make more sense? It already does all the ModRefBehavior queries, so I'd expect it to fit naturally.

Sounds good, I first wanted to see if there's any concerns with adding argmemonly in general here. Updated the code to integrate the logic in checkFunctionMemoryAccess.

I also updated the logic to to allow accesses to allocas in argmemonly functions.

In D121415#3375389, @reames wrote:

Working backwards from each memory instruction seems potentially extremely expensive. We could clearly do a forward walk to identify an OK set, then return false if any memory access is in that set. Not sure the complexity is worth it though, have you run compile time measurements with this patch?

The other benefit to the forward walk is that you could process the entire SCC at once, and have speculative membership in that set.

The compile-time impact was low for the original version:

NewPM-O3: +0.01%
NewPM-ReleaseThinLTO: +0.03%
NewPM-ReleaseLTO+g: +0.05%

I now updated the code to integrate the checks in the existing checkFunctionMemoryAccess and the impact is similar (+0.02%, +0.04%, +0.04%). The updated version handles a few additional cases than the original one though. https://llvm-compile-time-tracker.com/compare.php?from=1d4268224bedfcbfefbf2c902c9b6417114a688d&to=0192ddd08f36f6b6e8a847d65a4f03a75eca71e6&stat=instructions

I'm not sure if we have access to memoryssa at this point, but MSSA could probably be used to avoid iterating over all instructions in a function.

[nikic]

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

153

The way the code is structured, can't you put the fallback into an else at the end? It looks like all other branches update it.

246

To avoid enumerating all combinations, may it would be cleaner to construct ModRefInfo and then or it with either FMRL_ArgumentPointees or FMRL_Anywhere?

283

I think you'd want to ReadsMemory |= isRefSet(MR) etc below and then early exit on ReadMemory && WritesMemory && !ArgMemOnly to avoid some duplication here.

[fhahn]

Fix build failure, restructured code as suggeted, thanks!

[fhahn]

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

153

Done, thanks!

246

Yes that's better, updated!

283

Thanks, using isRefSet/isModSet works well!

[nikic]

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

145

getUnderlyingObject() never returns null.

192

Hm, should these checks be going below the pointsToConstantMemory() checks, for the same reason why we don't consider a load from constant memory as a "ref"?

As an aside, pointsToConstantMemory() already does a recursive underlying object walk to check whether the pointer is based on a constant global or alloca, and this adds an additional (weaker, non-recursive) walk. It would probably make sense to convert the underlying pointerToConstantMemory() API into something like classifyLocation() that can return one of Constant, Local, Argument or Unknown, so all of these can be handled in a uniform manner.

292

I think this check if redundant, as we already check the same thing inside the loop.

[fhahn]

Address latest comments.

[fhahn]

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

192

I moved the check below AAR.pointsToConstantMemory( in all cases below.

It would probably make sense to convert the underlying pointerToConstantMemory() API into something like classifyLocation() that can return one of Constant, Local, Argument or Unknown, so all of these can be handled in a uniform manner.

Sounds good. Are you aware of any other places that would benefit from a more general version?

292

Thanks, I replaced it with an assertion.

[fhahn]

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

145

Removed the extra check, thanks!

[nikic]

LGTM

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

192

Nope, not aware of other places.

[fhahn]

Rebase and fix AMDGPU test. I'm planning this after the pre-commit tests come back.

[akdandrea]

Seeing a test-suite failure for benchmark SingleSource/Benchmarks/Linpack/linpack-pc.test with -flto. This is not being seen without LTO.
The failing output I'm seeing is as follows:

...Linux_LTO.220324/test-suite-build-2022-03-24_14-28-44/tools/fpcmp-target: Compared: 5.949578e+05 and 1.600000e+00
abs. diff = 5.949562e+05 rel.diff = 3.718476e+05
Out of tolerance: rel/abs: 1.000000e-04/0.000000e+00

Reverting this commit allows it to pass.

@fhahn let me know if you need more details to reproduce

[fhahn]

In D121415#3408554, @akdandrea wrote:

Seeing a test-suite failure for benchmark SingleSource/Benchmarks/Linpack/linpack-pc.test with -flto. This is not being seen without LTO.
The failing output I'm seeing is as follows:

...Linux_LTO.220324/test-suite-build-2022-03-24_14-28-44/tools/fpcmp-target: Compared: 5.949578e+05 and 1.600000e+00
abs. diff = 5.949562e+05 rel.diff = 3.718476e+05
Out of tolerance: rel/abs: 1.000000e-04/0.000000e+00

Reverting this commit allows it to pass.

@fhahn let me know if you need more details to reproduce

Thanks for the heads-up! I am unable to reproduce this on X86 on my system. Could you share the exact configuration & flags which cause the failure?

[akdandrea]

In D121415#3410352, @fhahn wrote:

In D121415#3408554, @akdandrea wrote:

Seeing a test-suite failure for benchmark SingleSource/Benchmarks/Linpack/linpack-pc.test with -flto. This is not being seen without LTO.
The failing output I'm seeing is as follows:

...Linux_LTO.220324/test-suite-build-2022-03-24_14-28-44/tools/fpcmp-target: Compared: 5.949578e+05 and 1.600000e+00
abs. diff = 5.949562e+05 rel.diff = 3.718476e+05
Out of tolerance: rel/abs: 1.000000e-04/0.000000e+00

Reverting this commit allows it to pass.

@fhahn let me know if you need more details to reproduce

Thanks for the heads-up! I am unable to reproduce this on X86 on my system. Could you share the exact configuration & flags which cause the failure?

Ah that's interesting that you couldn't reproduce it on x86. I did this test on a RHEL 8.4 Power system. Here was how I compiled and ran the test:

clang  -flto    -O3 -DNDEBUG   -w -Werror=date-time -ffp-contract=off -ffp-contract=off -DFMA_DISABLED=1 -ffp-contract=off -DFMA_DISABLED=1 -o linpack-pc.c.o   -c test-suite/SingleSource/Benchmarks/Linpack/linpack-pc.c
clang -flto    -O3 -DNDEBUG      linpack-pc.c.o  -o linpack-pc  -lm
.../Linux_LTO.220324/test-suite-build-2022-03-24_14-28-44/tools/timeit-target --limit-core 0 --limit-cpu 7200 --timeout 7200 --limit-file-size 104857600 --limit-rss-size 838860800 --append-exitstatus --redirect-output linpack-pc.test.out --redirect-input /dev/null --summary linpack-pc.test.time linpack-pc
.../Linux_LTO.220324/test-suite-build-2022-03-24_14-28-44/tools/fpcmp-target -r 0.0001 linpack-pc.test.out .../Linux_LTO.220324/test-suite-build-2022-03-24_14-28-44/SingleSource/Benchmarks/Linpack/linpack-pc.reference_output

[fhahn]

In D121415#3411483, @akdandrea wrote:

Ah that's interesting that you couldn't reproduce it on x86. I did this test on a RHEL 8.4 Power system. Here was how I compiled and ran the test:

clang  -flto    -O3 -DNDEBUG   -w -Werror=date-time -ffp-contract=off -ffp-contract=off -DFMA_DISABLED=1 -ffp-contract=off -DFMA_DISABLED=1 -o linpack-pc.c.o   -c test-suite/SingleSource/Benchmarks/Linpack/linpack-pc.c
clang -flto    -O3 -DNDEBUG      linpack-pc.c.o  -o linpack-pc  -lm

I am unable to build for PowerPC. Could you share linpack-pc.c.o ?

[akdandrea]

In D121415#3411543, @fhahn wrote:

In D121415#3411483, @akdandrea wrote:

Ah that's interesting that you couldn't reproduce it on x86. I did this test on a RHEL 8.4 Power system. Here was how I compiled and ran the test:

clang  -flto    -O3 -DNDEBUG   -w -Werror=date-time -ffp-contract=off -ffp-contract=off -DFMA_DISABLED=1 -ffp-contract=off -DFMA_DISABLED=1 -o linpack-pc.c.o   -c test-suite/SingleSource/Benchmarks/Linpack/linpack-pc.c
clang -flto    -O3 -DNDEBUG      linpack-pc.c.o  -o linpack-pc  -lm

I am unable to build for PowerPC. Could you share linpack-pc.c.o ?

Sure, here it is:

linpack-pc.c.o128 KBDownload

[fhahn]

In D121415#3411555, @akdandrea wrote:

In D121415#3411543, @fhahn wrote:

In D121415#3411483, @akdandrea wrote:

Ah that's interesting that you couldn't reproduce it on x86. I did this test on a RHEL 8.4 Power system. Here was how I compiled and ran the test:

clang  -flto    -O3 -DNDEBUG   -w -Werror=date-time -ffp-contract=off -ffp-contract=off -DFMA_DISABLED=1 -ffp-contract=off -DFMA_DISABLED=1 -o linpack-pc.c.o   -c test-suite/SingleSource/Benchmarks/Linpack/linpack-pc.c
clang -flto    -O3 -DNDEBUG      linpack-pc.c.o  -o linpack-pc  -lm

I am unable to build for PowerPC. Could you share linpack-pc.c.o ?

Sure, here it is:

linpack-pc.c.o128 KBDownload

Looks like the issue is already happening when preparing for LTO. Could you also share the original input IR, before optimizations (passing -emit-llvm -mllvm -disable-llvm-optzns to the first clang invocation)?

[akdandrea]

In D121415#3411562, @fhahn wrote:

In D121415#3411555, @akdandrea wrote:

In D121415#3411543, @fhahn wrote:

In D121415#3411483, @akdandrea wrote:

Ah that's interesting that you couldn't reproduce it on x86. I did this test on a RHEL 8.4 Power system. Here was how I compiled and ran the test:

clang  -flto    -O3 -DNDEBUG   -w -Werror=date-time -ffp-contract=off -ffp-contract=off -DFMA_DISABLED=1 -ffp-contract=off -DFMA_DISABLED=1 -o linpack-pc.c.o   -c test-suite/SingleSource/Benchmarks/Linpack/linpack-pc.c
clang -flto    -O3 -DNDEBUG      linpack-pc.c.o  -o linpack-pc  -lm

I am unable to build for PowerPC. Could you share linpack-pc.c.o ?

Sure, here it is:

linpack-pc.c.o128 KBDownload

Looks like the issue is already happening when preparing for LTO. Could you also share the original input IR, before optimizations (passing -emit-llvm -mllvm -disable-llvm-optzns to the first clang invocation)?

Sure, here is the LLVM IR generated:

linpack-pc.ll168 KBDownload

[fhahn]

In D121415#3412110, @akdandrea wrote:

snip

Looks like the issue is already happening when preparing for LTO. Could you also share the original input IR, before optimizations (passing -emit-llvm -mllvm -disable-llvm-optzns to the first clang invocation)?

Sure, here is the LLVM IR generated:

linpack-pc.ll168 KBDownload

Thanks for sharing the IR. There are a couple of places where we now infer argmemonly, but they seem to be correctly marked as argmemonly, so I suspect some other optimization may be causing the issue. Do you happen to know which function is getting miscompiled?

[akdandrea]

In D121415#3427704, @fhahn wrote:

In D121415#3412110, @akdandrea wrote:

snip

Looks like the issue is already happening when preparing for LTO. Could you also share the original input IR, before optimizations (passing -emit-llvm -mllvm -disable-llvm-optzns to the first clang invocation)?

Sure, here is the LLVM IR generated:

linpack-pc.ll168 KBDownload

Thanks for sharing the IR. There are a couple of places where we now infer argmemonly, but they seem to be correctly marked as argmemonly, so I suspect some other optimization may be causing the issue. Do you happen to know which function is getting miscompiled?

Just to close the loop on this, it appears that I can now get this to pass with LLVM ToT on Power. So, I don't think there is any further investigation required at this time. I appreciate your help on this, thanks again

llvm/lib/Transforms/IPO/FunctionAttrs.cpp

 #include <iterator>
 #include <map>
 #include <vector>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "function-attrs"
 
+STATISTIC(NumArgMemOnly, "Number of functions marked argmemonly");
 STATISTIC(NumReadNone, "Number of functions marked readnone");
 STATISTIC(NumReadOnly, "Number of functions marked readonly");
 STATISTIC(NumWriteOnly, "Number of functions marked writeonly");
 STATISTIC(NumNoCapture, "Number of arguments marked nocapture");
 STATISTIC(NumReturned, "Number of arguments marked returned");
 STATISTIC(NumReadNoneArg, "Number of arguments marked readnone");
 STATISTIC(NumReadOnlyArg, "Number of arguments marked readonly");
 STATISTIC(NumWriteOnlyArg, "Number of arguments marked writeonly");
if (MRB == FMRB_DoesNotAccessMemory)
     return MRB;
 
   if (!ThisBody)
     return MRB;
 
   // Scan the function body for instructions that may read or write memory.
   bool ReadsMemory = false;
   bool WritesMemory = false;
+  // Track if the function accesses memory not based on pointer arguments or
+  // allocas.
+  bool AccessesNonArgsOrAlloca = false;
+  // Returns true if Ptr is not based on a function argument.
+  auto IsArgumentOrAlloca = [](const Value *Ptr) {
+    const Value *UO = getUnderlyingObject(Ptr);
+    return isa<Argument>(UO) || isa<AllocaInst>(UO);
+  };
   for (Instruction &I : instructions(F)) {
     // Some instructions can be ignored even if they read or write memory.
     // Detect these now, skipping to the next instruction if one is found.
     if (auto *Call = dyn_cast<CallBase>(&I)) {
       // Ignore calls to functions in the same SCC, as long as the call sites
       // don't have operand bundles.  Calls with operand bundles are allowed to
       // have memory effects not described by the memory effects of the call
       // target.
       if (!Call->hasOperandBundles() && Call->getCalledFunction() &&
           SCCNodes.count(Call->getCalledFunction()))
         continue;
       FunctionModRefBehavior MRB = AAR.getModRefBehavior(Call);
       ModRefInfo MRI = createModRefInfo(MRB);
 
       // If the call doesn't access memory, we're done.
if (auto *Call = dyn_cast<CallBase>(&I)) {
 
       if (!AliasAnalysis::onlyAccessesArgPointees(MRB)) {
         // The call could access any memory. If that includes writes, note it.
         if (isModSet(MRI))
           WritesMemory = true;
         // If it reads, note it.
         if (isRefSet(MRI))
           ReadsMemory = true;
+        AccessesNonArgsOrAlloca = true;
         continue;
       }
 
       // Check whether all pointer arguments point to local memory, and
       // ignore calls that only access local memory.
       for (const Use &U : Call->args()) {
         const Value *Arg = U;
         if (!Arg->getType()->isPtrOrPtrVectorTy())
           continue;
 
         MemoryLocation Loc =
             MemoryLocation::getBeforeOrAfter(Arg, I.getAAMetadata());
-
         // Skip accesses to local or constant memory as they don't impact the
         // externally visible mod/ref behavior.
         if (AAR.pointsToConstantMemory(Loc, /*OrLocal=*/true))
           continue;
 
+        AccessesNonArgsOrAlloca |= !IsArgumentOrAlloca(Loc.Ptr);
+
         if (isModSet(MRI))
           // Writes non-local memory.
           WritesMemory = true;
         if (isRefSet(MRI))
           // Ok, it reads non-local memory.
           ReadsMemory = true;
       }
       continue;
     } else if (LoadInst *LI = dyn_cast<LoadInst>(&I)) {
-      // Ignore non-volatile loads from local memory. (Atomic is okay here.)
-      if (!LI->isVolatile()) {
-        MemoryLocation Loc = MemoryLocation::get(LI);
-        if (AAR.pointsToConstantMemory(Loc, /*OrLocal=*/true))
-          continue;
-      }
+      // Ignore non-volatile loads from local memory. (Atomic is okay here.)
+      if (!LI->isVolatile() &&
+          AAR.pointsToConstantMemory(Loc, /*OrLocal=*/true))
+      AccessesNonArgsOrAlloca |= !IsArgumentOrAlloca(Loc.Ptr);
     } else if (StoreInst *SI = dyn_cast<StoreInst>(&I)) {
-      // Ignore non-volatile stores to local memory. (Atomic is okay here.)
-      if (!SI->isVolatile()) {
-        MemoryLocation Loc = MemoryLocation::get(SI);
-        if (AAR.pointsToConstantMemory(Loc, /*OrLocal=*/true))
-          continue;
-      }
+      // Ignore non-volatile stores to local memory. (Atomic is okay here.)
+      if (!SI->isVolatile() &&
+          AAR.pointsToConstantMemory(Loc, /*OrLocal=*/true))
+      AccessesNonArgsOrAlloca |= !IsArgumentOrAlloca(Loc.Ptr);
     } else if (VAArgInst *VI = dyn_cast<VAArgInst>(&I)) {
       // Ignore vaargs on local memory.
       MemoryLocation Loc = MemoryLocation::get(VI);
       if (AAR.pointsToConstantMemory(Loc, /*OrLocal=*/true))
         continue;
+      AccessesNonArgsOrAlloca |= !IsArgumentOrAlloca(Loc.Ptr);
+    } else {
+      // If AccessesNonArgsOrAlloca has not been updated above, set it
+      // conservatively.
+      AccessesNonArgsOrAlloca |= I.mayReadOrWriteMemory();
     }
 
     // Any remaining instructions need to be taken seriously!  Check if they
     // read or write memory.
     //
     // Writes memory, remember that.
     WritesMemory |= I.mayWriteToMemory();
 
     // If this instruction may read memory, remember that.
     ReadsMemory |= I.mayReadFromMemory();
   }
 
-  if (WritesMemory) {
-    if (!ReadsMemory)
-      return FMRB_OnlyWritesMemory;
-    else
-      return FMRB_UnknownModRefBehavior;
-  }
+  if (!WritesMemory && !ReadsMemory)
+    return FMRB_DoesNotAccessMemory;
 
-  return ReadsMemory ? FMRB_OnlyReadsMemory : FMRB_DoesNotAccessMemory;
+  FunctionModRefBehavior Result = FunctionModRefBehavior(FMRL_Anywhere);
+  if (!AccessesNonArgsOrAlloca)
+    Result = FunctionModRefBehavior(FMRL_ArgumentPointees);
+  if (WritesMemory)
+    Result = FunctionModRefBehavior(Result | static_cast<int>(ModRefInfo::Mod));
+  if (ReadsMemory)
+    Result = FunctionModRefBehavior(Result | static_cast<int>(ModRefInfo::Ref));
+  return Result;
 }
 
 FunctionModRefBehavior llvm::computeFunctionBodyMemoryAccess(Function &F,
                                                              AAResults &AAR) {
   return checkFunctionMemoryAccess(F, /*ThisBody=*/true, AAR, {});
 }
 
 /// Deduce readonly/readnone/writeonly attributes for the SCC.
 template <typename AARGetterT>
 static void addMemoryAttrs(const SCCNodeSet &SCCNodes, AARGetterT &&AARGetter,
                            SmallSet<Function *, 8> &Changed) {
   // Check if any of the functions in the SCC read or write memory.  If they
   // write memory then they can't be marked readnone or readonly.
   bool ReadsMemory = false;
   bool WritesMemory = false;
+  // Check if all functions only access memory through their arguments.
+  bool ArgMemOnly = true;
   for (Function *F : SCCNodes) {
     // Call the callable parameter to look up AA results for this function.
     AAResults &AAR = AARGetter(*F);
-
     // Non-exact function definitions may not be selected at link time, and an
     // alternative version that writes to memory may be selected.  See the
     // comment on GlobalValue::isDefinitionExact for more details.
     FunctionModRefBehavior FMRB =
         checkFunctionMemoryAccess(*F, F->hasExactDefinition(), AAR, SCCNodes);
-    if (isModAndRefSet(createModRefInfo(FMRB)))
-      return;
     if (FMRB == FMRB_DoesNotAccessMemory)
       continue;
-    ReadsMemory |= AliasAnalysis::onlyReadsMemory(FMRB);
-    WritesMemory |= AliasAnalysis::onlyWritesMemory(FMRB);
-  }
-
-  // If the SCC contains both functions that read and functions that write, then
-  // we cannot add readonly attributes.
-  if (ReadsMemory && WritesMemory)
+    ModRefInfo MR = createModRefInfo(FMRB);
+    ReadsMemory |= isRefSet(MR);
+    WritesMemory |= isModSet(MR);
+    ArgMemOnly &= AliasAnalysis::onlyAccessesArgPointees(FMRB);
+    // Reached neither readnone, readonly, writeonly nor argmemonly can be
+    // inferred. Exit.
+    if (ReadsMemory && WritesMemory && !ArgMemOnly)
       return;
+  }
 
-  // Success!  Functions in this SCC do not access memory, or only read memory.
-  // Give them the appropriate attribute.
+  assert((!ReadsMemory || !WritesMemory || ArgMemOnly) &&
+         "no memory attributes can be added for this SCC, should have exited "
+         "earlier");
+  // Success!  Functions in this SCC do not access memory, only read memory,
+  // only write memory, or only access memory through its arguments. Give them
+  // the appropriate attribute.
 
   for (Function *F : SCCNodes) {
+    // If possible add argmemonly attribute to F, if it accesses memory.
+    if (ArgMemOnly && !F->onlyAccessesArgMemory() &&
+        (ReadsMemory || WritesMemory)) {
+      NumArgMemOnly++;
+      F->addFnAttr(Attribute::ArgMemOnly);
+      Changed.insert(F);
+    }
+
+    // The SCC contains functions both writing and reading from memory. We
+    // cannot add readonly or writeonline attributes.
+    if (ReadsMemory && WritesMemory)
+      continue;
     if (F->doesNotAccessMemory())
       // Already perfect!
       continue;
 
     if (F->onlyReadsMemory() && ReadsMemory)
       // No change.
       continue;
 

llvm/test/Analysis/TypeBasedAliasAnalysis/functionattrs.ll

define i32 @test3_no(i8* %p) nounwind {
   %t = va_arg i8* %p, i32, !tbaa !2
   ret i32 %t
 }
 
 declare void @callee(i32* %p) nounwind
 declare void @llvm.memcpy.p0i8.p0i8.i64(i8*, i8*, i64, i1) nounwind
 
 ; CHECK: attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone willreturn }
-; CHECK: attributes #1 = { mustprogress nofree norecurse nosync nounwind willreturn writeonly }
+; CHECK: attributes #1 = { argmemonly mustprogress nofree norecurse nosync nounwind willreturn writeonly }
 ; CHECK: attributes #2 = { nofree nounwind readonly }
 ; CHECK: attributes #3 = { nounwind }
 ; CHECK: attributes #4 = { mustprogress nofree nosync nounwind readnone willreturn }
-; CHECK: attributes #5 = { mustprogress nofree nosync nounwind willreturn }
-; CHECK: attributes #6 = { mustprogress nofree norecurse nosync nounwind willreturn }
+; CHECK: attributes #5 = { argmemonly mustprogress nofree nosync nounwind willreturn }
+; CHECK: attributes #6 = { argmemonly mustprogress nofree norecurse nosync nounwind willreturn }
 ; CHECK: attributes #7 = { argmemonly nofree nounwind willreturn }
 
 ; Root note.
 !0 = !{ }
 
 ; Invariant memory.
 !1 = !{!3, !3, i64 0, i1 1 }
 ; Not invariant memory.
 !2 = !{!3, !3, i64 0, i1 0 }
 !3 = !{ !"foo", !0 }

llvm/test/CodeGen/AMDGPU/inline-attr.ll

 ; RUN: opt -mtriple=amdgcn--amdhsa -S -O3 -enable-unsafe-fp-math %s  | FileCheck -check-prefix=GCN -check-prefix=UNSAFE %s
 ; RUN: opt -mtriple=amdgcn--amdhsa -S -O3 -enable-no-nans-fp-math %s | FileCheck -check-prefix=GCN -check-prefix=NONANS %s
 ; RUN: opt -mtriple=amdgcn--amdhsa -S -O3 -enable-no-infs-fp-math %s | FileCheck -check-prefix=GCN -check-prefix=NOINFS %s
 
 ; GCN: define float @foo(float %x) local_unnamed_addr #0 {
 ; GCN: define amdgpu_kernel void @caller(float addrspace(1)* nocapture %p) local_unnamed_addr #1 {
 ; GCN: %mul.i = fmul float %load, 1.500000e+01
 
 ; UNSAFE: attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone willreturn "unsafe-fp-math"="true" }
-; UNSAFE: attributes #1 = { mustprogress nofree norecurse nosync nounwind willreturn "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="true" }
+; UNSAFE: attributes #1 = { argmemonly mustprogress nofree norecurse nosync nounwind willreturn "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="true" }
 
 ; NOINFS: attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone willreturn "no-infs-fp-math"="true" }
-; NOINFS: attributes #1 = { mustprogress nofree norecurse nosync nounwind willreturn "less-precise-fpmad"="false" "no-infs-fp-math"="true" "no-nans-fp-math"="false" "unsafe-fp-math"="false" }
+; NOINFS: attributes #1 = { argmemonly mustprogress nofree norecurse nosync nounwind willreturn "less-precise-fpmad"="false" "no-infs-fp-math"="true" "no-nans-fp-math"="false" "unsafe-fp-math"="false" }
 
 ; NONANS: attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone willreturn "no-nans-fp-math"="true" }
-; NONANS: attributes #1 = { mustprogress nofree norecurse nosync nounwind willreturn "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="true" "unsafe-fp-math"="false" }
+; NONANS: attributes #1 = { argmemonly mustprogress nofree norecurse nosync nounwind willreturn "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="true" "unsafe-fp-math"="false" }
 
 define float @foo(float %x) #0 {
 entry:
   %mul = fmul float %x, 1.500000e+01
   ret float %mul
 }
 
 define amdgpu_kernel void @caller(float addrspace(1)* %p) #1 {

llvm/test/Transforms/FunctionAttrs/argmemonly.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --check-attributes
 ; RUN: opt -passes=function-attrs -S %s | FileCheck %s
 
 @g = global i32 20
 
 define void @test_no_read_or_write() {
 ; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind readnone willreturn
 ; CHECK-LABEL: @test_no_read_or_write(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   ret void
 }
 
 define i32 @test_only_read_arg(i32* %ptr) {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind readonly willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn
 ; CHECK-LABEL: @test_only_read_arg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[L:%.*]] = load i32, i32* [[PTR:%.*]], align 4
 ; CHECK-NEXT:    ret i32 [[L]]
 ;
 entry:
   %l = load i32, i32* %ptr
   ret i32 %l

 ;
 entry:
   %l = load i32*, i32** %ptr
   %l.2 = load i32, i32* %l
   ret i32 %l.2
 }
 
 define void @test_only_write_arg(i32* %ptr) {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn writeonly
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind willreturn writeonly
 ; CHECK-LABEL: @test_only_write_arg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    store i32 0, i32* [[PTR:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   store i32 0, i32* %ptr
   ret void

 entry:
   call void @fn_may_access_memory()
   ret void
 }
 
 declare i32 @fn_readnone() readnone
 
 define void @test_call_readnone(i32* %ptr) {
-; CHECK: Function Attrs: writeonly
+; CHECK: Function Attrs: argmemonly writeonly
 ; CHECK-LABEL: @test_call_readnone(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C:%.*]] = call i32 @fn_readnone()
 ; CHECK-NEXT:    store i32 [[C]], i32* [[PTR:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c = call i32 @fn_readnone()

 ; CHECK-NEXT:    ret i32 [[C]]
 ;
 entry:
   %c = call i32 @fn_argmemonly(i32* %ptr)
   ret i32 %c
 }
 
 define i32 @test_call_fn_where_argmemonly_can_be_inferred(i32* %ptr) {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind readonly willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn
 ; CHECK-LABEL: @test_call_fn_where_argmemonly_can_be_inferred(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C:%.*]] = call i32 @test_only_read_arg(i32* [[PTR:%.*]])
 ; CHECK-NEXT:    ret i32 [[C]]
 ;
 entry:
   %c = call i32 @test_only_read_arg(i32* %ptr)
   ret i32 %c
 }
 
 define void @test_memcpy_argonly(i8* %dst, i8* %src) {
-; CHECK: Function Attrs: mustprogress nofree nosync nounwind willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree nosync nounwind willreturn
 ; CHECK-LABEL: @test_memcpy_argonly(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    call void @llvm.memcpy.p0i8.p0i8.i64(i8* [[DST:%.*]], i8* [[SRC:%.*]], i64 32, i1 false)
 ; CHECK-NEXT:    ret void
 ;
 entry:
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* %dst, i8* %src, i64 32, i1 false)
   ret void

 ;
 entry:
   %bc = bitcast [32 x i8]* @arr to i8*
   call void @llvm.memcpy.p0i8.p0i8.i64(i8* %bc, i8* %src, i64 32, i1 false)
   ret void
 }
 
 define i32 @test_read_arg_access_alloca(i32* %ptr) {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind readonly willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn
 ; CHECK-LABEL: @test_read_arg_access_alloca(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[A:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[L:%.*]] = load i32, i32* [[PTR:%.*]], align 4
 ; CHECK-NEXT:    store i32 [[L]], i32* [[A]], align 4
 ; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[A]], align 4
 ; CHECK-NEXT:    ret i32 [[L_2]]
 ;
 entry:
   %a = alloca i32
   %l = load i32, i32* %ptr
   store i32 %l, i32* %a
   %l.2 = load i32, i32* %a
   ret i32 %l.2
 }

llvm/test/Transforms/FunctionAttrs/atomic.ll

 define i32 @test2(i32* %x) uwtable ssp {
 ; CHECK: define i32 @test2(i32* nocapture readonly %x) #1 {
 entry:
   %r = load atomic i32, i32* %x seq_cst, align 4
   ret i32 %r
 }
 
 ; CHECK: attributes #0 = { mustprogress nofree norecurse nosync nounwind readnone ssp willreturn uwtable }
-; CHECK: attributes #1 = { mustprogress nofree norecurse nounwind ssp willreturn uwtable }
+; CHECK: attributes #1 = { argmemonly mustprogress nofree norecurse nounwind ssp willreturn uwtable }

llvm/test/Transforms/FunctionAttrs/nofree.ll

entry:
   %0 = bitcast i32* %a to i8*
   tail call void @free(i8* %0) #2
   ret void
 }
 
 declare void @free(i8* nocapture) local_unnamed_addr #2
 
 define i32 @_Z4foo3Pi(i32* nocapture readonly %a) local_unnamed_addr #3 {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind readonly willreturn uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn uwtable
 ; CHECK-LABEL: @_Z4foo3Pi(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[A:%.*]], align 4
 ; CHECK-NEXT:    ret i32 [[TMP0]]
 ;
 entry:
   %0 = load i32, i32* %a, align 4
   ret i32 %0

llvm/test/Transforms/FunctionAttrs/nosync.ll

 ; CHECK-NEXT:    ret i32 [[A]]
 ;
   %add = add i32 %a, %b
   ret i32 %a
 }
 
 ; negative case - explicit sync
 define void @test5(i8* %p) {
-; CHECK: Function Attrs: mustprogress nofree norecurse nounwind willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nounwind willreturn
 ; CHECK-LABEL: @test5(
 ; CHECK-NEXT:    store atomic i8 0, i8* [[P:%.*]] seq_cst, align 1
 ; CHECK-NEXT:    ret void
 ;
   store atomic i8 0, i8* %p seq_cst, align 1
   ret void
 }
 
 ; negative case - explicit sync
 define i8 @test6(i8* %p) {
-; CHECK: Function Attrs: mustprogress nofree norecurse nounwind willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nounwind willreturn
 ; CHECK-LABEL: @test6(
 ; CHECK-NEXT:    [[V:%.*]] = load atomic i8, i8* [[P:%.*]] seq_cst, align 1
 ; CHECK-NEXT:    ret i8 [[V]]
 ;
   %v = load atomic i8, i8* %p seq_cst, align 1
   ret i8 %v
 }
 

 ; CHECK-NEXT:    ret void
 ;
   fence syncscope("singlethread") seq_cst
   ret void
 }
 
 ; atomic load with monotonic ordering
 define i32 @load_monotonic(i32* nocapture readonly %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: mustprogress nofree norecurse nounwind willreturn uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nounwind willreturn uwtable
 ; CHECK-LABEL: @load_monotonic(
 ; CHECK-NEXT:    [[TMP2:%.*]] = load atomic i32, i32* [[TMP0:%.*]] monotonic, align 4
 ; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
   %2 = load atomic i32, i32* %0 monotonic, align 4
   ret i32 %2
 }
 
 ; atomic store with monotonic ordering.
 define void @store_monotonic(i32* nocapture %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: mustprogress nofree norecurse nounwind willreturn uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nounwind willreturn uwtable
 ; CHECK-LABEL: @store_monotonic(
 ; CHECK-NEXT:    store atomic i32 10, i32* [[TMP0:%.*]] monotonic, align 4
 ; CHECK-NEXT:    ret void
 ;
   store atomic i32 10, i32* %0 monotonic, align 4
   ret void
 }
 
 ; negative, should not deduce nosync
 ; atomic load with acquire ordering.
 define i32 @load_acquire(i32* nocapture readonly %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: mustprogress nofree norecurse nounwind willreturn uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nounwind willreturn uwtable
 ; CHECK-LABEL: @load_acquire(
 ; CHECK-NEXT:    [[TMP2:%.*]] = load atomic i32, i32* [[TMP0:%.*]] acquire, align 4
 ; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
   %2 = load atomic i32, i32* %0 acquire, align 4
   ret i32 %2
 }
 
 define i32 @load_unordered(i32* nocapture readonly %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind readonly willreturn uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind readonly willreturn uwtable
 ; CHECK-LABEL: @load_unordered(
 ; CHECK-NEXT:    [[TMP2:%.*]] = load atomic i32, i32* [[TMP0:%.*]] unordered, align 4
 ; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
   %2 = load atomic i32, i32* %0 unordered, align 4
   ret i32 %2
 }
 
 ; atomic store with unordered ordering.
 define void @store_unordered(i32* nocapture %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn writeonly uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nosync nounwind willreturn writeonly uwtable
 ; CHECK-LABEL: @store_unordered(
 ; CHECK-NEXT:    store atomic i32 10, i32* [[TMP0:%.*]] unordered, align 4
 ; CHECK-NEXT:    ret void
 ;
   store atomic i32 10, i32* %0 unordered, align 4
   ret void
 }
 
 
 ; negative, should not deduce nosync
 ; atomic load with release ordering
 define void @load_release(i32* nocapture %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: nofree norecurse nounwind uwtable
+; CHECK: Function Attrs: argmemonly nofree norecurse nounwind uwtable
 ; CHECK-LABEL: @load_release(
 ; CHECK-NEXT:    store atomic volatile i32 10, i32* [[TMP0:%.*]] release, align 4
 ; CHECK-NEXT:    ret void
 ;
   store atomic volatile i32 10, i32* %0 release, align 4
   ret void
 }
 
 ; negative volatile, relaxed atomic
 define void @load_volatile_release(i32* nocapture %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: nofree norecurse nounwind uwtable
+; CHECK: Function Attrs: argmemonly nofree norecurse nounwind uwtable
 ; CHECK-LABEL: @load_volatile_release(
 ; CHECK-NEXT:    store atomic volatile i32 10, i32* [[TMP0:%.*]] release, align 4
 ; CHECK-NEXT:    ret void
 ;
   store atomic volatile i32 10, i32* %0 release, align 4
   ret void
 }
 
 ; volatile store.
 define void @volatile_store(i32* %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: nofree norecurse nounwind uwtable
+; CHECK: Function Attrs: argmemonly nofree norecurse nounwind uwtable
 ; CHECK-LABEL: @volatile_store(
 ; CHECK-NEXT:    store volatile i32 14, i32* [[TMP0:%.*]], align 4
 ; CHECK-NEXT:    ret void
 ;
   store volatile i32 14, i32* %0, align 4
   ret void
 }
 
 ; negative, should not deduce nosync
 ; volatile load.
 define i32 @volatile_load(i32* %0) norecurse nounwind uwtable {
-; CHECK: Function Attrs: mustprogress nofree norecurse nounwind willreturn uwtable
+; CHECK: Function Attrs: argmemonly mustprogress nofree norecurse nounwind willreturn uwtable
 ; CHECK-LABEL: @volatile_load(
 ; CHECK-NEXT:    [[TMP2:%.*]] = load volatile i32, i32* [[TMP0:%.*]], align 4
 ; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
   %2 = load volatile i32, i32* %0, align 4
   ret i32 %2
 }
 
;
   ret void
 }
 
 declare void @llvm.memcpy(i8* %dest, i8* %src, i32 %len, i1 %isvolatile)
 declare void @llvm.memset(i8* %dest, i8 %val, i32 %len, i1 %isvolatile)
 
 ; negative, checking volatile intrinsics.
 define i32 @memcpy_volatile(i8* %ptr1, i8* %ptr2) {
-; CHECK: Function Attrs: mustprogress nofree nounwind willreturn
+; CHECK: Function Attrs: argmemonly mustprogress nofree nounwind willreturn
 ; CHECK-LABEL: @memcpy_volatile(
 ; CHECK-NEXT:    call void @llvm.memcpy.p0i8.p0i8.i32(i8* [[PTR1:%.*]], i8* [[PTR2:%.*]], i32 8, i1 true)
 ; CHECK-NEXT:    ret i32 4
 ;
   call void @llvm.memcpy(i8* %ptr1, i8* %ptr2, i32 8, i1 1)
   ret i32 4
 }
 
 ; positive, non-volatile intrinsic.
 define i32 @memset_non_volatile(i8* %ptr1, i8 %val) {
-; CHECK: Function Attrs: mustprogress nofree nosync nounwind willreturn writeonly
+; CHECK: Function Attrs: argmemonly mustprogress nofree nosync nounwind willreturn writeonly
 ; CHECK-LABEL: @memset_non_volatile(
 ; CHECK-NEXT:    call void @llvm.memset.p0i8.i32(i8* [[PTR1:%.*]], i8 [[VAL:%.*]], i32 8, i1 false)
 ; CHECK-NEXT:    ret i32 4
 ;
   call void @llvm.memset(i8* %ptr1, i8 %val, i32 8, i1 0)
   ret i32 4
 }
 

llvm/test/Transforms/FunctionAttrs/stats.ll

 }
 
 define void @test_write_global() {
 entry:
   store i32 0, i32* @g
   ret void
 }
 
-; CHECK:      1 function-attrs - Number of arguments marked nocapture
+; CHECK:      1 function-attrs - Number of functions marked argmemonly
+; CHECK-NEXT: 1 function-attrs - Number of arguments marked nocapture
 ; CHECK-NEXT: 1 function-attrs - Number of functions marked as nofree
 ; CHECK-NEXT: 2 function-attrs - Number of functions marked as norecurse
 ; CHECK-NEXT: 2 function-attrs - Number of functions marked as nosync
 ; CHECK-NEXT: 2 function-attrs - Number of functions marked as nounwind
 ; CHECK-NEXT: 1 function-attrs - Number of functions marked readonly
 ; CHECK-NEXT: 1 function-attrs - Number of arguments marked readonly
 ; CHECK-NEXT: 2 function-attrs - Number of functions marked as willreturn
 ; CHECK-NEXT: 1 function-attrs - Number of functions marked writeonly