Allow creating llvm::Function in non-zero address spaces

[arichardson]

Summary

Most users won't have to worry about this as all of the
'getOrInsertFunction' functions on Module will default to the program
address space.

An overload has been added to Function::Create to abstract away the
details for most callers.

This is based on https://reviews.llvm.org/D37054 but without the changes to
make passing a Module to Function::Create() mandatory. I have also added
some more tests and fixed the LLParser to accept call instructions for
types in the program address space.

[arichardson]

Ping?

It would be really good if something like this could land soon as I'm currently in the process of doing a merge with upstream LLVM in our fork.
I just ran into the problem that always_inline wasn't working at -O0 for us due to functions being in AS0 but the calls in AS200 and the alwaysInliner only handles direct calls without an addressspacecast. This would be fixed if we could have functions in AS200.

[arichardson]

Ping? I've applied this to our fork and it has improved inlining and helps us avoid bugs due to using the wrong address space. I would very much like to have this upstream.

[bjope]

lib/AsmParser/LLParser.cpp

1337

This only works if we already have parsed the DataLayout string.

If we need to rely on having parsed the DataLayout string (or even relying on that there is one in the LL file), why should we then need to decorate functions with addrspace attributes instead of simply using the ProgramAddressSpace from the DataLayout?

Wasn't the usage of addrspace attributes on functions a way to avoid the dependecy to a potentially not yet parsed DataLayout inside LLParser (see RFC discussion here http://lists.llvm.org/pipermail/llvm-dev/2017-July/115388.html).

I think there are at least three alternatives:

1. Make the DataLayout string with 'P' mandatory (early in LL file) when using non-zero program addrspace. Then maybe the simplest thing is to use M->getDataLayout().getProgramAddressSpace() in ParseFunctionHeader (line 5064) and skip the explicit addrspace attributes on function define/declare.

2. Skip verifying (during parsing) that the addrspace matches for forward declaration of a function (at least when originating from a call). The define/declare then needs to replace/fixup the Function to use the correct address space. Although I'm not sure exactly how to do this.

3. Make it possible to indicate addrspace of a function in the call statement. I guess we would need to come up with a new syntax for that.

I'm a little worried that the need to add addrspace attributes on function declarations will make it a bit more cumbersome to write test cases. That is ofcourse not a major problem. Alternative 3 will make it even more complicated. Alternative 1 is probably the simplest, and even if it does not address everything from the RFC, at least it will move us forward a little bit.

[bjope]

lib/AsmParser/LLParser.cpp

1337

We can add at the following as a fourth altenative (as a variant of 1):

4. Make the DataLayout string with 'P' mandatory (early in LL file or on the command line) when using non-zero program addrspace (at least until we find a better solution). But also require that addrspace is explicitly set on function definition/declarations. I think that makes alternative 4 what we currently have in this patch.

[bjope]

lib/AsmParser/LLParser.cpp

1321

My comments in checkValidVariableType are related to a test case that looked something like this:

; RUN: llc -o /dev/null %s

define void @main() {
entry:
  %0 = tail call i32 @llvm.phx.aload(i16* null)
  ...
}

declare i32 @llvm.phx.aload(i16* nocapture) addrspace(40)

And when creating the fwd ref for @llvm.phx.aload.i32.p21i16 when parsing the call it will be given the default addrspace zero. As far as I can see that will happen here even if we have specified "Px" in the datalayout string. So I'm getting errors when parsing the declare since it will find the function from the forward reference having a different address space.

If I understand correctly this special handling of IsCall in checkValidVariableType would have accepted the missing addrspace notation in the call if it wasn't a fwd ref (e.g. if I had declared @llvm.phx.aload.i32.p21i16 before @main). But with the order above we create the function with addrspace zero here, and then when we get to the declare we get into trouble.

So even with alterntative (1) and (4) from the discussion in checkValidVariableType I think we would need to use the ProgramAddressSpace here when creating the Function. Or am I missing something in my analysis?

[arichardson]

lib/AsmParser/LLParser.cpp

1321

Yes, that's exactly the issue that I ran into after applying the patch to our fork. I've currently added a horrible hack to always allow address space 200 declarations, but I'm trying to find a better solution.

I am tempted to use option 2 since we were previously using an addrspacecast in every function call and that broke stuff like always_inline at -O0. A new syntax for the addressspace could probably avoid that but I'm not sure if that is the best solution.

I wonder if just setting the address space for foward declaration address space to -1 and fixing it up once we have the data layout would work. I'll try this approach and see if it works.

[arichardson]

Chose option 3) to handle forward-declarations.
I tried updating the address space later but that caused lots of problems.
Instead the call/invoke instruction can now accept an optional addrspace(N)
qualifier to indicate the address space of the called function.

[arsenm]

include/llvm/IR/Function.h

155–156

Creating this looks like an entirely unrelated change that should be a separate patch

[bjope]

test/Bitcode/function-address-space-fwd-decl.ll

6

Please call this @bar (or something) instead of @llvm.phx.aload.
(I should probably have done that in the example I gave you earlier, because I prefer if this doesn't match with the name of an intrinsic in our out-of-tree target).

[arichardson]

Remove unrelated change and don't use llvm.phx.aload()

[bjope]

This seems to work quite nice for me now! Nice!

I added a few inline nit comments. Apart from that it looks good to me.
Although, I don't know much about the bitcode reader/writer. Anyone else who can help out reviewing that part?

PS. I still have a few worries about how well this plays together with instrprof, we have had some hacks in that area earlier. Methods like appendToGlobalArray (in ModuleUtils.cpp) still uses PointerType::getUnqual(FnTy) to strip off the addrspace from function pointers (and afaik the size of a pointer can be different in different adress spaces).
Although, I don't think we can solve all problems right away. This patch at least gives us the power to set addrspace on functions. And that should help out if I for example want to reproduce instrprof problems on trunk (using an "experimental target" with non-zero program addrspace). So I think we can leave that for later.

lib/AsmParser/LLParser.cpp

1333

nit: End code comment with a punctuation.

1521

Fwiw, I created a ParseOptionalProgramAddrSpace method when adapting this to our out-of-tree target. And then I use that method when parsing the optional addrspace string when we know that it is a program address space (in define/declare/call/invoke).
One still needs to be explicit with addrspace when dealing with function pointers (in casts etc), but that is how we have been doing it in the past. And the use of datalayout ofcourse means that I need to make sure the datalayout is specified early in the ll-file.
Anyway, the above gave me the possibility to override the default value 0 (e.g. by using M->getDataLayout().getProgramAddressSpace()) for our target. We will at least use that as a temporary solution until we have adapted all our frontends (and test cases) to use the new syntax.

lib/IR/Function.cpp

234

nit: punctuation

[arichardson]

In D47541#1158529, @bjope wrote:

This seems to work quite nice for me now! Nice!

I added a few inline nit comments. Apart from that it looks good to me.
Although, I don't know much about the bitcode reader/writer. Anyone else who can help out reviewing that part?

PS. I still have a few worries about how well this plays together with instrprof, we have had some hacks in that area earlier. Methods like appendToGlobalArray (in ModuleUtils.cpp) still uses PointerType::getUnqual(FnTy) to strip off the addrspace from function pointers (and afaik the size of a pointer can be different in different adress spaces).
Although, I don't think we can solve all problems right away. This patch at least gives us the power to set addrspace on functions. And that should help out if I for example want to reproduce instrprof problems on trunk (using an "experimental target" with non-zero program addrspace). So I think we can leave that for later.

In our fork I've deleted all uses of PointerType::getUnqual() from clang, and removed those in LLVM that have caused issues. I'm hoping to slowly upstream those patches and eventually remove PointerType::getUnqual().
I agree that those problems should be addressed by a later patch.

lib/AsmParser/LLParser.cpp

1521

Thanks, that's a great suggestion! It should allow us to keep our existing tests without adding addrspace(200) to every call instr. I'll update the patch to use that instead.

[arichardson]

Add parseOptionalProgrammAddrSpace() as suggested
Update tests

[bjope]

LGTM!

As mentioned earlier there still are some PointerType::getUnqual calls (mainly related to sanitizers and profiling?) that might need attention in the future. I don't see that as a stopper for this patch, specially since this shouldn't impact targets where function address space is zero (except that there will be an extra field for function address space in the bitcode). And I think it will be easier to do future fixes related to program address space when we have this basic framework in place.

docs/LangRef.rst

 optional :ref:`linkage type <linkage>`, an optional :ref:`runtime preemption
 specifier <runtime_preemption_model>`,  an optional :ref:`visibility
 style <visibility>`, an optional :ref:`DLL storage class <dllstorageclass>`,
 an optional :ref:`calling convention <callingconv>`,
 an optional ``unnamed_addr`` attribute, a return type, an optional
 :ref:`parameter attribute <paramattrs>` for the return type, a function
 name, a (possibly empty) argument list (each with optional :ref:`parameter
 attributes <paramattrs>`), optional :ref:`function attributes <fnattrs>`,
-an optional section, an optional alignment,
+an optional address space, an optional section, an optional alignment,
 an optional :ref:`comdat <langref_comdats>`,
 an optional :ref:`garbage collector name <gc>`, an optional :ref:`prefix <prefixdata>`,
 an optional :ref:`prologue <prologuedata>`,
 an optional :ref:`personality <personalityfn>`,
 an optional list of attached :ref:`metadata <metadata>`,
 an opening curly brace, a list of basic blocks, and a closing curly brace.
 
 LLVM function declarations consist of the "``declare``" keyword, an
 optional :ref:`linkage type <linkage>`, an optional :ref:`visibility style
 <visibility>`, an optional :ref:`DLL storage class <dllstorageclass>`, an
 optional :ref:`calling convention <callingconv>`, an optional ``unnamed_addr``
-or ``local_unnamed_addr`` attribute, a return type, an optional :ref:`parameter
-attribute <paramattrs>` for the return type, a function name, a possibly
+or ``local_unnamed_addr`` attribute, an optional address space, a return type,
+an optional :ref:`parameter attribute <paramattrs>` for the return type, a function name, a possibly
 empty list of arguments, an optional alignment, an optional :ref:`garbage
 collector name <gc>`, an optional :ref:`prefix <prefixdata>`, and an optional
 :ref:`prologue <prologuedata>`.
 
 A function definition contains a list of basic blocks, forming the CFG (Control
 Flow Graph) for the function. Each basic block may optionally start with a label
 (giving the basic block a symbol table entry), contains a list of instructions,
 and ends with a :ref:`terminator <terminators>` instruction (such as a branch or

 alignment. All alignments must be a power of 2.
 
 If the ``unnamed_addr`` attribute is given, the address is known to not
 be significant and two identical functions can be merged.
 
 If the ``local_unnamed_addr`` attribute is given, the address is known to
 not be significant within the module.
 
+If an explicit address space is not given, it will default to zero.
+
 Syntax::
 
     define [linkage] [PreemptionSpecifier] [visibility] [DLLStorageClass]
            [cconv] [ret attrs]
            <ResultType> @<FunctionName> ([argument list])
-           [(unnamed_addr|local_unnamed_addr)] [fn Attrs] [section "name"]
-           [comdat [($name)]] [align N] [gc] [prefix Constant]
+           [(unnamed_addr|local_unnamed_addr)] [AddrSpace] [fn Attrs]
+           [section "name"] [comdat [($name)]] [align N] [gc] [prefix Constant]
            [prologue Constant] [personality Constant] (!name !N)* { ... }
 
 The argument list is a comma separated sequence of arguments where each
 argument is of the following form:
 
 Syntax::
 
    <type> [parameter Attrs] [name]

 '``invoke``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <result> = invoke [cconv] [ret attrs] <ty>|<fnty> <fnptrval>(<function args>) [fn attrs]
+      <result> = invoke [cconv] [ret attrs] [addrspace(<num>)] [<ty>|<fnty> <fnptrval>(<function args>) [fn attrs]
                     [operand bundles] to label <normal label> unwind label <exception label>
 
 Overview:
 """""""""
 
 The '``invoke``' instruction causes control to transfer to a specified
 function, with the possibility of control flow transfer to either the
 '``normal``' label or the '``exception``' label. If the callee function

 This instruction requires several arguments:
 
 #. The optional "cconv" marker indicates which :ref:`calling
    convention <callingconv>` the call should use. If none is
    specified, the call defaults to using C calling conventions.
 #. The optional :ref:`Parameter Attributes <paramattrs>` list for return
    values. Only '``zeroext``', '``signext``', and '``inreg``' attributes
    are valid here.
+#. The optional addrspace attribute can be used to indicate the adress space
+   of the called function. If it is not specified, address space 0 will be used.
 #. '``ty``': the type of the call instruction itself which is also the
    type of the return value. Functions that return no value are marked
    ``void``.
 #. '``fnty``': shall be the signature of the function being invoked. The
    argument types must match the types implied by this signature. This
    type can be omitted if the function is not varargs.
 #. '``fnptrval``': An LLVM value containing a pointer to a function to
    be invoked. In most cases, this is a direct function invocation, but

 '``call``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <result> = [tail | musttail | notail ] call [fast-math flags] [cconv] [ret attrs] <ty>|<fnty> <fnptrval>(<function args>) [fn attrs]
-                   [ operand bundles ]
+      <result> = [tail | musttail | notail ] call [fast-math flags] [cconv] [ret attrs] [addrspace(<num>)]
+                 [<ty>|<fnty> <fnptrval>(<function args>) [fn attrs] [ operand bundles ]
 
 Overview:
 """""""""
 
 The '``call``' instruction represents a simple function call.
 
 Arguments:
 """"""""""

 #. The optional "cconv" marker indicates which :ref:`calling
    convention <callingconv>` the call should use. If none is
    specified, the call defaults to using C calling conventions. The
    calling convention of the call must match the calling convention of
    the target function, or else the behavior is undefined.
 #. The optional :ref:`Parameter Attributes <paramattrs>` list for return
    values. Only '``zeroext``', '``signext``', and '``inreg``' attributes
    are valid here.
+#. The optional addrspace attribute can be used to indicate the adress space
+   of the called function. If it is not specified, address space 0 will be used.
 #. '``ty``': the type of the call instruction itself which is also the
    type of the return value. Functions that return no value are marked
    ``void``.
 #. '``fnty``': shall be the signature of the function being called. The
    argument types must match the types implied by this signature. This
    type can be omitted if the function is not varargs.
 #. '``fnptrval``': An LLVM value containing a pointer to a function to
    be called. In most cases, this is a direct function call, but

include/llvm/IR/Function.h

private:
   void BuildLazyArguments() const;
 
   void clearArguments();
 
   /// Function ctor - If the (optional) Module argument is specified, the
   /// function is automatically inserted into the end of the function list for
   /// the module.
   ///
-  Function(FunctionType *Ty, LinkageTypes Linkage,
+  Function(FunctionType *Ty, LinkageTypes Linkage, unsigned AddrSpace,
            const Twine &N = "", Module *M = nullptr);
 
 public:
   Function(const Function&) = delete;
   void operator=(const Function&) = delete;
   ~Function();
 
   // This is here to help easily convert from FunctionT * (Function * or
   // MachineFunction *) in BlockFrequencyInfoImpl to Function * by calling
   // FunctionT->getFunction().
   const Function &getFunction() const { return *this; }
 
   static Function *Create(FunctionType *Ty, LinkageTypes Linkage,
+                          unsigned AddrSpace, const Twine &N = "",
+                          Module *M = nullptr) {
+    return new Function(Ty, Linkage, AddrSpace, N, M);
+  }
+
+  // TODO: remove this once all users have been updated to pass an AddrSpace
+  static Function *Create(FunctionType *Ty, LinkageTypes Linkage,
                           const Twine &N = "", Module *M = nullptr) {
-    return new Function(Ty, Linkage, N, M);
+    return new Function(Ty, Linkage, static_cast<unsigned>(-1), N, M);
   }
 
+  /// Creates a new function and attaches it to a module.
+  ///
+  /// Places the function in the program address space as specified
+  /// by the module's data layout.
+  static Function *Create(FunctionType *Ty, LinkageTypes Linkage,
+                          const Twine &N, Module &M);
+
+  static Function *CreateBefore(Function &InsertBefore, FunctionType *Ty,
+                                LinkageTypes Linkage, const Twine &N = "");
+
   // Provide fast operand accessors.
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   /// Returns the number of non-debug IR instructions in this function.
   /// This is equivalent to the sum of the sizes of each basic block contained
   /// within this function.
   unsigned getInstructionCount();
 

include/llvm/IR/GlobalValue.h

enum ThreadLocalMode {
     LocalDynamicTLSModel,
     InitialExecTLSModel,
     LocalExecTLSModel
   };
 
   GlobalValue(const GlobalValue &) = delete;
 
   unsigned getAlignment() const;
+  unsigned getAddressSpace() const;
 
   enum class UnnamedAddr {
     None,
     Local,
     Global,
   };
 
   bool hasGlobalUnnamedAddr() const {

lib/AsmParser/LLParser.h

private:
 
     /// Restore the internal name and slot mappings using the mappings that
     /// were created at an earlier parsing stage.
     void restoreParsingState(const SlotMapping *Slots);
 
     /// GetGlobalVal - Get a value with the specified name or ID, creating a
     /// forward reference record if needed.  This can return null if the value
     /// exists but does not have the right type.
-    GlobalValue *GetGlobalVal(const std::string &N, Type *Ty, LocTy Loc);
-    GlobalValue *GetGlobalVal(unsigned ID, Type *Ty, LocTy Loc);
+    GlobalValue *GetGlobalVal(const std::string &N, Type *Ty, LocTy Loc,
+                              bool IsCall);
+    GlobalValue *GetGlobalVal(unsigned ID, Type *Ty, LocTy Loc, bool IsCall);
 
     /// Get a Comdat with the specified name, creating a forward reference
     /// record if needed.
     Comdat *getComdat(const std::string &N, LocTy Loc);
 
     // Helper Routines.
     bool ParseToken(lltok::Kind T, const char *ErrMsg);
     bool EatIfPresent(lltok::Kind T) {
public:
       BasicBlock *DefineBB(const std::string &Name, LocTy Loc);
 
       bool resolveForwardRefBlockAddresses();
     };
 
     bool ConvertValIDToValue(Type *Ty, ValID &ID, Value *&V,
                              PerFunctionState *PFS, bool IsCall);
 
+    Value *checkValidVariableType(LocTy Loc, const Twine &Name, Type *Ty,
+                                  Value *Val, bool IsCall);
+
     bool parseConstantValue(Type *Ty, Constant *&C);
     bool ParseValue(Type *Ty, Value *&V, PerFunctionState *PFS);
     bool ParseValue(Type *Ty, Value *&V, PerFunctionState &PFS) {
       return ParseValue(Ty, V, &PFS);
     }
 
     bool ParseValue(Type *Ty, Value *&V, LocTy &Loc,
                     PerFunctionState &PFS) {

lib/AsmParser/LLParser.cpp

 
 //===----------------------------------------------------------------------===//
 // GlobalValue Reference/Resolution Routines.
 //===----------------------------------------------------------------------===//
 
 static inline GlobalValue *createGlobalFwdRef(Module *M, PointerType *PTy,
                                               const std::string &Name) {
   if (auto *FT = dyn_cast<FunctionType>(PTy->getElementType()))
-    return Function::Create(FT, GlobalValue::ExternalWeakLinkage, Name, M);
+    return Function::Create(FT, GlobalValue::ExternalWeakLinkage,
+                            PTy->getAddressSpace(), Name, M);
   else
     return new GlobalVariable(*M, PTy->getElementType(), false,
                               GlobalValue::ExternalWeakLinkage, nullptr, Name,
                               nullptr, GlobalVariable::NotThreadLocal,
                               PTy->getAddressSpace());
 }
 
+Value *LLParser::checkValidVariableType(LocTy Loc, const Twine &Name, Type *Ty,
+                                        Value *Val, bool IsCall) {
+  if (Val->getType() == Ty)
+    return Val;
+  // For calls we also accept variables in the program address space
+  Type* SuggestedTy = Ty;
+  if (IsCall && isa<PointerType>(Ty)) {
+    Type *TyInProgAS = cast<PointerType>(Ty)->getElementType()->getPointerTo(
+        M->getDataLayout().getProgramAddressSpace());
+    SuggestedTy = TyInProgAS;
+    if (Val->getType() == TyInProgAS)
+      return Val;
+  }
+  if (Ty->isLabelTy())
+    Error(Loc, "'" + Name + "' is not a basic block");
+  else
+    Error(Loc, "'" + Name + "' defined with type '" +
+                   getTypeString(Val->getType()) + "' but expected '" +
+                   getTypeString(SuggestedTy) + "'");
+  return nullptr;
+}
+
 /// GetGlobalVal - Get a value with the specified name or ID, creating a
 /// forward reference record if needed.  This can return null if the value
 /// exists but does not have the right type.
 GlobalValue *LLParser::GetGlobalVal(const std::string &Name, Type *Ty,
-                                    LocTy Loc) {
+                                    LocTy Loc, bool IsCall) {
   PointerType *PTy = dyn_cast<PointerType>(Ty);
   if (!PTy) {
     Error(Loc, "global variable reference must have pointer type");
     return nullptr;
   }
 
   // Look this name up in the normal function symbol table.
   GlobalValue *Val =
     cast_or_null<GlobalValue>(M->getValueSymbolTable().lookup(Name));
 
   // If this is a forward reference for the value, see if we already created a
   // forward ref record.
   if (!Val) {
     auto I = ForwardRefVals.find(Name);
     if (I != ForwardRefVals.end())
       Val = I->second.first;
   }
 
   // If we have the value in the symbol table or fwd-ref table, return it.
-  if (Val) {
-    if (Val->getType() == Ty) return Val;
-    Error(Loc, "'@" + Name + "' defined with type '" +
-          getTypeString(Val->getType()) + "'");
-    return nullptr;
-  }
+  if (Val)
+    return cast_or_null<GlobalValue>(
+        checkValidVariableType(Loc, "@" + Name, Ty, Val, IsCall));
 
   // Otherwise, create a new forward reference for this value and remember it.
   GlobalValue *FwdVal = createGlobalFwdRef(M, PTy, Name);
   ForwardRefVals[Name] = std::make_pair(FwdVal, Loc);
   return FwdVal;
 }
 
-GlobalValue *LLParser::GetGlobalVal(unsigned ID, Type *Ty, LocTy Loc) {
+GlobalValue *LLParser::GetGlobalVal(unsigned ID, Type *Ty, LocTy Loc,
+                                    bool IsCall) {
   PointerType *PTy = dyn_cast<PointerType>(Ty);
   if (!PTy) {
     Error(Loc, "global variable reference must have pointer type");
     return nullptr;
   }
 
   GlobalValue *Val = ID < NumberedVals.size() ? NumberedVals[ID] : nullptr;
 
   // If this is a forward reference for the value, see if we already created a
   // forward ref record.
   if (!Val) {
     auto I = ForwardRefValIDs.find(ID);
     if (I != ForwardRefValIDs.end())
       Val = I->second.first;
   }
 
   // If we have the value in the symbol table or fwd-ref table, return it.
-  if (Val) {
-    if (Val->getType() == Ty) return Val;
-    Error(Loc, "'@" + Twine(ID) + "' defined with type '" +
-          getTypeString(Val->getType()) + "'");
-    return nullptr;
-  }
+  if (Val)
+    return cast_or_null<GlobalValue>(
+        checkValidVariableType(Loc, "@" + Twine(ID), Ty, Val, IsCall));
 
   // Otherwise, create a new forward reference for this value and remember it.
   GlobalValue *FwdVal = createGlobalFwdRef(M, PTy, "");
   ForwardRefValIDs[ID] = std::make_pair(FwdVal, Loc);
   return FwdVal;
 }
 
 //===----------------------------------------------------------------------===//
return ParseTLSModel(TLM) ||
       ParseToken(lltok::rparen, "expected ')' after thread local model");
   }
   return false;
 }
 
 /// ParseOptionalAddrSpace
 ///   := /*empty*/
 ///   := 'addrspace' '(' uint32 ')'
 bool LLParser::ParseOptionalAddrSpace(unsigned &AddrSpace) {
   AddrSpace = 0;
   if (!EatIfPresent(lltok::kw_addrspace))
     return false;
   return ParseToken(lltok::lparen, "expected '(' in address space") ||
          ParseUInt32(AddrSpace) ||
          ParseToken(lltok::rparen, "expected ')' in address space");
 }
 
return P.Error(ForwardRefVals.begin()->second.second,
                    "'");
   if (!ForwardRefValIDs.empty())
     return P.Error(ForwardRefValIDs.begin()->second.second,
                    "use of undefined value '%" +
                    Twine(ForwardRefValIDs.begin()->first) + "'");
   return false;
 }
 
-static bool isValidVariableType(Module *M, Type *Ty, Value *Val, bool IsCall) {
-  if (Val->getType() == Ty)
-    return true;
-  // For calls we also accept variables in the program address space
-  if (IsCall && isa<PointerType>(Ty)) {
-    Type *TyInProgAS = cast<PointerType>(Ty)->getElementType()->getPointerTo(
-        M->getDataLayout().getProgramAddressSpace());
-    if (Val->getType() == TyInProgAS)
-      return true;
-  }
-  return false;
-}
-
 /// GetVal - Get a value with the specified name or ID, creating a
 /// forward reference record if needed.  This can return null if the value
 /// exists but does not have the right type.
 Value *LLParser::PerFunctionState::GetVal(const std::string &Name, Type *Ty,
                                           LocTy Loc, bool IsCall) {
   // Look this name up in the normal function symbol table.
   Value *Val = F.getValueSymbolTable()->lookup(Name);
 
   // If this is a forward reference for the value, see if we already created a
   // forward ref record.
   if (!Val) {
     auto I = ForwardRefVals.find(Name);
     if (I != ForwardRefVals.end())
       Val = I->second.first;
   }
 
   // If we have the value in the symbol table or fwd-ref table, return it.
-  if (Val) {
-    if (isValidVariableType(P.M, Ty, Val, IsCall))
-      return Val;
-    if (Ty->isLabelTy())
-      P.Error(Loc, "'%" + Name + "' is not a basic block");
-    else
-      P.Error(Loc, "'%" + Name + "' defined with type '" +
-              getTypeString(Val->getType()) + "'");
-    return nullptr;
-  }
+  if (Val)
+    return P.checkValidVariableType(Loc, "%" + Name, Ty, Val, IsCall);
 
   // Don't make placeholders with invalid type.
   if (!Ty->isFirstClassType()) {
     P.Error(Loc, "invalid use of a non-first-class type");
     return nullptr;
   }
 
   // Otherwise, create a new forward reference for this value and remember it.
Value *LLParser::PerFunctionState::GetVal(unsigned ID, Type *Ty, LocTy Loc,
   // forward ref record.
   if (!Val) {
     auto I = ForwardRefValIDs.find(ID);
     if (I != ForwardRefValIDs.end())
       Val = I->second.first;
   }
 
   // If we have the value in the symbol table or fwd-ref table, return it.
-  if (Val) {
-    if (isValidVariableType(P.M, Ty, Val, IsCall))
-      return Val;
-    if (Ty->isLabelTy())
-      P.Error(Loc, "'%" + Twine(ID) + "' is not a basic block");
-    else
-      P.Error(Loc, "'%" + Twine(ID) + "' defined with type '" +
-              getTypeString(Val->getType()) + "'");
-    return nullptr;
-  }
+  if (Val)
+    return P.checkValidVariableType(Loc, "%" + Twine(ID), Ty, Val, IsCall);
 
   if (!Ty->isFirstClassType()) {
     P.Error(Loc, "invalid use of a non-first-class type");
     return nullptr;
   }
 
   // Otherwise, create a new forward reference for this value and remember it.
   Value *FwdVal;
case ValID::t_InlineAsm: {
     if (!ID.FTy || !InlineAsm::Verify(ID.FTy, ID.StrVal2))
       return Error(ID.Loc, "invalid type for inline asm constraint string");
     V = InlineAsm::get(ID.FTy, ID.StrVal, ID.StrVal2, ID.UIntVal & 1,
                        (ID.UIntVal >> 1) & 1,
                        (InlineAsm::AsmDialect(ID.UIntVal >> 2)));
     return false;
   }
   case ValID::t_GlobalName:
-    V = GetGlobalVal(ID.StrVal, Ty, ID.Loc);
+    V = GetGlobalVal(ID.StrVal, Ty, ID.Loc, IsCall);
     return V == nullptr;
   case ValID::t_GlobalID:
-    V = GetGlobalVal(ID.UIntVal, Ty, ID.Loc);
+    V = GetGlobalVal(ID.UIntVal, Ty, ID.Loc, IsCall);
     return V == nullptr;
   case ValID::t_APSInt:
     if (!Ty->isIntegerTy())
       return Error(ID.Loc, "integer constant must have integer type");
     ID.APSIntVal = ID.APSIntVal.extOrTrunc(Ty->getPrimitiveSizeInBits());
     V = ConstantInt::get(Context, ID.APSIntVal);
     return false;
   case ValID::t_APFloat:
bool LLParser::ParseFunctionHeader(Function *&Fn, bool isDefine) {
   bool isVarArg;
   AttrBuilder FuncAttrs;
   std::vector<unsigned> FwdRefAttrGrps;
   LocTy BuiltinLoc;
   std::string Section;
   unsigned Alignment;
   std::string GC;
   GlobalValue::UnnamedAddr UnnamedAddr = GlobalValue::UnnamedAddr::None;
+  unsigned AddrSpace = 0;
   Constant *Prefix = nullptr;
   Constant *Prologue = nullptr;
   Constant *PersonalityFn = nullptr;
   Comdat *C;
 
   if (ParseArgumentList(ArgList, isVarArg) ||
       ParseOptionalUnnamedAddr(UnnamedAddr) ||
+      ParseOptionalAddrSpace(AddrSpace) ||
       ParseFnAttributeValuePairs(FuncAttrs, FwdRefAttrGrps, false,
                                  BuiltinLoc) ||
       (EatIfPresent(lltok::kw_section) &&
        ParseStringConstant(Section)) ||
       parseOptionalComdat(FunctionName, C) ||
       ParseOptionalAlignment(Alignment) ||
       (EatIfPresent(lltok::kw_gc) &&
        ParseStringConstant(GC)) ||
AttributeList PAL =
       AttributeList::get(Context, AttributeSet::get(Context, FuncAttrs),
                          AttributeSet::get(Context, RetAttrs), Attrs);
 
   if (PAL.hasAttribute(1, Attribute::StructRet) && !RetType->isVoidTy())
     return Error(RetTypeLoc, "functions with 'sret' argument must return void");
 
   FunctionType *FT =
     FunctionType::get(RetType, ParamTypeList, isVarArg);
-  PointerType *PFT = PointerType::getUnqual(FT);
+  PointerType *PFT = PointerType::get(FT, AddrSpace);
 
   Fn = nullptr;
   if (!FunctionName.empty()) {
     // If this was a definition of a forward reference, remove the definition
     // from the forward reference table and fill in the forward ref.
     auto FRVI = ForwardRefVals.find(FunctionName);
     if (FRVI != ForwardRefVals.end()) {
       Fn = M->getFunction(FunctionName);
       if (!Fn)
         return Error(FRVI->second.second, "invalid forward reference to "
                      "function as global value!");
       if (Fn->getType() != PFT)
         return Error(FRVI->second.second, "invalid forward reference to "
-                     "function '" + FunctionName + "' with wrong type!");
-
+                     "function '" + FunctionName + "' with wrong type: "
+                     "expected '" + getTypeString(PFT) + "' but was '" +
+                     getTypeString(Fn->getType()) + "'");
       ForwardRefVals.erase(FRVI);
     } else if ((Fn = M->getFunction(FunctionName))) {
       // Reject redefinitions.
       return Error(NameLoc, "invalid redefinition of function '" +
                    FunctionName + "'");
     } else if (M->getNamedValue(FunctionName)) {
       return Error(NameLoc, "redefinition of function '@" + FunctionName + "'");
     }
 
   } else {
     // If this is a definition of a forward referenced function, make sure the
     // types agree.
     auto I = ForwardRefValIDs.find(NumberedVals.size());
     if (I != ForwardRefValIDs.end()) {
       Fn = cast<Function>(I->second.first);
       if (Fn->getType() != PFT)
         return Error(NameLoc, "type of definition and forward reference of '@" +
-                     Twine(NumberedVals.size()) + "' disagree");
+                     Twine(NumberedVals.size()) + "' disagree: "
+                     "expected '" + getTypeString(PFT) + "' but was '" +
+                     getTypeString(Fn->getType()) + "'");
       ForwardRefValIDs.erase(I);
     }
   }
 
   if (!Fn)
-    Fn = Function::Create(FT, GlobalValue::ExternalLinkage, FunctionName, M);
+    Fn = Function::Create(FT, GlobalValue::ExternalLinkage, AddrSpace,
+                          FunctionName, M);
   else // Move the forward-reference to the correct spot in the module.
     M->getFunctionList().splice(M->end(), M->getFunctionList(), Fn);
 
+  assert(Fn->getAddressSpace() == AddrSpace && "Created function in wrong AS");
+
   if (FunctionName.empty())
     NumberedVals.push_back(Fn);
 
   Fn->setLinkage((GlobalValue::LinkageTypes)Linkage);
   maybeSetDSOLocal(DSOLocal, *Fn);
   Fn->setVisibility((GlobalValue::VisibilityTypes)Visibility);
   Fn->setDLLStorageClass((GlobalValue::DLLStorageClassTypes)DLLStorageClass);
   Fn->setCallingConv(CC);

 ///   ::= 'invoke' OptionalCallingConv OptionalAttrs Type Value ParamList
 ///       OptionalAttrs 'to' TypeAndValue 'unwind' TypeAndValue
 bool LLParser::ParseInvoke(Instruction *&Inst, PerFunctionState &PFS) {
   LocTy CallLoc = Lex.getLoc();
   AttrBuilder RetAttrs, FnAttrs;
   std::vector<unsigned> FwdRefAttrGrps;
   LocTy NoBuiltinLoc;
   unsigned CC;
+  unsigned InvokeAddrSpace;
   Type *RetType = nullptr;
   LocTy RetTypeLoc;
   ValID CalleeID;
   SmallVector<ParamInfo, 16> ArgList;
   SmallVector<OperandBundleDef, 2> BundleList;
 
   BasicBlock *NormalBB, *UnwindBB;
   if (ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
+      ParseOptionalAddrSpace(InvokeAddrSpace) ||
       ParseType(RetType, RetTypeLoc, true /*void allowed*/) ||
       ParseValID(CalleeID) || ParseParameterList(ArgList, PFS) ||
       ParseFnAttributeValuePairs(FnAttrs, FwdRefAttrGrps, false,
                                  NoBuiltinLoc) ||
       ParseOptionalOperandBundles(BundleList, PFS) ||
       ParseToken(lltok::kw_to, "expected 'to' in invoke") ||
       ParseTypeAndBasicBlock(NormalBB, PFS) ||
       ParseToken(lltok::kw_unwind, "expected 'unwind' in invoke") ||
if (!Ty) {
 
     Ty = FunctionType::get(RetType, ParamTypes, false);
   }
 
   CalleeID.FTy = Ty;
 
   // Look up the callee.
   Value *Callee;
-  if (ConvertValIDToValue(PointerType::getUnqual(Ty), CalleeID, Callee, &PFS,
+  if (ConvertValIDToValue(PointerType::get(Ty, InvokeAddrSpace), CalleeID, Callee, &PFS,
                           /*IsCall=*/true))
     return true;
 
   // Set up the Attribute for the function.
   SmallVector<Value *, 8> Args;
   SmallVector<AttributeSet, 8> ArgAttrs;
 
   // Loop through FunctionType's arguments and ensure they are specified

 ///           OptionalAttrs Type Value ParameterList OptionalAttrs
 ///   ::= 'notail' 'call'  OptionalFastMathFlags OptionalCallingConv
 ///           OptionalAttrs Type Value ParameterList OptionalAttrs
 bool LLParser::ParseCall(Instruction *&Inst, PerFunctionState &PFS,
                          CallInst::TailCallKind TCK) {
   AttrBuilder RetAttrs, FnAttrs;
   std::vector<unsigned> FwdRefAttrGrps;
   LocTy BuiltinLoc;
+  unsigned CallAddrSpace;
   unsigned CC;
   Type *RetType = nullptr;
   LocTy RetTypeLoc;
   ValID CalleeID;
   SmallVector<ParamInfo, 16> ArgList;
   SmallVector<OperandBundleDef, 2> BundleList;
   LocTy CallLoc = Lex.getLoc();
 
   if (TCK != CallInst::TCK_None &&
       ParseToken(lltok::kw_call,
                  "expected 'tail call', 'musttail call', or 'notail call'"))
     return true;
 
   FastMathFlags FMF = EatFastMathFlagsIfPresent();
 
   if (ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
+      ParseOptionalAddrSpace(CallAddrSpace) ||
       ParseType(RetType, RetTypeLoc, true /*void allowed*/) ||
       ParseValID(CalleeID) ||
       ParseParameterList(ArgList, PFS, TCK == CallInst::TCK_MustTail,
                          PFS.getFunction().isVarArg()) ||
       ParseFnAttributeValuePairs(FnAttrs, FwdRefAttrGrps, false, BuiltinLoc) ||
       ParseOptionalOperandBundles(BundleList, PFS))
     return true;
 
if (!Ty) {
 
     Ty = FunctionType::get(RetType, ParamTypes, false);
   }
 
   CalleeID.FTy = Ty;
 
   // Look up the callee.
   Value *Callee;
-  if (ConvertValIDToValue(PointerType::getUnqual(Ty), CalleeID, Callee, &PFS,
+  if (ConvertValIDToValue(PointerType::get(Ty, CallAddrSpace), CalleeID, Callee, &PFS,
                           /*IsCall=*/true))
     return true;
 
   // Set up the Attribute for the function.
   SmallVector<AttributeSet, 8> Attrs;
 
   SmallVector<Value*, 8> Args;
 

lib/Bitcode/Reader/BitcodeReader.cpp

Error BitcodeReader::parseGlobalVarRecord(ArrayRef<uint64_t> Record) {
   }
 
   return Error::success();
 }
 
 Error BitcodeReader::parseFunctionRecord(ArrayRef<uint64_t> Record) {
   // v1: [type, callingconv, isproto, linkage, paramattr, alignment, section,
   // visibility, gc, unnamed_addr, prologuedata, dllstorageclass, comdat,
-  // prefixdata,  personalityfn, preemption specifier] (name in VST)
+  // prefixdata,  personalityfn, preemption specifier, addrspace] (name in VST)
   // v2: [strtab_offset, strtab_size, v1]
   StringRef Name;
   std::tie(Name, Record) = readNameFromStrtab(Record);
 
   if (Record.size() < 8)
     return error("Invalid record");
   Type *Ty = getTypeByID(Record[0]);
   if (!Ty)
     return error("Invalid record");
   if (auto *PTy = dyn_cast<PointerType>(Ty))
     Ty = PTy->getElementType();
   auto *FTy = dyn_cast<FunctionType>(Ty);
   if (!FTy)
     return error("Invalid type for value");
   auto CC = static_cast<CallingConv::ID>(Record[1]);
   if (CC & ~CallingConv::MaxID)
     return error("Invalid calling convention ID");
 
-  Function *Func =
-      Function::Create(FTy, GlobalValue::ExternalLinkage, Name, TheModule);
+  unsigned AddrSpace = TheModule->getDataLayout().getProgramAddressSpace();
+  if (Record.size() > 16)
+    AddrSpace = Record[16];
+
+  Function *Func = Function::Create(FTy, GlobalValue::ExternalLinkage,
+                                    AddrSpace, Name, TheModule);
 
   Func->setCallingConv(CC);
   bool isProto = Record[2];
   uint64_t RawLinkage = Record[3];
   Func->setLinkage(getDecodedLinkage(RawLinkage));
   Func->setAttributes(getAttributes(Record[4]));
 
   unsigned Alignment;

lib/Bitcode/Writer/BitcodeWriter.cpp

for (const GlobalVariable &GV : M.globals()) {
     Vals.clear();
   }
 
   // Emit the function proto information.
   for (const Function &F : M) {
     // FUNCTION:  [strtab offset, strtab size, type, callingconv, isproto,
     //             linkage, paramattrs, alignment, section, visibility, gc,
     //             unnamed_addr, prologuedata, dllstorageclass, comdat,
-    //             prefixdata, personalityfn, DSO_Local]
+    //             prefixdata, personalityfn, DSO_Local, addrspace]
     Vals.push_back(addToStrtab(F.getName()));
     Vals.push_back(F.getName().size());
     Vals.push_back(VE.getTypeID(F.getFunctionType()));
     Vals.push_back(F.getCallingConv());
     Vals.push_back(F.isDeclaration());
     Vals.push_back(getEncodedLinkage(F));
     Vals.push_back(VE.getAttributeListID(F.getAttributes()));
     Vals.push_back(Log2_32(F.getAlignment())+1);
     Vals.push_back(F.hasSection() ? SectionMap[F.getSection()] : 0);
     Vals.push_back(getEncodedVisibility(F));
     Vals.push_back(F.hasGC() ? GCMap[F.getGC()] : 0);
     Vals.push_back(getEncodedUnnamedAddr(F));
     Vals.push_back(F.hasPrologueData() ? (VE.getValueID(F.getPrologueData()) + 1)
                                        : 0);
     Vals.push_back(getEncodedDLLStorageClass(F));
     Vals.push_back(F.hasComdat() ? VE.getComdatID(F.getComdat()) : 0);
     Vals.push_back(F.hasPrefixData() ? (VE.getValueID(F.getPrefixData()) + 1)
                                      : 0);
     Vals.push_back(
         F.hasPersonalityFn() ? (VE.getValueID(F.getPersonalityFn()) + 1) : 0);
 
     Vals.push_back(F.isDSOLocal());
+    Vals.push_back(F.getAddressSpace());
+
     unsigned AbbrevToUse = 0;
     Stream.EmitRecord(bitc::MODULE_CODE_FUNCTION, Vals, AbbrevToUse);
     Vals.clear();
   }
 
   // Emit the alias information.
   for (const GlobalAlias &A : M.aliases()) {
     // ALIAS: [strtab offset, strtab size, alias type, aliasee val#, linkage,

lib/IR/AsmWriter.cpp

void AssemblyWriter::printFunction(const Function *F) {
   if (FT->isVarArg()) {
     if (FT->getNumParams()) Out << ", ";
     Out << "...";  // Output varargs portion of signature!
   }
   Out << ')';
   StringRef UA = getUnnamedAddrEncoding(F->getUnnamedAddr());
   if (!UA.empty())
     Out << ' ' << UA;
+  if (F->getAddressSpace() != 0)
+    Out << " addrspace(" << F->getAddressSpace() << ")";
   if (Attrs.hasAttributes(AttributeList::FunctionIndex))
     Out << " #" << Machine.getAttributeGroupSlot(Attrs.getFnAttributes());
   if (F->hasSection()) {
     Out << " section \"";
     printEscapedString(F->getSection(), Out);
     Out << '"';
   }
   maybePrintComdat(Out, *F);
if (isa<BranchInst>(I) && cast<BranchInst>(I).isConditional()) {
 
     if (PAL.hasAttributes(AttributeList::ReturnIndex))
       Out << ' ' << PAL.getAsString(AttributeList::ReturnIndex);
 
     // If possible, print out the short form of the call instruction.  We can
     // only do this if the first argument is a pointer to a nonvararg function,
     // and if the return type is not a pointer to a function.
     //
+    if (Operand->getType()->getPointerAddressSpace() != 0)
+      Out << " addrspace(" << Operand->getType()->getPointerAddressSpace() << ")";
     Out << ' ';
     TypePrinter.print(FTy->isVarArg() ? FTy : RetTy, Out);
     Out << ' ';
     writeOperand(Operand, false);
     Out << '(';
     for (unsigned op = 0, Eop = CI->getNumArgOperands(); op < Eop; ++op) {
       if (op > 0)
         Out << ", ";
if (isa<BranchInst>(I) && cast<BranchInst>(I).isConditional()) {
 
     if (PAL.hasAttributes(AttributeList::ReturnIndex))
       Out << ' ' << PAL.getAsString(AttributeList::ReturnIndex);
 
     // If possible, print out the short form of the invoke instruction. We can
     // only do this if the first argument is a pointer to a nonvararg function,
     // and if the return type is not a pointer to a function.
     //
+    if (Operand->getType()->getPointerAddressSpace() != 0)
+      Out << " addrspace(" << Operand->getType()->getPointerAddressSpace() << ")";
     Out << ' ';
     TypePrinter.print(FTy->isVarArg() ? FTy : RetTy, Out);
     Out << ' ';
     writeOperand(Operand, false);
     Out << '(';
     for (unsigned op = 0, Eop = II->getNumArgOperands(); op < Eop; ++op) {
       if (op)
         Out << ", ";

lib/IR/AutoUpgrade.cpp

if (Name.startswith("arm.neon.vclz")) {
       Type* args[2] = {
         F->arg_begin()->getType(),
         Type::getInt1Ty(F->getContext())
       };
       // Can't use Intrinsic::getDeclaration here as it adds a ".i1" to
       // the end of the name. Change name from llvm.arm.neon.vclz.* to
       //  llvm.ctlz.*
       FunctionType* fType = FunctionType::get(F->getReturnType(), args, false);
-      NewFn = Function::Create(fType, F->getLinkage(),
+      NewFn = Function::Create(fType, F->getLinkage(), F->getAddressSpace(),
                                "llvm.ctlz." + Name.substr(14), F->getParent());
       return true;
     }
     if (Name.startswith("arm.neon.vcnt")) {
       NewFn = Intrinsic::getDeclaration(F->getParent(), Intrinsic::ctpop,
                                         F->arg_begin()->getType());
       return true;
     }
     Regex vldRegex("^arm\\.neon\\.vld([1234]|[234]lane)\\.v[a-z0-9]*$");
     if (vldRegex.match(Name)) {
       auto fArgs = F->getFunctionType()->params();
       SmallVector<Type *, 4> Tys(fArgs.begin(), fArgs.end());
       // Can't use Intrinsic::getDeclaration here as the return types might
       // then only be structurally equal.
       FunctionType* fType = FunctionType::get(F->getReturnType(), Tys, false);
-      NewFn = Function::Create(fType, F->getLinkage(),
+      NewFn = Function::Create(fType, F->getLinkage(), F->getAddressSpace(),
                                "llvm." + Name + ".p0i8", F->getParent());
       return true;
     }
     Regex vstRegex("^arm\\.neon\\.vst([1234]|[234]lane)\\.v[a-z0-9]*$");
     if (vstRegex.match(Name)) {
       static const Intrinsic::ID StoreInts[] = {Intrinsic::arm_neon_vst1,
                                                 Intrinsic::arm_neon_vst2,
                                                 Intrinsic::arm_neon_vst3,

lib/IR/Function.cpp

 unsigned Function::getInstructionCount() {
   unsigned NumInstrs = 0;
   for (BasicBlock &BB : BasicBlocks)
     NumInstrs += std::distance(BB.instructionsWithoutDebug().begin(),
                                BB.instructionsWithoutDebug().end());
   return NumInstrs;
 }
 
+Function *Function::Create(FunctionType *Ty, LinkageTypes Linkage,
+                           const Twine &N, Module &M) {
+  return Create(Ty, Linkage, M.getDataLayout().getProgramAddressSpace(), N, &M);
+}
+
+Function *Function::CreateBefore(Function &InsertBefore, FunctionType *Ty,
+                                 LinkageTypes Linkage, const Twine &N) {
+  auto &M = *InsertBefore.getParent();
+  unsigned AddrSpace = M.getDataLayout().getProgramAddressSpace();
+  Function *F = Create(Ty, Linkage, AddrSpace, N);
+
+  M.getFunctionList().insert(InsertBefore.getIterator(), F);
+  return F;
+}
+
 void Function::removeFromParent() {
   getParent()->getFunctionList().remove(getIterator());
 }
 
 void Function::eraseFromParent() {
   getParent()->getFunctionList().erase(getIterator());
 }
 
 //===----------------------------------------------------------------------===//
 // Function Implementation
 //===----------------------------------------------------------------------===//
 
-Function::Function(FunctionType *Ty, LinkageTypes Linkage, const Twine &name,
-                   Module *ParentModule)
+static unsigned computeAddrSpace(unsigned AddrSpace, Module *M) {
+  // If AS == -1 and we are passed a valid module pointer we place the function
+  // in the program address space and otherwise we default to AS0
+  if (AddrSpace == static_cast<unsigned>(-1))
+    return M ? M->getDataLayout().getProgramAddressSpace() : 0;
+  return AddrSpace;
+}
+
+Function::Function(FunctionType *Ty, LinkageTypes Linkage, unsigned AddrSpace,
+                   const Twine &name, Module *ParentModule)
     : GlobalObject(Ty, Value::FunctionVal,
-                   OperandTraits<Function>::op_begin(this), 0, Linkage, name),
+                   OperandTraits<Function>::op_begin(this), 0, Linkage, name,
+                   computeAddrSpace(AddrSpace, ParentModule)),
       NumArgs(Ty->getNumParams()) {
   assert(FunctionType::isValidReturnType(getReturnType()) &&
          "invalid return type");
   setGlobalObjectSubClassData(0);
 
   // We only need a symbol table for a function if the context keeps value names
   if (!getContext().shouldDiscardValueNames())
     SymTab = make_unique<ValueSymbolTable>();

lib/IR/Globals.cpp

if (auto *GA = dyn_cast<GlobalAlias>(this)) {
     // FIXME: we should also be able to handle:
     // Alias = Global + Offset
     // Alias = Absolute
     return 0;
   }
   return cast<GlobalObject>(this)->getAlignment();
 }
 
+unsigned GlobalValue::getAddressSpace() const {
+  PointerType *PtrTy = getType();
+  return PtrTy->getAddressSpace();
+}
+
 void GlobalObject::setAlignment(unsigned Align) {
   assert((Align & (Align-1)) == 0 && "Alignment is not a power of 2!");
   assert(Align <= MaximumAlignment &&
          "Alignment is greater than MaximumAlignment!");
   unsigned AlignmentData = Log2_32(Align) + 1;
   unsigned OldData = getGlobalValueSubClassData();
   setGlobalValueSubClassData((OldData & ~AlignmentMask) | AlignmentData);
   assert(getAlignment() == Align && "Alignment representation error!");

lib/IR/Module.cpp

 // the symbol table directly for this common task.
 //
 Constant *Module::getOrInsertFunction(StringRef Name, FunctionType *Ty,
                                       AttributeList AttributeList) {
   // See if we have a definition for the specified function already.
   GlobalValue *F = getNamedValue(Name);
   if (!F) {
     // Nope, add it
-    Function *New = Function::Create(Ty, GlobalVariable::ExternalLinkage, Name);
+    Function *New = Function::Create(Ty, GlobalVariable::ExternalLinkage,
+                                     DL.getProgramAddressSpace(), Name);
     if (!New->isIntrinsic())       // Intrinsics get attrs set on construction
       New->setAttributes(AttributeList);
     FunctionList.push_back(New);
     return New;                    // Return the new prototype.
   }
 
   // If the function exists but has the wrong type, return a bitcast to the
   // right type.
-  if (F->getType() != PointerType::getUnqual(Ty))
-    return ConstantExpr::getBitCast(F, PointerType::getUnqual(Ty));
+  auto *PTy = PointerType::get(Ty, F->getAddressSpace());
+  if (F->getType() != PTy)
+    return ConstantExpr::getBitCast(F, PTy);
 
   // Otherwise, we just found the existing function or a prototype.
   return F;
 }
 
 Constant *Module::getOrInsertFunction(StringRef Name,
                                       FunctionType *Ty) {
   return getOrInsertFunction(Name, Ty, AttributeList());

lib/Target/AMDGPU/AMDGPURewriteOutArguments.cpp

bool AMDGPURewriteOutArguments::runOnFunction(Function &F) {
   StructType *NewRetTy = StructType::create(Ctx, ReturnTypes, F.getName());
 
   FunctionType *NewFuncTy = FunctionType::get(NewRetTy,
                                               F.getFunctionType()->params(),
                                               F.isVarArg());
 
   LLVM_DEBUG(dbgs() << "Computed new return type: " << *NewRetTy << '\n');
 
-  Function *NewFunc = Function::Create(NewFuncTy, Function::PrivateLinkage,
-                                       F.getName() + ".body");
-  F.getParent()->getFunctionList().insert(F.getIterator(), NewFunc);
+  Function *NewFunc = Function::CreateBefore(
+      F, NewFuncTy, Function::PrivateLinkage, F.getName() + ".body");
   NewFunc->copyAttributesFrom(&F);
   NewFunc->setComdat(F.getComdat());
 
   // We want to preserve the function and param attributes, but need to strip
   // off any return attributes, e.g. zeroext doesn't make sense with a struct.
   NewFunc->stealArgumentListFrom(F);
 
   AttrBuilder RetAttrs;

lib/Transforms/IPO/ArgumentPromotion.cpp

doPromotion(Function *F, SmallPtrSetImpl<Argument *> &ArgsToPromote,
   }
 
   Type *RetTy = FTy->getReturnType();
 
   // Construct the new function type using the new arguments.
   FunctionType *NFTy = FunctionType::get(RetTy, Params, FTy->isVarArg());
 
   // Create the new function body and insert it into the module.
-  Function *NF = Function::Create(NFTy, F->getLinkage(), F->getName());
+  Function *NF =
+      Function::CreateBefore(*F, NFTy, F->getLinkage(), F->getName());
   NF->copyAttributesFrom(F);
 
   // Patch the pointer to LLVM function in debug info descriptor.
   NF->setSubprogram(F->getSubprogram());
   F->setSubprogram(nullptr);
 
   LLVM_DEBUG(dbgs() << "ARG PROMOTION:  Promoting to:" << *NF << "\n"
                     << "From: " << *F);
 
   // Recompute the parameter attributes list based on the new arguments for
   // the function.
   NF->setAttributes(AttributeList::get(F->getContext(), PAL.getFnAttributes(),
                                        PAL.getRetAttributes(), ArgAttrVec));
   ArgAttrVec.clear();
 
-  F->getParent()->getFunctionList().insert(F->getIterator(), NF);
   NF->takeName(F);
 
   // Loop over all of the callers of the function, transforming the call sites
   // to pass in the loaded pointers.
   //
   SmallVector<Value *, 16> Args;
   while (!F->use_empty()) {
     CallSite CS(F->user_back());

lib/Transforms/IPO/DeadArgumentElimination.cpp

bool DeadArgumentEliminationPass::DeleteDeadVarargs(Function &Fn) {
   FunctionType *FTy = Fn.getFunctionType();
 
   std::vector<Type *> Params(FTy->param_begin(), FTy->param_end());
   FunctionType *NFTy = FunctionType::get(FTy->getReturnType(),
                                                 Params, false);
   unsigned NumArgs = Params.size();
 
   // Create the new function body and insert it into the module...
-  Function *NF = Function::Create(NFTy, Fn.getLinkage());
+  Function *NF = Function::CreateBefore(Fn, NFTy, Fn.getLinkage());
   NF->copyAttributesFrom(&Fn);
   NF->setComdat(Fn.getComdat());
-  Fn.getParent()->getFunctionList().insert(Fn.getIterator(), NF);
   NF->takeName(&Fn);
 
   // Loop over all of the callers of the function, transforming the call sites
   // to pass in a smaller number of arguments into the new function.
   //
   std::vector<Value *> Args;
   for (Value::user_iterator I = Fn.user_begin(), E = Fn.user_end(); I != E; ) {
     CallSite CS(*I++);
bool DeadArgumentEliminationPass::RemoveDeadStuffFromFunction(Function *F) {
 
   // Create the new function type based on the recomputed parameters.
   FunctionType *NFTy = FunctionType::get(NRetTy, Params, FTy->isVarArg());
 
   // No change?
   if (NFTy == FTy)
     return false;
 
-  // Create the new function body and insert it into the module...
-  Function *NF = Function::Create(NFTy, F->getLinkage());
+  // Create the new function body.
+  // Insert the new function before the old function, so we won't be processing
+  // it again.
+  Function *NF = Function::CreateBefore(*F, NFTy, F->getLinkage());
   NF->copyAttributesFrom(F);
   NF->setComdat(F->getComdat());
   NF->setAttributes(NewPAL);
-  // Insert the new function before the old function, so we won't be processing
-  // it again.
-  F->getParent()->getFunctionList().insert(F->getIterator(), NF);
   NF->takeName(F);
 
   // Loop over all of the callers of the function, transforming the call sites
   // to pass in a smaller number of arguments into the new function.
   std::vector<Value*> Args;
   while (!F->use_empty()) {
     CallSite CS(F->user_back());
     Instruction *Call = CS.getInstruction();

lib/Transforms/Instrumentation/DataFlowSanitizer.cpp

void DataFlowSanitizer::addGlobalNamePrefix(GlobalValue *GV) {
   }
 }
 
 Function *
 DataFlowSanitizer::buildWrapperFunction(Function *F, StringRef NewFName,
                                         GlobalValue::LinkageTypes NewFLink,
                                         FunctionType *NewFT) {
   FunctionType *FT = F->getFunctionType();
-  Function *NewF = Function::Create(NewFT, NewFLink, NewFName,
-                                    F->getParent());
+  Function *NewF = Function::Create(NewFT, NewFLink, F->getAddressSpace(),
+                                    NewFName, F->getParent());
   NewF->copyAttributesFrom(F);
   NewF->removeAttributes(
       AttributeList::ReturnIndex,
       AttributeFuncs::typeIncompatible(NewFT->getReturnType()));
 
   BasicBlock *BB = BasicBlock::Create(*Ctx, "entry", NewF);
   if (F->isVarArg()) {
     NewF->removeAttributes(AttributeList::FunctionIndex,
for (std::vector<Function *>::iterator i = FnsToInstrument.begin(),
     bool IsZeroArgsVoidRet = (FT->getNumParams() == 0 && !FT->isVarArg() &&
                               FT->getReturnType()->isVoidTy());
 
     if (isInstrumented(&F)) {
       // Instrumented functions get a 'dfs$' prefix.  This allows us to more
       // easily identify cases of mismatching ABIs.
       if (getInstrumentedABI() == IA_Args && !IsZeroArgsVoidRet) {
         FunctionType *NewFT = getArgsFunctionType(FT);
-        Function *NewF = Function::Create(NewFT, F.getLinkage(), "", &M);
+        Function *NewF = Function::Create(NewFT, F.getLinkage(),
+                                          F.getAddressSpace(), "", &M);
         NewF->copyAttributesFrom(&F);
         NewF->removeAttributes(
             AttributeList::ReturnIndex,
             AttributeFuncs::typeIncompatible(NewFT->getReturnType()));
         for (Function::arg_iterator FArg = F.arg_begin(),
                                     NewFArg = NewF->arg_begin(),
                                     FArgEnd = F.arg_end();
              FArg != FArgEnd; ++FArg, ++NewFArg) {

lib/Transforms/Utils/CloneFunction.cpp

for (const Argument &I : F->args())
     if (VMap.count(&I) == 0) // Haven't mapped the argument to anything yet?
       ArgTypes.push_back(I.getType());
 
   // Create a new function type...
   FunctionType *FTy = FunctionType::get(F->getFunctionType()->getReturnType(),
                                     ArgTypes, F->getFunctionType()->isVarArg());
 
   // Create the new function...
-  Function *NewF =
-      Function::Create(FTy, F->getLinkage(), F->getName(), F->getParent());
+  Function *NewF = Function::Create(FTy, F->getLinkage(), F->getAddressSpace(),
+                                    F->getName(), F->getParent());
 
   // Loop over the arguments, copying the names of the mapped arguments over...
   Function::arg_iterator DestI = NewF->arg_begin();
   for (const Argument & I : F->args())
     if (VMap.count(&I) == 0) {     // Is this argument preserved?
       DestI->setName(I.getName()); // Copy the name over...
       VMap[&I] = &*DestI++;        // Add mapping to VMap
     }

lib/Transforms/Utils/CloneModule.cpp

GlobalVariable *GV = new GlobalVariable(*New,
                                             I->getThreadLocalMode(),
                                             I->getType()->getAddressSpace());
     GV->copyAttributesFrom(&*I);
     VMap[&*I] = GV;
   }
 
   // Loop over the functions in the module, making external functions as before
   for (const Function &I : M) {
-    Function *NF = Function::Create(cast<FunctionType>(I.getValueType()),
-                                    I.getLinkage(), I.getName(), New.get());
+    Function *NF =
+        Function::Create(cast<FunctionType>(I.getValueType()), I.getLinkage(),
+                         I.getAddressSpace(), I.getName(), New.get());
     NF->copyAttributesFrom(&I);
     VMap[&I] = NF;
   }
 
   // Loop over the aliases in the module
   for (Module::const_alias_iterator I = M.alias_begin(), E = M.alias_end();
        I != E; ++I) {
     if (!ShouldCloneDefinition(&*I)) {
       // An alias cannot act as an external reference, so we need to create
       // either a function or a global variable depending on the value type.
       // FIXME: Once pointee types are gone we can probably pick one or the
       // other.
       GlobalValue *GV;
       if (I->getValueType()->isFunctionTy())
         GV = Function::Create(cast<FunctionType>(I->getValueType()),
-                              GlobalValue::ExternalLinkage, I->getName(),
-                              New.get());
+                              GlobalValue::ExternalLinkage,
+                              I->getAddressSpace(), I->getName(), New.get());
       else
         GV = new GlobalVariable(
             *New, I->getValueType(), false, GlobalValue::ExternalLinkage,
             nullptr, I->getName(), nullptr,
             I->getThreadLocalMode(), I->getType()->getAddressSpace());
       VMap[&*I] = GV;
       // We do not copy attributes (mainly because copying between different
       // kinds of globals is forbidden), but this is generally not required for

lib/Transforms/Utils/CodeExtractor.cpp

if (AggregateArgs && (inputs.size() + outputs.size() > 0)) {
     paramTy.clear();
     paramTy.push_back(PointerType::getUnqual(StructTy));
   }
   FunctionType *funcType =
                   FunctionType::get(RetTy, paramTy,
                                     AllowVarArgs && oldFunction->isVarArg());
 
   // Create the new function
-  Function *newFunction = Function::Create(funcType,
-                                           GlobalValue::InternalLinkage,
-                                           oldFunction->getName() + "_" +
-                                           header->getName(), M);
+  Function *newFunction = Function::Create(
+      funcType, GlobalValue::InternalLinkage, oldFunction->getAddressSpace(),
+      oldFunction->getName() + "_" + header->getName(), M);
   // If the old function is no-throw, so is the new one.
   if (oldFunction->doesNotThrow())
     newFunction->setDoesNotThrow();
 
   // Inherit the uwtable attribute if we need to.
   if (oldFunction->hasUWTable())
     newFunction->setHasUWTable();
 

test/Bitcode/function-address-space-fwd-decl.ll

+; Verify that forward declarations from call instructions work even with non-zero AS
+; RUN: llvm-as %s -o - | llvm-dis - | FileCheck %s
+
+define void @call_named() {
+entry:
+  %0 = tail call addrspace(40) i32 @llvm.phx.aload(i16* null)
+  ; CHECK: %0 = tail call addrspace(40) i32 @llvm.phx.aload(i16* null)
+  ret void
+}
+
+define void @call_numbered() {
+entry:
+  %0 = tail call addrspace(40) i32 @0(i16* null)
+  ; CHECK: %0 = tail call addrspace(40) i32 @0(i16* null)
+  ret void
+}
+
+
+define i32 @invoked() personality i8* null {
+entry:
+  %0 = invoke addrspace(40) i32 @foo() to label %l1 unwind label %lpad
+  ; CHECK: invoke addrspace(40) i32 @foo()
+l1:
+  br label %return
+lpad:
+  %1 = landingpad { i8*, i32 }
+    catch i8* null
+    catch i8* null
+  ret i32 0
+return:
+  ret i32 0
+}
+
+declare i32 @foo() addrspace(40)
+; CHECK: declare i32 @foo() addrspace(40)
+declare i32 @llvm.phx.aload(i16* nocapture) addrspace(40)
+; CHECK: declare i32 @llvm.phx.aload(i16* nocapture) addrspace(40)
+declare i32 @0(i16*) addrspace(40)
+; CHECK: declare i32 @0(i16*) addrspace(40)

test/Bitcode/function-default-address-spaces.ll

+; RUN: llvm-as %s  -o - | llvm-dis - | FileCheck %s -check-prefixes CHECK,PROG-AS0
+; RUN: llvm-as -data-layout "P200" %s  -o - | llvm-dis | FileCheck %s -check-prefixes CHECK,PROG-AS200
+; RUN: not llvm-as -data-layout "P123456789" %s -o /dev/null 2>&1 | FileCheck %s -check-prefix BAD-DATALAYOUT
+; BAD-DATALAYOUT: LLVM ERROR: Invalid address space, must be a 24-bit integer
+
+; PROG-AS0-NOT: target datalayout
+; PROG-AS200: target datalayout = "P200"
+
+; Check that a function declaration without an address space (i.e. AS0) does not
+; have the addrspace() attribute printed:
+
+; CHECK: define void @no_as() {
+define void @no_as() {
+  ret void
+}
+
+; A function with an explicit addrspace should only have the addrspace printed
+; if it is non-zero
+; CHECK: define void @explit_as0()  {
+define void @explit_as0() addrspace(0) {
+  ret void
+}
+
+; CHECK: define void @explit_as200() addrspace(200) {
+define void @explit_as200() addrspace(200) {
+  ret void
+}
+
+; CHECK: define void @explicit_as3() addrspace(3) {
+define void @explicit_as3() addrspace(3) {
+  ret void
+}

test/Bitcode/function-nonzero-address-spaces-types.ll

+; Verify that we accept calls to variables in the program AS:
+; RUN: llvm-as -data-layout "P40" %s -o - | llvm-dis - | FileCheck %s
+; CHECK: target datalayout = "P40"
+
+; We should get a sensible error for a non-program address call:
+; RUN: not llvm-as -data-layout "P39" %s -o /dev/null 2>&1 | FileCheck %s -check-prefix ERR-AS39
+; ERR-AS39: error: '%0' defined with type 'i16 (i16) addrspace(40)*' but expected 'i16 (i16) addrspace(39)*'
+
+; And also if we don't set a custom program address space:
+; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s -check-prefix ERR-AS0
+; ERR-AS0: error: '%0' defined with type 'i16 (i16) addrspace(40)*' but expected 'i16 (i16)*'
+
+%fun1 = type i16 (i16)
+%funptr1 = type %fun1 addrspace(40)*
+
+@fun_ptr = global %funptr1 @fun
+
+define i16 @fun(i16 %arg) addrspace(40) {
+entry:
+  %0 = load %funptr1, %funptr1* @fun_ptr
+  %result = call i16 %0(i16 %arg)
+  ret i16 %result
+}

test/Bitcode/function-nonzero-address-spaces.ll

+; Verify that we accept calls to variables in the program AS:
+; RUN: llvm-as -data-layout "P40" %s -o - | llvm-dis - | FileCheck %s
+; CHECK: target datalayout = "P40"
+
+; We should get a sensible error for a non-program address call:
+; RUN: not llvm-as -data-layout "P39" %s -o /dev/null 2>&1 | FileCheck %s -check-prefix ERR-AS39
+; ERR-AS39: error: '%fnptr' defined with type 'void (i16) addrspace(40)*' but expected 'void (i16) addrspace(39)*'
+
+; And also if we don't set a custom program address space:
+; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s -check-prefix ERR-AS0
+; ERR-AS0: error: '%fnptr' defined with type 'void (i16) addrspace(40)*' but expected 'void (i16)*'
+
+define void @f_named(i16 %n, void (i16) addrspace(40)* %f) addrspace(40) {
+entry:
+  %f.addr = alloca void (i16) addrspace(40)*, align 1
+  store void (i16) addrspace(40)* %f, void (i16) addrspace(40)** %f.addr
+  %fnptr = load void (i16) addrspace(40)*, void (i16) addrspace(40)** %f.addr
+  call void %fnptr(i16 8)
+  ret void
+}
+
+define void @f_numbered(i16 %n, void (i16) addrspace(40)* %f) addrspace(40){
+entry:
+  %f.addr = alloca void (i16) addrspace(40)*, align 1
+  store void (i16) addrspace(40)* %f, void (i16) addrspace(40)** %f.addr
+  %0 = load void (i16) addrspace(40)*, void (i16) addrspace(40)** %f.addr
+  call void %0(i16 8)
+  ret void
+}

unittests/Analysis/LazyCallGraphTest.cpp

TEST(LazyCallGraphTest, ReplaceNodeFunction) {
   EXPECT_EQ(&C2, CG.lookupSCC(BN));
   EXPECT_EQ(&RC1, CG.lookupRefSCC(DN));
   EXPECT_EQ(&RC1, CG.lookupRefSCC(CN));
   EXPECT_EQ(&RC1, CG.lookupRefSCC(BN));
   EXPECT_EQ(&RC2, CG.lookupRefSCC(AN));
 
   // Now we need to build a new function 'e' with the same signature as 'd'.
   Function &D = DN.getFunction();
-  Function &E = *Function::Create(D.getFunctionType(), D.getLinkage(), "e");
-  D.getParent()->getFunctionList().insert(D.getIterator(), &E);
+  Function &E =
+      *Function::CreateBefore(D, D.getFunctionType(), D.getLinkage(), "e");
 
   // Change each use of 'd' to use 'e'. This is particularly easy as they have
   // the same type.
   D.replaceAllUsesWith(&E);
 
   // Splice the body of the old function into the new one.
   E.getBasicBlockList().splice(E.begin(), D.getBasicBlockList());
   // And fix up the one argument.