Introduce RThreadAccess pass

[yrouban]

Summary

If the option -rthread-register=<register-name> is specified then:

• Protect rthread register from register allocation.
• Add rhread register as a base pointer to pointer operand of load and store instructions if the pointer operand has addrspace(X), where X is specified with -rthread-addrspace option (256 by default).

This pass might be useful for Thread Local Access operations with a dedicated TLS register.

RThreadAccess is added to X86 and AArch64 pipeline in addPreISel() and can be added to other targets as needed.

[arsenm]

llvm/include/llvm/Transforms/Scalar/RThreadAccess.h

39

global option codegen options like this are bad. If this is for more than debugging it should be a function attribute or something

[arsenm]

Would this be better served with an intrinsic to read this register? Does the register need to be globally reserved or just within the local function?

[yrouban]

In D152899#4444040, @arsenm wrote:

Would this be better served with an intrinsic to read this register? Does the register need to be globally reserved or just within the local function?

Actually this new pass RThreadAccess generates reading the rthread register with @llvm.read_register().
I believe that, the easiest rule would be to reserve the rthread register for all functions (in one module at least). This allow to not think if this register can be changed or not by other functions called.
So I'm thinking if it would be ok to set these options as module level metadata instead of cl::opt.

[arsenm]

In D152899#4446947, @yrouban wrote:

In D152899#4444040, @arsenm wrote:

Would this be better served with an intrinsic to read this register? Does the register need to be globally reserved or just within the local function?

Actually this new pass RThreadAccess generates reading the rthread register with @llvm.read_register().

Right, this is the ugly thing I'm trying to guide you away from doing but need to understand exactly what the constraints are on the register. An intrinsic with a defined meaning is better than an unrestricted register read

[yrouban]

In D152899#4447418, @arsenm wrote:

Right, this is the ugly thing I'm trying to guide you away from doing but need to understand exactly what the constraints are on the register. An intrinsic with a defined meaning is better than an unrestricted register read.

Thank you. An excuse for reading register is that it is done in the very late stage of the pipeline and is almost target agnostic.

[arsenm]

In D152899#4470137, @yrouban wrote:

In D152899#4447418, @arsenm wrote:

Right, this is the ugly thing I'm trying to guide you away from doing but need to understand exactly what the constraints are on the register. An intrinsic with a defined meaning is better than an unrestricted register read.

Thank you. An excuse for reading register is that it is done in the very late stage of the pipeline and is almost target agnostic.

I find the areas where people try to bypass the normal legalization process difficult to follow and implement. I think expecting to parse meaning out of an unrestricted read and a side channel to try to interpret it is more difficult to deal with rather than something higher level

llvm/include/llvm/Transforms/Scalar/RThreadAccess.h

+//===- RThreadAccess.h ------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// \file
+// This file provides the interface for RThreadAccess pass.
+// To pointer operands of load and store instructions RThreadAccess adds Thread
+// Pointer Register (rthread), specified by the option -rthread-register as a
+// base to pointers of addrspace specified by the option -rthread-addrspace.
+//
+// For example, the following load and store:
+//
+//   %v1 = load volatile i64, ptr addrspace(256)
+//                            inttoptr (i64 816 to ptr addrspace(256)), align 8
+//   store volatile i64 164, ptr addrspace(255)
+//                           inttoptr(i64 416 to ptr addrspace(256)), align 8
+//
+// are converted to the following (where -rthread-register=r15
+// -rthread-addrspace=256):
+//
+//  %tls1 = call i64 @llvm.read_register.i64(metadata !{!"r15"})
+//  %ptri1 = add i64 %tls1, 816
+//  %tls1.field = inttoptr i64 %ptri1 to ptr
+//  %v1 = load volatile i64, ptr %tls1.field, align 8
+//
+//  %tls2 = call i64 @llvm.read_register.i64(metadata !{!"r15"})
+//  %ptri2 = add i64 %tls2, 416
+//  %tls2.field = inttoptr i64 %ptri2 to ptr
+//  store volatile i64 164, ptr %tls2.field, align 8
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/PassManager.h"
+#include "llvm/Support/CommandLine.h"
+
+extern llvm::cl::opt<std::string> ReservedThreadPointerRegister;
+
+namespace llvm {
+
+class FunctionPass;
+
+FunctionPass *createRThreadAccessPass();
+
+class RThreadAccess : public PassInfoMixin<RThreadAccess> {
+public:
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
+
+  bool runImpl(Function &F);
+};
+
+} // end namespace llvm

llvm/lib/Passes/PassBuilder.cpp

 #include "llvm/Transforms/Scalar/MakeGuardsExplicit.h"
 #include "llvm/Transforms/Scalar/MemCpyOptimizer.h"
 #include "llvm/Transforms/Scalar/MergeICmps.h"
 #include "llvm/Transforms/Scalar/MergedLoadStoreMotion.h"
 #include "llvm/Transforms/Scalar/NaryReassociate.h"
 #include "llvm/Transforms/Scalar/NewGVN.h"
 #include "llvm/Transforms/Scalar/PartiallyInlineLibCalls.h"
 #include "llvm/Transforms/Scalar/PlaceSafepoints.h"
+#include "llvm/Transforms/Scalar/RThreadAccess.h"
 #include "llvm/Transforms/Scalar/Reassociate.h"
 #include "llvm/Transforms/Scalar/Reg2Mem.h"
 #include "llvm/Transforms/Scalar/RewriteStatepointsForGC.h"
 #include "llvm/Transforms/Scalar/SCCP.h"
 #include "llvm/Transforms/Scalar/SROA.h"
 #include "llvm/Transforms/Scalar/ScalarizeMaskedMemIntrin.h"
 #include "llvm/Transforms/Scalar/Scalarizer.h"
 #include "llvm/Transforms/Scalar/SeparateConstOffsetFromGEP.h"

llvm/lib/Passes/PassRegistry.def

 FUNCTION_PASS("print-cfg-sccs", CFGSCCPrinterPass(dbgs()))
 FUNCTION_PASS("print-predicateinfo", PredicateInfoPrinterPass(dbgs()))
 FUNCTION_PASS("print-mustexecute", MustExecutePrinterPass(dbgs()))
 FUNCTION_PASS("print-memderefs", MemDerefPrinterPass(dbgs()))
 FUNCTION_PASS("print<uniformity>", UniformityInfoPrinterPass(dbgs()))
 FUNCTION_PASS("reassociate", ReassociatePass())
 FUNCTION_PASS("redundant-dbg-inst-elim", RedundantDbgInstEliminationPass())
 FUNCTION_PASS("reg2mem", RegToMemPass())
+FUNCTION_PASS("rthread-access", RThreadAccess())
 FUNCTION_PASS("scalarize-masked-mem-intrin", ScalarizeMaskedMemIntrinPass())
 FUNCTION_PASS("scalarizer", ScalarizerPass())
 FUNCTION_PASS("separate-const-offset-from-gep", SeparateConstOffsetFromGEPPass())
 FUNCTION_PASS("sccp", SCCPPass())
 FUNCTION_PASS("sink", SinkingPass())
 FUNCTION_PASS("slp-vectorizer", SLPVectorizerPass())
 FUNCTION_PASS("slsr", StraightLineStrengthReducePass())
 FUNCTION_PASS("speculative-execution", SpeculativeExecutionPass())

llvm/lib/Target/AArch64/AArch64Subtarget.cpp

 #include "GISel/AArch64LegalizerInfo.h"
 #include "GISel/AArch64RegisterBankInfo.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
 #include "llvm/CodeGen/GlobalISel/InstructionSelect.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineScheduler.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/TargetParser/AArch64TargetParser.h"
+#include "llvm/Transforms/Scalar/RThreadAccess.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "aarch64-subtarget"
 
 #define GET_SUBTARGETINFO_CTOR
 #define GET_SUBTARGETINFO_TARGET_DESC
 #include "AArch64GenSubtargetInfo.inc"
case Ampere1A:
     CacheLineSize = 64;
     PrefFunctionAlignment = Align(64);
     PrefLoopAlignment = Align(64);
     MaxInterleaveFactor = 4;
     break;
   }
 }
 
+#define GET_REGISTER_MATCHER
+#include "AArch64GenAsmMatcher.inc"
+
 AArch64Subtarget::AArch64Subtarget(const Triple &TT, StringRef CPU,
                                    StringRef TuneCPU, StringRef FS,
                                    const TargetMachine &TM, bool LittleEndian,
                                    unsigned MinSVEVectorSizeInBitsOverride,
                                    unsigned MaxSVEVectorSizeInBitsOverride,
                                    bool StreamingSVEModeDisabled)
     : AArch64GenSubtargetInfo(TT, CPU, TuneCPU, FS),
       ReserveXRegister(AArch64::GPR64commonRegClass.getNumRegs()),
       ReserveXRegisterForRA(AArch64::GPR64commonRegClass.getNumRegs()),
       CustomCallSavedXRegs(AArch64::GPR64commonRegClass.getNumRegs()),
       IsLittle(LittleEndian),
       StreamingSVEModeDisabled(StreamingSVEModeDisabled),
       MinSVEVectorSizeInBits(MinSVEVectorSizeInBitsOverride),
       MaxSVEVectorSizeInBits(MaxSVEVectorSizeInBitsOverride), TargetTriple(TT),
       InstrInfo(initializeSubtargetDependencies(FS, CPU, TuneCPU)),
       TLInfo(TM, *this) {
   if (AArch64::isX18ReservedByDefault(TT))
     ReserveXRegister.set(18);
 
+  if (!ReservedThreadPointerRegister.empty()) // if defined.
+    if (auto Reg = MatchRegisterName(ReservedThreadPointerRegister.c_str())) {
+      unsigned DwarfRegNum = getRegisterInfo()->getDwarfRegNum(Reg, false);
+      ReserveXRegister.set(DwarfRegNum);
+    }
+
   CallLoweringInfo.reset(new AArch64CallLowering(*getTargetLowering()));
   InlineAsmLoweringInfo.reset(new InlineAsmLowering(getTargetLowering()));
   Legalizer.reset(new AArch64LegalizerInfo(*this));
 
   auto *RBI = new AArch64RegisterBankInfo(*getRegisterInfo());
 
   // FIXME: At this point, we can't rely on Subtarget having RBI.
   // It's awkward to mix passing RBI and the Subtarget; should we pass

llvm/lib/Target/AArch64/AArch64TargetMachine.cpp

 #include "llvm/Pass.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/TargetParser/Triple.h"
 #include "llvm/Transforms/CFGuard.h"
 #include "llvm/Transforms/Scalar.h"
+#include "llvm/Transforms/Scalar/RThreadAccess.h"
 #include <memory>
 #include <optional>
 #include <string>
 
 using namespace llvm;
 
 static cl::opt<bool> EnableCCMP("aarch64-enable-ccmp",
                                 cl::desc("Enable the CCMP formation pass"),
if ((TM->getOptLevel() != CodeGenOpt::None &&
     // because there are some regressions with it also enabled for performance.
     if (!OnlyOptimizeForSize)
       MergeExternalByDefault = false;
 
     addPass(createGlobalMergePass(TM, 4095, OnlyOptimizeForSize,
                                   MergeExternalByDefault));
   }
 
+  if (!ReservedThreadPointerRegister.empty()) // if defined.
+    addPass(createRThreadAccessPass());
+
   return false;
 }
 
 void AArch64PassConfig::addCodeGenPrepare() {
   if (getOptLevel() != CodeGenOpt::None)
     addPass(createTypePromotionLegacyPass());
   TargetPassConfig::addCodeGenPrepare();
 }

llvm/lib/Target/X86/X86ISelLowering.cpp

assert(((FrameReg == X86::RBP && VT == MVT::i64) ||
          "Invalid Frame Register!");
   SDValue FrameAddr = DAG.getCopyFromReg(DAG.getEntryNode(), dl, FrameReg, VT);
   while (Depth--)
     FrameAddr = DAG.getLoad(VT, dl, DAG.getEntryNode(), FrameAddr,
                             MachinePointerInfo());
   return FrameAddr;
 }
 
+#define GET_REGISTER_MATCHER
+#include "X86GenAsmMatcher.inc"
+
 // FIXME? Maybe this could be a TableGen attribute on some registers and
 // this table could be generated automatically from RegInfo.
 Register X86TargetLowering::getRegisterByName(const char* RegName, LLT VT,
                                               const MachineFunction &MF) const {
   const TargetFrameLowering &TFI = *Subtarget.getFrameLowering();
 
   Register Reg = StringSwitch<unsigned>(RegName)
                        .Case("esp", X86::ESP)
                        .Case("rsp", X86::RSP)
                        .Case("ebp", X86::EBP)
                        .Case("rbp", X86::RBP)
                        .Default(0);
 
+  // TODO: Intention of getRegisterByName() is to return only non-allocatable
+  // regs. See LLVM Lang Ref about @lvm.read.register().
+  if (!Reg)
+    Reg = MatchRegisterName(RegName);
+
   if (Reg == X86::EBP || Reg == X86::RBP) {
     if (!TFI.hasFP(MF))
       report_fatal_error("register " + StringRef(RegName) +
                          " is allocatable: function has no frame pointer");
 #ifndef NDEBUG
     else {
       const X86RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
       Register FrameReg = RegInfo->getPtrSizedFrameRegister(MF);

llvm/lib/Target/X86/X86RegisterInfo.cpp

 #include "llvm/CodeGen/VirtRegMap.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/Type.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
+#include "llvm/Transforms/Scalar/RThreadAccess.h"
 
 using namespace llvm;
 
 #define GET_REGINFO_TARGET_DESC
 #include "X86GenRegisterInfo.inc"
 
 static cl::opt<bool>
 EnableBasePointer("x86-use-base-pointer", cl::Hidden, cl::init(true),

 X86RegisterInfo::getNoPreservedMask() const {
   return CSR_NoRegs_RegMask;
 }
 
 const uint32_t *X86RegisterInfo::getDarwinTLSCallPreservedMask() const {
   return CSR_64_TLS_Darwin_RegMask;
 }
 
+#define GET_REGISTER_MATCHER
+#include "X86GenAsmMatcher.inc"
+
 BitVector X86RegisterInfo::getReservedRegs(const MachineFunction &MF) const {
   BitVector Reserved(getNumRegs());
   const X86FrameLowering *TFI = getFrameLowering(MF);
 
   // Set the floating point control register as reserved.
   Reserved.set(X86::FPCW);
 
   // Set the floating point status register as reserved.
for (const MCPhysReg &SubReg : subregs_inclusive(X86::RIP))
     Reserved.set(SubReg);
 
   // Set the frame-pointer register and its aliases as reserved if needed.
   if (TFI->hasFP(MF)) {
     for (const MCPhysReg &SubReg : subregs_inclusive(X86::RBP))
       Reserved.set(SubReg);
   }
 
+  // Set rthread as reserved.
+  if (!ReservedThreadPointerRegister.empty()) {
+    auto Reg = MatchRegisterName(ReservedThreadPointerRegister.c_str());
+    for (const MCPhysReg &SubReg : subregs_inclusive(Reg))
+      Reserved.set(SubReg);
+  }
+
   // Set the base-pointer register and its aliases as reserved if needed.
   if (hasBasePointer(MF)) {
     CallingConv::ID CC = MF.getFunction().getCallingConv();
     const uint32_t *RegMask = getCallPreservedMask(MF, CC);
     if (MachineOperand::clobbersPhysReg(RegMask, getBaseRegister()))
       report_fatal_error(
         "Stack realignment in presence of dynamic allocas is not supported with"
         "this calling convention.");

llvm/lib/Target/X86/X86TargetMachine.cpp

 #include "llvm/Pass.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/TargetParser/Triple.h"
 #include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Scalar/RThreadAccess.h"
 #include <memory>
 #include <optional>
 #include <string>
 
 using namespace llvm;
 
 static cl::opt<bool> EnableMachineCombinerPass("x86-machine-combiner",
                                cl::desc("Enable the machine combiner pass"),
bool X86PassConfig::addILPOpts() {
   addPass(&EarlyIfConverterID);
   if (EnableMachineCombinerPass)
     addPass(&MachineCombinerID);
   addPass(createX86CmovConverterPass());
   return true;
 }
 
 bool X86PassConfig::addPreISel() {
+  if (!ReservedThreadPointerRegister.empty()) // if defined.
+    addPass(createRThreadAccessPass());
+
   // Only add this pass for 32-bit x86 Windows.
   const Triple &TT = TM->getTargetTriple();
   if (TT.isOSWindows() && TT.getArch() == Triple::x86)
     addPass(createX86WinEHStatePass());
   return true;
 }
 
 void X86PassConfig::addPreRegAlloc() {

llvm/lib/Transforms/Scalar/CMakeLists.txt

add_llvm_component_library(LLVMScalarOpts
   MergedLoadStoreMotion.cpp
   NaryReassociate.cpp
   NewGVN.cpp
   PartiallyInlineLibCalls.cpp
   PlaceSafepoints.cpp
   Reassociate.cpp
   Reg2Mem.cpp
   RewriteStatepointsForGC.cpp
+  RThreadAccess.cpp
   SCCP.cpp
   SROA.cpp
   Scalar.cpp
   Scalarizer.cpp
   ScalarizeMaskedMemIntrin.cpp
   SeparateConstOffsetFromGEP.cpp
   SimpleLoopUnswitch.cpp
   SimplifyCFGPass.cpp

llvm/lib/Transforms/Scalar/RThreadAccess.cpp

+//===- RThreadAccess.cpp - rthread-access pass ----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/Scalar/RThreadAccess.h"
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/IR/DataLayout.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InlineAsm.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/Module.h"
+#include "llvm/Pass.h"
+#include "llvm/PassRegistry.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "rthread-access"
+
+llvm::cl::opt<std::string> ReservedThreadPointerRegister(
+    "rthread-register", cl::ZeroOrMore, cl::Hidden,
+    cl::desc("Register that must be hidden from register allocator and is "
+             "preserved across function calls. This register is used as base "
+             "pointer for addrspace(-rthread-addrspace)."));
+
+llvm::cl::opt<unsigned>
+    ThreadPointerAddrspace("rthread-addrspace", cl::init(256), cl::ZeroOrMore,
+                           cl::Hidden,
+                           cl::desc("Addrspace denoting rthread access."));
+
+PreservedAnalyses RThreadAccess::run(Function &F, FunctionAnalysisManager &AM) {
+  bool MadeChange = runImpl(F);
+  if (!MadeChange)
+    return PreservedAnalyses::all();
+  PreservedAnalyses PA;
+  PA.preserveSet<CFGAnalyses>();
+  return PA;
+}
+
+#ifndef NDEBUG
+// Check that constant expression do not cast from
+// addrspace(ThreadPointerAddrspace).
+// Returns error message, null for successful verification.
+static const char *verificationErrorMsg(const ConstantExpr &CE) {
+  if (CE.getOpcode() == Instruction::AddrSpaceCast) {
+    if (CE.getOperand(0)->getType()->getPointerAddressSpace() ==
+            ThreadPointerAddrspace &&
+        CE.getType()->getPointerAddressSpace() != ThreadPointerAddrspace)
+      return "Prohibited AddrSpaceCast from ThreadPointerAddrspace";
+  } else if (CE.getOpcode() == Instruction::PtrToInt) {
+    if (CE.getOperand(0)->getType()->getPointerAddressSpace() ==
+        ThreadPointerAddrspace)
+      return "Prohibited PtrToInt from ThreadPointerAddrspace";
+  }
+
+  for (Value *Op : CE.operands())
+    if (auto *OpCE = dyn_cast<ConstantExpr>(Op))
+      if (auto *Error = verificationErrorMsg(*OpCE))
+        return Error;
+
+  return nullptr; // No error.
+}
+
+// Check that instruction do not cast from addrspace(ThreadPointerAddrspace).
+// Returns error message, null for successful verification.
+static const char *verificationErrorMsg(const Instruction &I) {
+  if (auto *ASCI = dyn_cast<AddrSpaceCastInst>(&I)) {
+    if (ASCI->getSrcAddressSpace() == ThreadPointerAddrspace &&
+        ASCI->getDestAddressSpace() != ThreadPointerAddrspace)
+      return "Prohibited AddrSpaceCast from ThreadPointerAddrspace";
+  } else if (auto *PTI = dyn_cast<PtrToIntInst>(&I)) {
+    if (PTI->getPointerAddressSpace() == ThreadPointerAddrspace)
+      return "Prohibited PtrToInt from ThreadPointerAddrspace";
+  }
+
+  for (Value *Op : I.operands())
+    if (auto *CE = dyn_cast<ConstantExpr>(Op))
+      if (auto *Error = verificationErrorMsg(*CE))
+        return Error;
+
+  return nullptr; // No error.
+}
+#endif // NDEBUG
+
+// One instance of RThreadAccessBuilder is lazily created if at least one
+// modification is needed in the passed function. This one instance is used
+// for all modifications.
+class RThreadAccessBuilder {
+  Module *M;
+  LLVMContext *C;
+  Type *IntptrTy;
+  MDNode *RegName;
+  Function *ReadRegister;
+  SmallVector<Value *, 1> RThreadArgs;
+
+public:
+  RThreadAccessBuilder(Function &F)
+      : M(F.getParent()), C(&M->getContext()),
+        IntptrTy(M->getDataLayout().getIntPtrType(*C, ThreadPointerAddrspace)),
+        RegName(MDNode::get(
+            *C, {MDString::get(*C, ReservedThreadPointerRegister)})),
+        ReadRegister(
+            Intrinsic::getDeclaration(M, Intrinsic::read_register, IntptrTy)),
+        RThreadArgs({MetadataAsValue::get(*C, RegName)}) {}
+
+  Value *build(Instruction &I, Value *Ptr) {
+    IRBuilder<> IRB(&I);
+    Value *RThread = IRB.CreateCall(ReadRegister, RThreadArgs, "tls");
+    Value *Offset = IRB.CreatePtrToInt(Ptr, IntptrTy, "offset");
+    Value *PtrI = IRB.CreateAdd(RThread, Offset, "ptri"); // TODO: nsw nuw?
+    Value *Rebased =
+        IRB.CreateIntToPtr(PtrI,
+                           PointerType::getWithSamePointeeType(
+                               cast<PointerType>(Ptr->getType()), 0),
+                           "tls.field");
+    return Rebased;
+  }
+};
+
+bool RThreadAccess::runImpl(Function &F) {
+  if (ReservedThreadPointerRegister.empty()) // if undef.
+    return false; // Consider not adding this pass to the pipeline.
+
+  std::optional<RThreadAccessBuilder> Builder; // Lazily created.
+
+  for (Instruction &I : instructions(F)) {
+    unsigned POI;
+    if (auto *LI = dyn_cast<LoadInst>(&I)) {
+      POI = LI->getPointerOperandIndex();
+    } else if (auto *SI = dyn_cast<StoreInst>(&I)) {
+      POI = SI->getPointerOperandIndex();
+    } else {
+#ifndef NDEBUG
+      // TODO: Consider moving this check along with the verificationErrorMsg()
+      // methods under the EXPENSIVE_CHECKS macro once tested for a while.
+      if (auto *Error = verificationErrorMsg(I))
+        llvm_unreachable(Error);
+#endif
+      continue;
+    }
+
+    Value *OP = I.getOperand(POI);
+    assert(OP->getType()->isPointerTy());
+    if (OP->getType()->getPointerAddressSpace() != ThreadPointerAddrspace)
+      continue;
+
+    if (!Builder)
+      Builder = RThreadAccessBuilder(F);
+
+    auto *Rebased = Builder->build(I, OP);
+    LLVM_DEBUG(dbgs() << "RThreadAccess: replaced operand " << POI
+                      << " in instruction " << I << "\n    with " << *Rebased
+                      << "\n";);
+    I.setOperand(POI, Rebased);
+  }
+
+  return Builder.has_value();
+}
+
+namespace llvm {
+void initializeRThreadAccessPassPass(PassRegistry &);
+}
+
+class RThreadAccessPass : public FunctionPass {
+  RThreadAccess Pass;
+
+public:
+  static char ID;
+
+  // Default constructor required for the INITIALIZE_PASS macro.
+  RThreadAccessPass() : FunctionPass(ID) {
+    initializeRThreadAccessPassPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnFunction(Function &F) override {
+    if (skipFunction(F))
+      return false;
+    return Pass.runImpl(F);
+  }
+};
+
+char RThreadAccessPass::ID = 0;
+
+INITIALIZE_PASS(RThreadAccessPass, DEBUG_TYPE, "RThreadAccess", false, false)
+
+FunctionPass *llvm::createRThreadAccessPass() {
+  return new RThreadAccessPass();
+}

llvm/test/CodeGen/AArch64/rthread.ll

+; RUN: opt -S -passes=rthread-access -rthread-register=x28 %s | FileCheck --check-prefixes=OPT %s
+; RUN: llc --filetype=asm --march=aarch64 -rthread-register=x28 -o - %s | FileCheck --check-prefixes=ASM %s
+
+define i1 @types() {
+; OPT-LABEL: @types(
+; OPT-NEXT:    [[TLS:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0:![0-9]+]])
+; OPT-NEXT:    [[PTRI:%.*]] = add i64 [[TLS]], 816
+; OPT-NEXT:    [[TLS_FIELD:%.*]] = inttoptr i64 [[PTRI]] to ptr
+; OPT-NEXT:    [[V1:%.*]] = load volatile i64, ptr [[TLS_FIELD]], align 8
+; OPT-NEXT:    [[TLS1:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI2:%.*]] = add i64 [[TLS1]], 824
+; OPT-NEXT:    [[TLS_FIELD3:%.*]] = inttoptr i64 [[PTRI2]] to ptr
+; OPT-NEXT:    [[V2:%.*]] = load volatile i32, ptr [[TLS_FIELD3]], align 8
+; OPT-NEXT:    [[TLS4:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI5:%.*]] = add i64 [[TLS4]], 832
+; OPT-NEXT:    [[TLS_FIELD6:%.*]] = inttoptr i64 [[PTRI5]] to ptr
+; OPT-NEXT:    [[V3:%.*]] = load volatile i16, ptr [[TLS_FIELD6]], align 8
+; OPT-NEXT:    [[TLS7:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI8:%.*]] = add i64 [[TLS7]], 840
+; OPT-NEXT:    [[TLS_FIELD9:%.*]] = inttoptr i64 [[PTRI8]] to ptr
+; OPT-NEXT:    [[V4:%.*]] = load volatile i8, ptr [[TLS_FIELD9]], align 8
+; OPT-NEXT:    [[TLS10:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI11:%.*]] = add i64 [[TLS10]], 848
+; OPT-NEXT:    [[TLS_FIELD12:%.*]] = inttoptr i64 [[PTRI11]] to ptr
+; OPT-NEXT:    [[V5:%.*]] = load volatile i1, ptr [[TLS_FIELD12]], align 8
+; OPT-NEXT:    [[TLS13:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI14:%.*]] = add i64 [[TLS13]], 856
+; OPT-NEXT:    [[TLS_FIELD15:%.*]] = inttoptr i64 [[PTRI14]] to ptr
+; OPT-NEXT:    [[V6:%.*]] = load volatile ptr, ptr [[TLS_FIELD15]], align 8
+; OPT-NEXT:    [[TLS16:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI17:%.*]] = add i64 [[TLS16]], 864
+; OPT-NEXT:    [[TLS_FIELD18:%.*]] = inttoptr i64 [[PTRI17]] to ptr
+; OPT-NEXT:    [[V7:%.*]] = load volatile ptr, ptr [[TLS_FIELD18]], align 8
+; OPT-NEXT:    [[TLS19:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI20:%.*]] = add i64 [[TLS19]], 872
+; OPT-NEXT:    [[TLS_FIELD21:%.*]] = inttoptr i64 [[PTRI20]] to ptr
+; OPT-NEXT:    [[V8:%.*]] = load volatile ptr, ptr [[TLS_FIELD21]], align 8
+; OPT-NEXT:    [[TLS22:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI23:%.*]] = add i64 [[TLS22]], 416
+; OPT-NEXT:    [[TLS_FIELD24:%.*]] = inttoptr i64 [[PTRI23]] to ptr
+; OPT-NEXT:    store volatile i64 164, ptr [[TLS_FIELD24]], align 8
+; OPT-NEXT:    [[TLS25:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI26:%.*]] = add i64 [[TLS25]], 424
+; OPT-NEXT:    [[TLS_FIELD27:%.*]] = inttoptr i64 [[PTRI26]] to ptr
+; OPT-NEXT:    store volatile i32 132, ptr [[TLS_FIELD27]], align 8
+; OPT-NEXT:    [[TLS28:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI29:%.*]] = add i64 [[TLS28]], 432
+; OPT-NEXT:    [[TLS_FIELD30:%.*]] = inttoptr i64 [[PTRI29]] to ptr
+; OPT-NEXT:    store volatile i16 116, ptr [[TLS_FIELD30]], align 8
+; OPT-NEXT:    [[TLS31:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI32:%.*]] = add i64 [[TLS31]], 440
+; OPT-NEXT:    [[TLS_FIELD33:%.*]] = inttoptr i64 [[PTRI32]] to ptr
+; OPT-NEXT:    store volatile i8 18, ptr [[TLS_FIELD33]], align 8
+; OPT-NEXT:    [[TLS34:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI35:%.*]] = add i64 [[TLS34]], 448
+; OPT-NEXT:    [[TLS_FIELD36:%.*]] = inttoptr i64 [[PTRI35]] to ptr
+; OPT-NEXT:    store volatile i1 true, ptr [[TLS_FIELD36]], align 8
+; OPT-NEXT:    [[TLS37:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI38:%.*]] = add i64 [[TLS37]], 456
+; OPT-NEXT:    [[TLS_FIELD39:%.*]] = inttoptr i64 [[PTRI38]] to ptr
+; OPT-NEXT:    store volatile ptr null, ptr [[TLS_FIELD39]], align 8
+; OPT-NEXT:    [[TLS40:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI41:%.*]] = add i64 [[TLS40]], 464
+; OPT-NEXT:    [[TLS_FIELD42:%.*]] = inttoptr i64 [[PTRI41]] to ptr
+; OPT-NEXT:    store volatile ptr null, ptr [[TLS_FIELD42]], align 8
+; OPT-NEXT:    [[TLS43:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI44:%.*]] = add i64 [[TLS43]], 472
+; OPT-NEXT:    [[TLS_FIELD45:%.*]] = inttoptr i64 [[PTRI44]] to ptr
+; OPT-NEXT:    store volatile ptr null, ptr [[TLS_FIELD45]], align 8
+;
+; ASM-LABEL: types:
+; ASM:    ldr x{{[0-9]+}}, [x28, #816]
+; ASM:    ldr w{{[0-9]+}}, [x28, #824]
+; ASM:    ldrh w{{[0-9]+}}, [x28, #832]
+; ASM:    ldrb w{{[0-9]+}}, [x28, #840]
+; ASM:    ldrb w{{[0-9]+}}, [x28, #848]
+; ASM:    ldr x{{[0-9]+}}, [x28, #856]
+; ASM:    ldr x{{[0-9]+}}, [x28, #864]
+; ASM:    ldr x{{[0-9]+}}, [x28, #872]
+; ASM:    str x{{[0-9]+}}, [x28, #416]
+; ASM:    str w{{[0-9]+}}, [x28, #424]
+; ASM:    strh w{{[0-9]+}}, [x28, #432]
+; ASM:    strb w{{[0-9]+}}, [x28, #440]
+; ASM:    strb w{{[0-9]+}}, [x28, #448]
+; ASM:    str xzr, [x28, #456]
+; ASM:    str xzr, [x28, #464]
+; ASM:    str xzr, [x28, #472]
+  %v1 = load volatile i64, ptr addrspace(256) inttoptr (i64 816 to ptr addrspace(256)), align 8
+  %v2 = load volatile i32, ptr addrspace(256) inttoptr (i64 824 to ptr addrspace(256)), align 8
+  %v3 = load volatile i16, ptr addrspace(256) inttoptr (i64 832 to ptr addrspace(256)), align 8
+  %v4 = load volatile i8, ptr addrspace(256) inttoptr (i64 840 to ptr addrspace(256)), align 8
+  %v5 = load volatile i1, ptr addrspace(256) inttoptr (i64 848 to ptr addrspace(256)), align 8
+  %v6 = load volatile ptr, ptr addrspace(256) inttoptr (i64 856 to ptr addrspace(256)), align 8
+  %v7 = load volatile ptr, ptr addrspace(256) inttoptr (i64 864 to ptr addrspace(256)), align 8
+  %v8 = load volatile ptr, ptr addrspace(256) inttoptr (i64 872 to ptr addrspace(256)), align 8
+
+  store volatile i64 164,  ptr addrspace(256) inttoptr (i64 416 to ptr addrspace(256)), align 8
+  store volatile i32 132,  ptr addrspace(256) inttoptr (i64 424 to ptr addrspace(256)), align 8
+  store volatile i16 116,  ptr addrspace(256) inttoptr (i64 432 to ptr addrspace(256)), align 8
+  store volatile i8   18,  ptr addrspace(256) inttoptr (i64 440 to ptr addrspace(256)), align 8
+  store volatile i1    1,  ptr addrspace(256) inttoptr (i64 448 to ptr addrspace(256)), align 8
+  store volatile ptr null, ptr addrspace(256) inttoptr (i64 456 to ptr addrspace(256)), align 8
+  store volatile ptr null, ptr addrspace(256) inttoptr (i64 464 to ptr addrspace(256)), align 8
+  store volatile ptr null, ptr addrspace(256) inttoptr (i64 472 to ptr addrspace(256)), align 8
+
+  ; just collect results
+  %x1 = icmp eq i64 %v1, 0
+  %x2 = icmp eq i32 %v2, 0
+  %x3 = icmp eq i16 %v3, 0
+  %x4 = icmp eq i8  %v4, 0
+  %x5 = icmp eq i1  %v5, 0
+  %x6 = icmp eq ptr %v6, null
+  %x7 = icmp eq ptr %v7, null
+  %x8 = icmp eq ptr %v8, null
+  %y2 = and i1 %x1, %x2
+  %y3 = and i1 %y2, %x3
+  %y4 = and i1 %y3, %x4
+  %y5 = and i1 %y4, %x5
+  %y6 = and i1 %y5, %x6
+  %y7 = and i1 %y6, %x7
+  %y8 = and i1 %y7, %x8
+
+  ret i1 %y8
+}
+
+define i1 @index_limits() {
+; OPT-LABEL: @index_limits(
+; OPT-NEXT:    [[TLS:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI:%.*]] = add i64 [[TLS]], -256
+; OPT-NEXT:    [[TLS_FIELD:%.*]] = inttoptr i64 [[PTRI]] to ptr
+; OPT-NEXT:    [[V8A:%.*]] = load volatile i8, ptr [[TLS_FIELD]], align 8
+; OPT-NEXT:    [[TLS1:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI2:%.*]] = add i64 [[TLS1]], 4095
+; OPT-NEXT:    [[TLS_FIELD3:%.*]] = inttoptr i64 [[PTRI2]] to ptr
+; OPT-NEXT:    [[V8B:%.*]] = load volatile i8, ptr [[TLS_FIELD3]], align 8
+; OPT-NEXT:    [[TLS4:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI5:%.*]] = add i64 [[TLS4]], -256
+; OPT-NEXT:    [[TLS_FIELD6:%.*]] = inttoptr i64 [[PTRI5]] to ptr
+; OPT-NEXT:    [[V16A:%.*]] = load volatile i16, ptr [[TLS_FIELD6]], align 8
+; OPT-NEXT:    [[TLS7:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI8:%.*]] = add i64 [[TLS7]], 8190
+; OPT-NEXT:    [[TLS_FIELD9:%.*]] = inttoptr i64 [[PTRI8]] to ptr
+; OPT-NEXT:    [[V16B:%.*]] = load volatile i16, ptr [[TLS_FIELD9]], align 8
+; OPT-NEXT:    [[TLS10:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI11:%.*]] = add i64 [[TLS10]], -256
+; OPT-NEXT:    [[TLS_FIELD12:%.*]] = inttoptr i64 [[PTRI11]] to ptr
+; OPT-NEXT:    [[V32A:%.*]] = load volatile i32, ptr [[TLS_FIELD12]], align 8
+; OPT-NEXT:    [[TLS13:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI14:%.*]] = add i64 [[TLS13]], 16380
+; OPT-NEXT:    [[TLS_FIELD15:%.*]] = inttoptr i64 [[PTRI14]] to ptr
+; OPT-NEXT:    [[V32B:%.*]] = load volatile i32, ptr [[TLS_FIELD15]], align 8
+; OPT-NEXT:    [[TLS16:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI17:%.*]] = add i64 [[TLS16]], -256
+; OPT-NEXT:    [[TLS_FIELD18:%.*]] = inttoptr i64 [[PTRI17]] to ptr
+; OPT-NEXT:    [[V64A:%.*]] = load volatile i64, ptr [[TLS_FIELD18]], align 8
+; OPT-NEXT:    [[TLS19:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI20:%.*]] = add i64 [[TLS19]], 32760
+; OPT-NEXT:    [[TLS_FIELD21:%.*]] = inttoptr i64 [[PTRI20]] to ptr
+; OPT-NEXT:    [[V64B:%.*]] = load volatile i64, ptr [[TLS_FIELD21]], align 8
+;
+; ASM-LABEL: index_limits:
+; ASM:    ldurb w{{[0-9]+}}, [x28, #-256]
+; ASM:    ldrb w{{[0-9]+}}, [x28, #4095]
+; ASM:    ldurh w{{[0-9]+}}, [x28, #-256]
+; ASM:    ldrh w{{[0-9]+}}, [x28, #8190]
+; ASM:    ldur w{{[0-9]+}}, [x28, #-256]
+; ASM:    ldr w{{[0-9]+}}, [x28, #16380]
+; ASM:    ldur x{{[0-9]+}}, [x28, #-256]
+; ASM:    ldr x{{[0-9]+}}, [x28, #32760]
+  %v8a = load volatile i8, ptr addrspace(256) inttoptr (i64 -256 to ptr addrspace(256)), align 8
+  %v8b = load volatile i8, ptr addrspace(256) inttoptr (i64 4095 to ptr addrspace(256)), align 8
+  %v16a = load volatile i16, ptr addrspace(256) inttoptr (i64 -256 to ptr addrspace(256)), align 8
+  %v16b = load volatile i16, ptr addrspace(256) inttoptr (i64 8190 to ptr addrspace(256)), align 8
+  %v32a = load volatile i32, ptr addrspace(256) inttoptr (i64 -256 to ptr addrspace(256)), align 8
+  %v32b = load volatile i32, ptr addrspace(256) inttoptr (i64 16380 to ptr addrspace(256)), align 8
+  %v64a = load volatile i64, ptr addrspace(256) inttoptr (i64 -256 to ptr addrspace(256)), align 8
+  %v64b = load volatile i64, ptr addrspace(256) inttoptr (i64 32760 to ptr addrspace(256)), align 8
+
+  ; just collect results
+  %x8 = icmp eq i8 %v8a, %v8b
+  %x16 = icmp eq i16 %v16a, %v16b
+  %x32 = icmp eq i32 %v32a, %v32b
+  %x64 = icmp eq i64 %v64a, %v64b
+  %y1 = xor i1 %x8, %x16
+  %y2 = xor i1 %x32, %x64
+  %y3 = xor i1 %y1, %y2
+
+  ret i1 %y3
+}
+
+; addrspace(256) must not be cast to int but
+; this case does not fail because LLVM parser translates it to "ret i64 816".
+define i64 @cast-from-p256-to-i64() {
+; OPT-LABEL: @cast-from-p256-to-i64
+; OPT-NEXT: ret i64 816
+  ret i64 ptrtoint (ptr addrspace(256) inttoptr (i64 816 to ptr addrspace(256)) to i64)
+}
+
+; No cast constraints for addrspace(257).
+define void @cast-from-p257-to-i64() {
+; OPT-LABEL: @cast-from-p257-to-i64
+; OPT-NEXT: ptrtoint
+  %v = ptrtoint ptr addrspace(257) null to i64
+  ret void
+}
+
+define void @cast-from-p257-to-p256() {
+; OPT-LABEL: @cast-from-p257-to-p256
+; OPT-NEXT: addrspacecast
+  %v = addrspacecast ptr addrspace(257) null to ptr addrspace(256)
+  ret void
+}
+

llvm/test/CodeGen/X86/named-reg-alloc.ll

 ; RUN: not --crash llc < %s -mtriple=x86_64-apple-darwin 2>&1 | FileCheck %s
 ; RUN: not --crash llc < %s -mtriple=x86_64-linux-gnueabi 2>&1 | FileCheck %s
 
 define i32 @get_stack() nounwind {
 entry:
 ; FIXME: Include an allocatable-specific error message
 ; CHECK: Invalid register name global variable
   %sp = call i32 @llvm.read_register.i32(metadata !0)
   ret i32 %sp
 }
 
 declare i32 @llvm.read_register.i32(metadata) nounwind
 
-!0 = !{!"eax\00"}
+!0 = !{!"InvalidRegister\00"}

llvm/test/CodeGen/X86/rthread.ll

+; RUN: opt -S -passes=rthread-access -rthread-addrspace=255 -rthread-register=r15 %s | FileCheck --check-prefixes=OPT %s
+; RUN: llc --filetype=asm --march=x86-64 -rthread-addrspace=255 -rthread-register=r15 -o - %s | FileCheck --check-prefixes=ASM %s
+
+define i1 @types() {
+; OPT-LABEL: @types(
+; OPT-NEXT:    [[TLS:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0:![0-9]+]])
+; OPT-NEXT:    [[PTRI:%.*]] = add i64 [[TLS]], 816
+; OPT-NEXT:    [[TLS_FIELD:%.*]] = inttoptr i64 [[PTRI]] to ptr
+; OPT-NEXT:    [[V1:%.*]] = load volatile i64, ptr [[TLS_FIELD]], align 8
+; OPT-NEXT:    [[TLS1:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI2:%.*]] = add i64 [[TLS1]], 824
+; OPT-NEXT:    [[TLS_FIELD3:%.*]] = inttoptr i64 [[PTRI2]] to ptr
+; OPT-NEXT:    [[V2:%.*]] = load volatile i32, ptr [[TLS_FIELD3]], align 8
+; OPT-NEXT:    [[TLS4:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI5:%.*]] = add i64 [[TLS4]], 832
+; OPT-NEXT:    [[TLS_FIELD6:%.*]] = inttoptr i64 [[PTRI5]] to ptr
+; OPT-NEXT:    [[V3:%.*]] = load volatile i16, ptr [[TLS_FIELD6]], align 8
+; OPT-NEXT:    [[TLS7:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI8:%.*]] = add i64 [[TLS7]], 840
+; OPT-NEXT:    [[TLS_FIELD9:%.*]] = inttoptr i64 [[PTRI8]] to ptr
+; OPT-NEXT:    [[V4:%.*]] = load volatile i8, ptr [[TLS_FIELD9]], align 8
+; OPT-NEXT:    [[TLS10:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI11:%.*]] = add i64 [[TLS10]], 848
+; OPT-NEXT:    [[TLS_FIELD12:%.*]] = inttoptr i64 [[PTRI11]] to ptr
+; OPT-NEXT:    [[V5:%.*]] = load volatile i1, ptr [[TLS_FIELD12]], align 8
+; OPT-NEXT:    [[TLS13:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI14:%.*]] = add i64 [[TLS13]], 856
+; OPT-NEXT:    [[TLS_FIELD15:%.*]] = inttoptr i64 [[PTRI14]] to ptr
+; OPT-NEXT:    [[V6:%.*]] = load volatile ptr, ptr [[TLS_FIELD15]], align 8
+; OPT-NEXT:    [[TLS16:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI17:%.*]] = add i64 [[TLS16]], 864
+; OPT-NEXT:    [[TLS_FIELD18:%.*]] = inttoptr i64 [[PTRI17]] to ptr
+; OPT-NEXT:    [[V7:%.*]] = load volatile ptr, ptr [[TLS_FIELD18]], align 8
+; OPT-NEXT:    [[TLS19:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI20:%.*]] = add i64 [[TLS19]], 872
+; OPT-NEXT:    [[TLS_FIELD21:%.*]] = inttoptr i64 [[PTRI20]] to ptr
+; OPT-NEXT:    [[V8:%.*]] = load volatile ptr, ptr [[TLS_FIELD21]], align 8
+; OPT-NEXT:    [[TLS22:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI23:%.*]] = add i64 [[TLS22]], 416
+; OPT-NEXT:    [[TLS_FIELD24:%.*]] = inttoptr i64 [[PTRI23]] to ptr
+; OPT-NEXT:    store volatile i64 164, ptr [[TLS_FIELD24]], align 8
+; OPT-NEXT:    [[TLS25:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI26:%.*]] = add i64 [[TLS25]], 424
+; OPT-NEXT:    [[TLS_FIELD27:%.*]] = inttoptr i64 [[PTRI26]] to ptr
+; OPT-NEXT:    store volatile i32 132, ptr [[TLS_FIELD27]], align 8
+; OPT-NEXT:    [[TLS28:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI29:%.*]] = add i64 [[TLS28]], 432
+; OPT-NEXT:    [[TLS_FIELD30:%.*]] = inttoptr i64 [[PTRI29]] to ptr
+; OPT-NEXT:    store volatile i16 116, ptr [[TLS_FIELD30]], align 8
+; OPT-NEXT:    [[TLS31:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI32:%.*]] = add i64 [[TLS31]], 440
+; OPT-NEXT:    [[TLS_FIELD33:%.*]] = inttoptr i64 [[PTRI32]] to ptr
+; OPT-NEXT:    store volatile i8 18, ptr [[TLS_FIELD33]], align 8
+; OPT-NEXT:    [[TLS34:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI35:%.*]] = add i64 [[TLS34]], 448
+; OPT-NEXT:    [[TLS_FIELD36:%.*]] = inttoptr i64 [[PTRI35]] to ptr
+; OPT-NEXT:    store volatile i1 true, ptr [[TLS_FIELD36]], align 8
+; OPT-NEXT:    [[TLS37:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI38:%.*]] = add i64 [[TLS37]], 456
+; OPT-NEXT:    [[TLS_FIELD39:%.*]] = inttoptr i64 [[PTRI38]] to ptr
+; OPT-NEXT:    store volatile ptr null, ptr [[TLS_FIELD39]], align 8
+; OPT-NEXT:    [[TLS40:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI41:%.*]] = add i64 [[TLS40]], 464
+; OPT-NEXT:    [[TLS_FIELD42:%.*]] = inttoptr i64 [[PTRI41]] to ptr
+; OPT-NEXT:    store volatile ptr null, ptr [[TLS_FIELD42]], align 8
+; OPT-NEXT:    [[TLS43:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI44:%.*]] = add i64 [[TLS43]], 472
+; OPT-NEXT:    [[TLS_FIELD45:%.*]] = inttoptr i64 [[PTRI44]] to ptr
+; OPT-NEXT:    store volatile ptr null, ptr [[TLS_FIELD45]], align 8
+;
+; ASM-LABEL: types:
+; ASM:    cmpq    $0, 816(%r15)
+; ASM:    cmpl    $0, 824(%r15)
+; ASM:    cmpw    $0, 832(%r15)
+; ASM:    cmpb    $0, 840(%r15)
+; ASM:    movzbl  848(%r15),
+; ASM:    cmpq    $0, 856(%r15)
+; ASM:    cmpq    $0, 864(%r15)
+; ASM:    cmpq    $0, 872(%r15)
+; ASM:    movq    $164, 416(%r15)
+; ASM:    movl    $132, 424(%r15)
+; ASM:    movw    $116, 432(%r15)
+; ASM:    movb    $18, 440(%r15)
+; ASM:    movb    $1, 448(%r15)
+; ASM:    movq    $0, 456(%r15)
+; ASM:    movq    $0, 464(%r15)
+; ASM:    movq    $0, 472(%r15)
+  %v1 = load volatile i64, ptr addrspace(255) inttoptr (i64 816 to ptr addrspace(255)), align 8
+  %v2 = load volatile i32, ptr addrspace(255) inttoptr (i64 824 to ptr addrspace(255)), align 8
+  %v3 = load volatile i16, ptr addrspace(255) inttoptr (i64 832 to ptr addrspace(255)), align 8
+  %v4 = load volatile i8 , ptr addrspace(255) inttoptr (i64 840 to ptr addrspace(255)), align 8
+  %v5 = load volatile i1 , ptr addrspace(255) inttoptr (i64 848 to ptr addrspace(255)), align 8
+  %v6 = load volatile ptr, ptr addrspace(255) inttoptr (i64 856 to ptr addrspace(255)), align 8
+  %v7 = load volatile ptr , ptr  addrspace(255) inttoptr (i64 864 to ptr  addrspace(255)), align 8
+  %v8 = load volatile ptr , ptr  addrspace(255) inttoptr (i64 872 to ptr  addrspace(255)), align 8
+
+  store volatile i64 164,    ptr addrspace(255) inttoptr (i64 416 to ptr addrspace(255)), align 8
+  store volatile i32 132,    ptr addrspace(255) inttoptr (i64 424 to ptr addrspace(255)), align 8
+  store volatile i16 116,    ptr addrspace(255) inttoptr (i64 432 to ptr addrspace(255)), align 8
+  store volatile i8   18,    ptr addrspace(255) inttoptr (i64 440 to ptr addrspace(255)), align 8
+  store volatile i1    1,    ptr addrspace(255) inttoptr (i64 448 to ptr addrspace(255)), align 8
+  store volatile ptr null,  ptr addrspace(255) inttoptr (i64 456 to ptr addrspace(255)), align 8
+  store volatile ptr  null,  ptr  addrspace(255) inttoptr (i64 464 to ptr  addrspace(255)), align 8
+  store volatile ptr  null,  ptr  addrspace(255) inttoptr (i64 472 to ptr  addrspace(255)), align 8
+
+  ; just collect results
+  %x1 = icmp eq i64 %v1, 0
+  %x2 = icmp eq i32 %v2, 0
+  %x3 = icmp eq i16 %v3, 0
+  %x4 = icmp eq i8  %v4, 0
+  %x5 = icmp eq i1  %v5, 0
+  %x6 = icmp eq ptr %v6, null
+  %x7 = icmp eq ptr %v7, null
+  %x8 = icmp eq ptr %v8, null
+  %y2 = and i1 %x1, %x2
+  %y3 = and i1 %y2, %x3
+  %y4 = and i1 %y3, %x4
+  %y5 = and i1 %y4, %x5
+  %y6 = and i1 %y5, %x6
+  %y7 = and i1 %y6, %x7
+  %y8 = and i1 %y7, %x8
+
+  ret i1 %y8
+}
+
+define i1 @index_limits() {
+; OPT-LABEL: @index_limits(
+; OPT-NEXT:    [[TLS:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI:%.*]] = add i64 [[TLS]], -255
+; OPT-NEXT:    [[TLS_FIELD:%.*]] = inttoptr i64 [[PTRI]] to ptr
+; OPT-NEXT:    [[V8A:%.*]] = load volatile i8, ptr [[TLS_FIELD]], align 8
+; OPT-NEXT:    [[TLS1:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI2:%.*]] = add i64 [[TLS1]], 4095
+; OPT-NEXT:    [[TLS_FIELD3:%.*]] = inttoptr i64 [[PTRI2]] to ptr
+; OPT-NEXT:    [[V8B:%.*]] = load volatile i8, ptr [[TLS_FIELD3]], align 8
+; OPT-NEXT:    [[TLS4:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI5:%.*]] = add i64 [[TLS4]], -255
+; OPT-NEXT:    [[TLS_FIELD6:%.*]] = inttoptr i64 [[PTRI5]] to ptr
+; OPT-NEXT:    [[V16A:%.*]] = load volatile i16, ptr [[TLS_FIELD6]], align 8
+; OPT-NEXT:    [[TLS7:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI8:%.*]] = add i64 [[TLS7]], 8190
+; OPT-NEXT:    [[TLS_FIELD9:%.*]] = inttoptr i64 [[PTRI8]] to ptr
+; OPT-NEXT:    [[V16B:%.*]] = load volatile i16, ptr [[TLS_FIELD9]], align 8
+; OPT-NEXT:    [[TLS10:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI11:%.*]] = add i64 [[TLS10]], -255
+; OPT-NEXT:    [[TLS_FIELD12:%.*]] = inttoptr i64 [[PTRI11]] to ptr
+; OPT-NEXT:    [[V32A:%.*]] = load volatile i32, ptr [[TLS_FIELD12]], align 8
+; OPT-NEXT:    [[TLS13:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI14:%.*]] = add i64 [[TLS13]], 16380
+; OPT-NEXT:    [[TLS_FIELD15:%.*]] = inttoptr i64 [[PTRI14]] to ptr
+; OPT-NEXT:    [[V32B:%.*]] = load volatile i32, ptr [[TLS_FIELD15]], align 8
+; OPT-NEXT:    [[TLS16:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI17:%.*]] = add i64 [[TLS16]], -255
+; OPT-NEXT:    [[TLS_FIELD18:%.*]] = inttoptr i64 [[PTRI17]] to ptr
+; OPT-NEXT:    [[V64A:%.*]] = load volatile i64, ptr [[TLS_FIELD18]], align 8
+; OPT-NEXT:    [[TLS19:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]])
+; OPT-NEXT:    [[PTRI20:%.*]] = add i64 [[TLS19]], 32760
+; OPT-NEXT:    [[TLS_FIELD21:%.*]] = inttoptr i64 [[PTRI20]] to ptr
+; OPT-NEXT:    [[V64B:%.*]] = load volatile i64, ptr [[TLS_FIELD21]], align 8
+  %v8a = load volatile i8 , ptr addrspace(255) inttoptr (i64 -255 to ptr addrspace(255)), align 8
+  %v8b = load volatile i8 , ptr addrspace(255) inttoptr (i64 4095 to ptr addrspace(255)), align 8
+  %v16a = load volatile i16 , ptr addrspace(255) inttoptr (i64 -255 to ptr addrspace(255)), align 8
+  %v16b = load volatile i16 , ptr addrspace(255) inttoptr (i64 8190 to ptr addrspace(255)), align 8
+  %v32a = load volatile i32 , ptr addrspace(255) inttoptr (i64 -255 to ptr addrspace(255)), align 8
+  %v32b = load volatile i32 , ptr addrspace(255) inttoptr (i64 16380 to ptr addrspace(255)), align 8
+  %v64a = load volatile i64 , ptr addrspace(255) inttoptr (i64 -255 to ptr addrspace(255)), align 8
+  %v64b = load volatile i64 , ptr addrspace(255) inttoptr (i64 32760 to ptr addrspace(255)), align 8
+
+  ; just collect results
+  %x8 = icmp eq i8 %v8a, %v8b
+  %x16 = icmp eq i16 %v16a, %v16b
+  %x32 = icmp eq i32 %v32a, %v32b
+  %x64 = icmp eq i64 %v64a, %v64b
+  %y1 = xor i1 %x8, %x16
+  %y2 = xor i1 %x32, %x64
+  %y3 = xor i1 %y1, %y2
+
+  ret i1 %y3
+}
+
+; addrspace(255)* must not be cast to int but
+; this case does not fail because LLVM parser translates it to "ret i64 816".
+define i64 @cast-from-p255-to-i64() {
+; OPT-LABEL: @cast-from-p255-to-i64
+; OPT-NEXT: ret i64 816
+  ret i64 ptrtoint (ptr addrspace(255) inttoptr (i64 816 to ptr addrspace(255)) to i64)
+}
+
+; No cast constraints for addrspace(257).
+define void @cast-from-p257-to-i64() {
+; OPT-LABEL: @cast-from-p257-to-i64
+; OPT-NEXT: ptrtoint
+  %v = ptrtoint ptr addrspace(257) null to i64
+  ret void
+}
+
+define void @cast-from-p257-to-p255() {
+; OPT-LABEL: @cast-from-p257-to-p255
+; OPT-NEXT: addrspacecast
+  %v = addrspacecast ptr addrspace(257) null to ptr addrspace(255)
+  ret void
+}
+