[WPD] Use new llvm.public.type.test intrinsic for potentially publicly visible classes

[aeubanks]

Summary

Turning on opaque pointers has uncovered an issue with WPD where we currently pattern match away assume(type.test) in WPD so that a later LTT doesn't resolve the type test to undef and introduce an assume(false). The pattern matching can fail in cases where we transform two assume(type.test)s into assume(phi(type.test.1, type.test.2)).

Currently we create assume(type.test) for all virtual calls that might be devirtualized. This is to support -Wl,--lto-whole-program-visibility.

To prevent this, all virtual calls that may not be in the same LTO module instead use a new llvm.public.type.test intrinsic in place of the llvm.type.test. Then when we know if -Wl,--lto-whole-program-visibility is passed or not, we can either replace all llvm.public.type.test with llvm.type.test, or replace all llvm.public.type.test with true. This prevents WPD from trying to pattern match away assume(type.test) for public virtual calls when failing the pattern matching will result in miscompiles.

[aeubanks]

update

[aeubanks]

update

[aeubanks]

make sure not to regress windows by checking HasHiddenLTOVisibility() to decide whether to emit public.type.test vs type.test

[aeubanks]

I did verify that -fvisibility=hidden still devirtualizes, and same with -Wl,--lto-whole-program-visibility, but not if both are missing

llvm/lib/LTO/LTOBackend.cpp

576

I'm not sure where the best place to put these is

[tejohnson]

Thanks for implementing this!

In D71913, I also modified this code, which might need something similar: https://github.com/llvm/llvm-project/blob/59afc4038b1096bc6fea7b322091f6e5e2dc0b38/clang/lib/CodeGen/ItaniumCXXABI.cpp#L707-L713

clang/test/CodeGenCXX/thinlto-distributed-type-metadata.cpp

22 ↗

(On Diff #444706)

Why remove this one here and below?

clang/test/CodeGenCXX/type-metadata.cpp

148 ↗

(On Diff #444706)

It's a little confusing to distinguish the results by prefixing them as "CFI" or "VTABLE". Afaict the behavior is determined by whether -fvisibility=hidden has been passed in all cases for Itanium.

And MS is always hidden from what I see here? So in that case no need to split the expected results into two copies.

lld/test/ELF/lto/update_public_type_test.ll

1 ↗

(On Diff #444706)

Add comments about what is being tested. Also good to test via llvm-lto2 which has a similar -whole-program-visibility option, rather than just via lld here. See for example llvm/test/ThinLTO/X86/devirt_vcall_vis_public.ll.

llvm/lib/LTO/LTOBackend.cpp

576

I don't think that this will work for distributed ThinLTO, where the ThinLTO Backends are run via clang, which isn't going to have this config variable set (since it is set only during LTO linking). I think something may need to be recorded in the index as a flag there at thin link / indexing time that can be checked here.

It would then be nice to consolidate this handling into WPD itself, e.g. at the start of DevirtModule::run, but unfortunately we won't have a summary index for pure regular LTO WPD so I don't think that would work. So in here is ok but I would do it early to handle some of the early return cases.

(Please add a distributed ThinLTO test too)

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

1849 ↗

(On Diff #444706)

Shouldn't we have converted all of these by now?

[aeubanks]

update

[aeubanks]

clang/test/CodeGenCXX/thinlto-distributed-type-metadata.cpp

22 ↗

(On Diff #444706)

we end up RAUWing the public.type.test with true and end up with assume(true) which is harmless and gets removed by something like instcombine

clang/test/CodeGenCXX/type-metadata.cpp

148 ↗

(On Diff #444706)

not sure why I had CFI and WPD as acting differently in my mind, fixed

lld/test/ELF/lto/update_public_type_test.ll

1 ↗

(On Diff #444706)

added comments

I extended llvm/test/ThinLTO/X86/devirt_vcall_vis_public.ll, is that ok?

llvm/lib/LTO/LTOBackend.cpp

576

Added a distributed ThinLTO test: clang/test/CodeGenCXX/thinlto_public_type_test_distributed.ll.

I added ModuleSummaryIndex::WithWholeProgramVisibility but I'm not sure where I'd set it, and that's causing the test to fail.

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

1849 ↗

(On Diff #444706)

Hmm, I vaguely remember adding this because some clang test failed (something to do with the extra LTT we add in BackendUtil.cpp), but reverting this doesn't cause any failures.

[aeubanks]

update

llvm/lib/LTO/LTOBackend.cpp

576

ah, I needed a separate WPV flag in llvm-lto2 to set llvm::lto::Config, now everything passes

[tejohnson]

clang/test/CodeGenCXX/thinlto-distributed-type-metadata.cpp

22 ↗

(On Diff #444706)

Sure, but it should still be gone and can confirm that here? If it isn't useful to check for, then the comment should probably be changed too since it mentions assumes.

clang/test/CodeGenCXX/thinlto_public_type_test_distributed.ll

23 ↗

(On Diff #445978)

Is there a reason why the hidden version checks after promote but the public version above checks after importing?

clang/test/CodeGenCXX/type-metadata.cpp

212 ↗

(On Diff #445978)

Is the CFI-VT-ITANIUM and CFI-VT-MS split needed? I don't see any differences in their checking. Can we go back to just a single CFI-VT here?

lld/test/ELF/lto/update_public_type_test.ll

1 ↗

(On Diff #444706)

Yeah sure that's fine. But maybe also add a direct check there in both cases after promote that the public.type.test/assume were transformed as expected as you do in this test.

llvm/lib/Analysis/ModuleSummaryAnalysis.cpp

169

Remove commented debugging code here and below

llvm/lib/LTO/LTO.cpp

1106 ↗

(On Diff #445978)

Probably need to do for the legacy LTO API as well, which also calls updateVCallVisibilityInModule (LTOCodeGenerator.cpp). This is used both some other linkers such as ld64. Otherwise I think the public.type.test intrinsics will never get converted for those LTO users, leading to errors probably? You can test the legacy LTO via llvm-lto (e.g. see tests in llvm/test/LTO/X86).

llvm/lib/LTO/LTOBackend.cpp

546

Can we just check the combined index flag now?

576

I see from your most recent update that you added code to set/check this new index flag. But I think you would want to set it around the same place where we update the vcall visibility during the thin link (see calls to updateVCallVisibilityInIndex). And I would do it via a new method in the WPD source file that uses the existing hasWholeProgramVisibility() that checks the OR of the config flag or the internal option. Then you don't need to add a new flag to llvm-lto2 for this purpose.

Also, as I noted for the regular LTO handling elsewhere, I think you need to add similar handling for the legacy LTO implementation. See other callers to updateVCallVisibilityInIndex.

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

1849 ↗

(On Diff #444706)

Related to some of my other comments about ensuring we do the conversion for the legacy LTO interfaces, perhaps this we should assert here if there are any remaining public.type.test intrinsics? Not sure what the failure mode is if any of those stick around otherwise.

[aeubanks]

update

[aeubanks]

the assert that there are no public.type.tests in LTT fails on CodeGenCXX/thinlto-distributed-type-metadata.cpp. for some reason clang doesn't go through the LTO API at [1], it just ends up calling the normal optimization pipeline. any ideas why?

[1] https://github.com/llvm/llvm-project/blob/0c1b32717bcffcf8edf95294e98933bd4c1e76ed/clang/lib/CodeGen/BackendUtil.cpp#L1173

clang/test/CodeGenCXX/thinlto-distributed-type-metadata.cpp

22 ↗

(On Diff #444706)

There's an -O0 RUN line below where the assume(true) won't be removed.
Updated comments.

clang/test/CodeGenCXX/thinlto_public_type_test_distributed.ll

23 ↗

(On Diff #445978)

unintentional, fixed. I've changed everything to check 0.preopt since 1.promote wasn't appearing in some cases, not sure why

lld/test/ELF/lto/update_public_type_test.ll

1 ↗

(On Diff #444706)

done

llvm/lib/LTO/LTOBackend.cpp

576

exposed the existing hasWholeProgramVisibility

llvm/lib/Transforms/IPO/LowerTypeTests.cpp

1849 ↗

(On Diff #444706)

added a check

[tejohnson]

In D128955#3666830, @aeubanks wrote:

the assert that there are no public.type.tests in LTT fails on CodeGenCXX/thinlto-distributed-type-metadata.cpp. for some reason clang doesn't go through the LTO API at [1], it just ends up calling the normal optimization pipeline. any ideas why?

[1] https://github.com/llvm/llvm-project/blob/0c1b32717bcffcf8edf95294e98933bd4c1e76ed/clang/lib/CodeGen/BackendUtil.cpp#L1173

Ah ok, that handling is for some cases where we may decide not to do ThinLTO, but rather compile the IR down to native code normally without a thin link (we use that feature for distributed build system reasons). Looks like we need a fallback mechanism to convert these in that case. Maybe right at the start of DevirtModule::run? We could do in LTT where you are currently asserting too, but to me it seems more natural to get rid of these at the start of WPD, because with LTO they are gone by then.

llvm/lib/LTO/LTOBackend.cpp

576

See note about needing this for legacy LTO API (ThinLTOCodeGenerator.cpp).

llvm/test/LTO/X86/public-type-test.ll

1 ↗

(On Diff #446269)

Needs some CHECKs?

[aeubanks]

update

[aeubanks]

In D128955#3666995, @tejohnson wrote:

In D128955#3666830, @aeubanks wrote:

the assert that there are no public.type.tests in LTT fails on CodeGenCXX/thinlto-distributed-type-metadata.cpp. for some reason clang doesn't go through the LTO API at [1], it just ends up calling the normal optimization pipeline. any ideas why?

[1] https://github.com/llvm/llvm-project/blob/0c1b32717bcffcf8edf95294e98933bd4c1e76ed/clang/lib/CodeGen/BackendUtil.cpp#L1173

Ah ok, that handling is for some cases where we may decide not to do ThinLTO, but rather compile the IR down to native code normally without a thin link (we use that feature for distributed build system reasons). Looks like we need a fallback mechanism to convert these in that case. Maybe right at the start of DevirtModule::run? We could do in LTT where you are currently asserting too, but to me it seems more natural to get rid of these at the start of WPD, because with LTO they are gone by then.

I went back to my previous implementation of doing it LTT since we're dropping normal type tests there anyway, and added a comment about this

llvm/lib/LTO/LTOBackend.cpp

576

Added both updatePublicTypeTestCalls and setWithWholeProgramVisibility to ThinLTOCodeGenerator.cpp, which public-type-test.ll tests

[tejohnson]

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

456 ↗

(On Diff #446566)

Add a comment to see note at call to updateVCallVisibilityInIndex for why parmeter is set to false.

llvm/test/LTO/X86/public-type-test.ll

3 ↗

(On Diff #446566)

The new version switched this from a regular LTO to ThinLTO test. I think we need both. Can you move the new version into llvm/test/ThinLTO/X86 and make this version back to testing the regular LTO old API handling, but with checks?

[aeubanks]

update

[aeubanks]

random question, if the old API is "legacy", are there any plans to remove it?

llvm/test/LTO/X86/public-type-test.ll

3 ↗

(On Diff #446566)

done, had to add a new flag to dump IR right before optimizations

[tejohnson]

In D128955#3674787, @aeubanks wrote:

random question, if the old API is "legacy", are there any plans to remove it?

@fhahn started to work on this at some point (see https://bugs.llvm.org/show_bug.cgi?id=41541), but I'm not sure of the status. It is used by ld64 and I believe the sony toolchain too.

lld/test/ELF/lto/update_public_type_test.ll

5 ↗

(On Diff #447170)

Just realized there isn't any testing of the regular LTO handling with the new LTO API. Can you extend this test to check regular LTO as well?

llvm/test/LTO/X86/public-type-test.ll

3 ↗

(On Diff #446566)

I see, looks like llvm-lto has a thinlto-save-temps that is wired up to dump the same outputs as the new LTO API's save temps, but regular LTO didn't have the same support. I'd go ahead and use the same naming convention (0.preopt.bc) for compatibility with the new LTO API, in case that save temps handling is ever generalized to support both thin and regular LTO.

[aeubanks]

update

[aeubanks]

In D128955#3676198, @tejohnson wrote:

In D128955#3674787, @aeubanks wrote:

random question, if the old API is "legacy", are there any plans to remove it?

@fhahn started to work on this at some point (see https://bugs.llvm.org/show_bug.cgi?id=41541), but I'm not sure of the status. It is used by ld64 and I believe the sony toolchain too.

if I'm seeing correctly, looks like there aren't any in-tree users of ThinLTOCodeGenerator/LTOCodeGenerator. are those out of tree?

lld/test/ELF/lto/update_public_type_test.ll

5 ↗

(On Diff #447170)

extended llvm/test/LTO/X86/public-type-test.ll

[tejohnson]

lgtm

In D128955#3676478, @aeubanks wrote:

In D128955#3676198, @tejohnson wrote:

In D128955#3674787, @aeubanks wrote:

random question, if the old API is "legacy", are there any plans to remove it?

@fhahn started to work on this at some point (see https://bugs.llvm.org/show_bug.cgi?id=41541), but I'm not sure of the status. It is used by ld64 and I believe the sony toolchain too.

if I'm seeing correctly, looks like there aren't any in-tree users of ThinLTOCodeGenerator/LTOCodeGenerator. are those out of tree?

Yes, all linker users are out of tree. Only llvm-lto is in tree for testing.

[aeubanks]

thanks for the review!

[fhahn]

In D128955#3679418, @tejohnson wrote:

lgtm

In D128955#3676478, @aeubanks wrote:

In D128955#3676198, @tejohnson wrote:

In D128955#3674787, @aeubanks wrote:

random question, if the old API is "legacy", are there any plans to remove it?

@fhahn started to work on this at some point (see https://bugs.llvm.org/show_bug.cgi?id=41541), but I'm not sure of the status. It is used by ld64 and I believe the sony toolchain too.

if I'm seeing correctly, looks like there aren't any in-tree users of ThinLTOCodeGenerator/LTOCodeGenerator. are those out of tree?

Yes, all linker users are out of tree. Only llvm-lto is in tree for testing.

AFAIK ld64 uses ThinLTOCodeGenerator/LTOCodeGenerator through libTLO, which is in-tree (llvm/tools/lto/lto.cpp)

clang/lib/CodeGen/CGClass.cpp

void CodeGenFunction::EmitTypeMetadataCodeForVCall(const CXXRecordDecl *RD,
                                                    llvm::Value *VTable,
                                                    SourceLocation Loc) {
   if (SanOpts.has(SanitizerKind::CFIVCall))
     EmitVTablePtrCheckForCall(RD, VTable, CodeGenFunction::CFITCK_VCall, Loc);
   else if (CGM.getCodeGenOpts().WholeProgramVTables &&
            // Don't insert type test assumes if we are forcing public
            // visibility.
            !CGM.AlwaysHasLTOVisibilityPublic(RD)) {
-    llvm::Metadata *MD =
-        CGM.CreateMetadataIdentifierForType(QualType(RD->getTypeForDecl(), 0));
+    QualType Ty = QualType(RD->getTypeForDecl(), 0);
+    llvm::Metadata *MD = CGM.CreateMetadataIdentifierForType(Ty);
     llvm::Value *TypeId =
         llvm::MetadataAsValue::get(CGM.getLLVMContext(), MD);
 
     llvm::Value *CastedVTable = Builder.CreateBitCast(VTable, Int8PtrTy);
+    // isExternallyVisible(Ty->getLinkage())?
+    llvm::Intrinsic::ID IID =
+        Ty->getVisibility() == Visibility::HiddenVisibility
+            ? llvm::Intrinsic::type_test
+            : llvm::Intrinsic::public_type_test;
     llvm::Value *TypeTest =
-        Builder.CreateCall(CGM.getIntrinsic(llvm::Intrinsic::type_test),
-                           {CastedVTable, TypeId});
+        Builder.CreateCall(CGM.getIntrinsic(IID), {CastedVTable, TypeId});
     Builder.CreateCall(CGM.getIntrinsic(llvm::Intrinsic::assume), TypeTest);
   }
 }
 
 void CodeGenFunction::EmitVTablePtrCheckForCall(const CXXRecordDecl *RD,
                                                 llvm::Value *VTable,
                                                 CFITypeCheckKind TCK,
                                                 SourceLocation Loc) {

llvm/include/llvm/IR/Intrinsics.td

 def int_type_test : DefaultAttrsIntrinsic<[llvm_i1_ty], [llvm_ptr_ty, llvm_metadata_ty],
                               [IntrNoMem, IntrWillReturn, IntrSpeculatable]>;
 
 // Safely loads a function pointer from a virtual table pointer using type metadata.
 def int_type_checked_load : DefaultAttrsIntrinsic<[llvm_ptr_ty, llvm_i1_ty],
                                       [llvm_ptr_ty, llvm_i32_ty, llvm_metadata_ty],
                                       [IntrNoMem, IntrWillReturn]>;
 
+// Test whether a pointer is associated with a type metadata identifier.
+def int_public_type_test : DefaultAttrsIntrinsic<[llvm_i1_ty], [llvm_ptr_ty, llvm_metadata_ty],
+                              [IntrNoMem, IntrWillReturn, IntrSpeculatable]>;
+
 // Create a branch funnel that implements an indirect call to a limited set of
 // callees. This needs to be a musttail call.
 def int_icall_branch_funnel : DefaultAttrsIntrinsic<[], [llvm_vararg_ty], []>;
 
 def int_load_relative: DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_ptr_ty, llvm_anyint_ty],
                                  [IntrReadMem, IntrArgMemOnly]>;
 
 def int_asan_check_memaccess :

llvm/include/llvm/Transforms/IPO/WholeProgramDevirt.h

struct WholeProgramDevirtPass : public PassInfoMixin<WholeProgramDevirtPass> {
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &);
 };
 
 struct VTableSlotSummary {
   StringRef TypeID;
   uint64_t ByteOffset;
 };
 
+void updatePublicTypeTestCalls(Module &M,
+                               bool WholeProgramVisibilityEnabledInLTO);
 void updateVCallVisibilityInModule(
     Module &M, bool WholeProgramVisibilityEnabledInLTO,
     const DenseSet<GlobalValue::GUID> &DynamicExportSymbols);
 void updateVCallVisibilityInIndex(
     ModuleSummaryIndex &Index, bool WholeProgramVisibilityEnabledInLTO,
     const DenseSet<GlobalValue::GUID> &DynamicExportSymbols);
 
 /// Perform index-based whole program devirtualization on the \p Summary

llvm/lib/Analysis/ModuleSummaryAnalysis.cpp

 static void addIntrinsicToSummary(
     const CallInst *CI, SetVector<GlobalValue::GUID> &TypeTests,
     SetVector<FunctionSummary::VFuncId> &TypeTestAssumeVCalls,
     SetVector<FunctionSummary::VFuncId> &TypeCheckedLoadVCalls,
     SetVector<FunctionSummary::ConstVCall> &TypeTestAssumeConstVCalls,
     SetVector<FunctionSummary::ConstVCall> &TypeCheckedLoadConstVCalls,
     DominatorTree &DT) {
   switch (CI->getCalledFunction()->getIntrinsicID()) {
-  case Intrinsic::type_test: {
+  case Intrinsic::type_test:
+  case Intrinsic::public_type_test: {
     auto *TypeMDVal = cast<MetadataAsValue>(CI->getArgOperand(1));
     auto *TypeId = dyn_cast<MDString>(TypeMDVal->getMetadata());
     if (!TypeId)
       break;
     GlobalValue::GUID Guid = GlobalValue::getGUID(TypeId->getString());
 
     // Produce a summary from type.test intrinsics. We only summarize type.test
     // intrinsics that are used other than by an llvm.assume intrinsic.
     // Intrinsics that are assumed are relevant only to the devirtualization

llvm/lib/Analysis/TypeMetadataUtils.cpp

for (const Use &U : VPtr->uses()) {
     }
   }
 }
 
 void llvm::findDevirtualizableCallsForTypeTest(
     SmallVectorImpl<DevirtCallSite> &DevirtCalls,
     SmallVectorImpl<CallInst *> &Assumes, const CallInst *CI,
     DominatorTree &DT) {
-  assert(CI->getCalledFunction()->getIntrinsicID() == Intrinsic::type_test);
+  assert(CI->getCalledFunction()->getIntrinsicID() == Intrinsic::type_test ||
+         CI->getCalledFunction()->getIntrinsicID() ==
+             Intrinsic::public_type_test);
 
   const Module *M = CI->getParent()->getParent()->getParent();
 
   // Find llvm.assume intrinsics for this llvm.type.test call.
   for (const Use &CIU : CI->uses())
     if (auto *Assume = dyn_cast<AssumeInst>(CIU.getUser()))
       Assumes.push_back(Assume);
 

llvm/lib/LTO/LTOBackend.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
+#include "llvm/Transforms/IPO/WholeProgramDevirt.h"
 #include "llvm/Transforms/Scalar/LoopPassManager.h"
 #include "llvm/Transforms/Utils/FunctionImportUtils.h"
 #include "llvm/Transforms/Utils/SplitModule.h"
 
 using namespace llvm;
 using namespace lto;
 
 #define DEBUG_TYPE "lto-backend"
if (EmbedBitcode == LTOBitcodeEmbedding::EmbedPostMergePreOptimized) {
     if (CmdArgs.empty())
       LLVM_DEBUG(
           dbgs() << "Post-(Thin)LTO merge bitcode embedding was requested, but "
                     "command line arguments are not available");
     llvm::embedBitcodeInModule(Mod, llvm::MemoryBufferRef(),
                                /*EmbedBitcode*/ true, /*EmbedCmdline*/ true,
                                /*Cmdline*/ CmdArgs);
   }
+  updatePublicTypeTestCalls(Mod, Conf.HasWholeProgramVisibility);
   // FIXME: Plumb the combined index into the new pass manager.
   runNewPMPasses(Conf, Mod, TM, Conf.OptLevel, IsThinLTO, ExportSummary,
                  ImportSummary);
   return !Conf.PostOptModuleHook || Conf.PostOptModuleHook(Task, Mod);
 }
 
 static void codegen(const Config &Conf, TargetMachine *TM,
                     AddStreamFn AddStream, unsigned Task, Module &Mod,
if (!DiagFileOrErr)
     return DiagFileOrErr.takeError();
   auto DiagnosticOutputFile = std::move(*DiagFileOrErr);
 
   // Set the partial sample profile ratio in the profile summary module flag of
   // the module, if applicable.
   Mod.setPartialSampleProfileRatio(CombinedIndex);
 
   if (Conf.CodeGenOnly) {
     codegen(Conf, TM.get(), AddStream, Task, Mod, CombinedIndex);
     return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
   }
 
   if (Conf.PreOptModuleHook && !Conf.PreOptModuleHook(Task, Mod))
     return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
 
   auto OptimizeAndCodegen =
       [&](Module &Mod, TargetMachine *TM,
Error lto::thinBackend(const Config &Conf, unsigned Task, AddStreamFn AddStream,
   // When linking an ELF shared object, dso_local should be dropped. We
   // conservatively do this for -fpic.
   bool ClearDSOLocalOnDeclarations =
       TM->getTargetTriple().isOSBinFormatELF() &&
       TM->getRelocationModel() != Reloc::Static &&
       Mod.getPIELevel() == PIELevel::Default;
   renameModuleForThinLTO(Mod, CombinedIndex, ClearDSOLocalOnDeclarations);
 
   dropDeadSymbols(Mod, DefinedGlobals, CombinedIndex);
 
   thinLTOFinalizeInModule(Mod, DefinedGlobals, /*PropagateAttrs=*/true);
 
   if (Conf.PostPromoteModuleHook && !Conf.PostPromoteModuleHook(Task, Mod))
     return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
 
   if (!DefinedGlobals.empty())
     thinLTOInternalizeModule(Mod, DefinedGlobals);

llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp

 /// If whole program visibility asserted, then upgrade all public vcall
 /// visibility metadata on vtable definitions to linkage unit visibility in
 /// Module IR (for regular or hybrid LTO).
 void updateVCallVisibilityInModule(
     Module &M, bool WholeProgramVisibilityEnabledInLTO,
     const DenseSet<GlobalValue::GUID> &DynamicExportSymbols) {
   if (!hasWholeProgramVisibility(WholeProgramVisibilityEnabledInLTO))
     return;
-  for (GlobalVariable &GV : M.globals())
+  for (GlobalVariable &GV : M.globals()) {
     // Add linkage unit visibility to any variable with type metadata, which are
     // the vtable definitions. We won't have an existing vcall_visibility
     // metadata on vtable definitions with public visibility.
     if (GV.hasMetadata(LLVMContext::MD_type) &&
         GV.getVCallVisibility() == GlobalObject::VCallVisibilityPublic &&
         // Don't upgrade the visibility for symbols exported to the dynamic
         // linker, as we have no information on their eventual use.
         !DynamicExportSymbols.count(GV.getGUID()))
       GV.setVCallVisibilityMetadata(GlobalObject::VCallVisibilityLinkageUnit);
   }
+}
+
+void updatePublicTypeTestCalls(Module &M,
+                               bool WholeProgramVisibilityEnabledInLTO) {
+  Function *PublicTypeTestFunc =
+      M.getFunction(Intrinsic::getName(Intrinsic::public_type_test));
+  if (!PublicTypeTestFunc)
+    return;
+  if (hasWholeProgramVisibility(WholeProgramVisibilityEnabledInLTO)) {
+    Function *TypeTestFunc =
+        Intrinsic::getDeclaration(&M, Intrinsic::type_test);
+    for (Use &U : make_early_inc_range(PublicTypeTestFunc->uses())) {
+      auto *CI = cast<CallInst>(U.getUser());
+      auto *NewCI = CallInst::Create(
+          TypeTestFunc, {CI->getArgOperand(0), CI->getArgOperand(1)}, None, "",
+          CI);
+      CI->replaceAllUsesWith(NewCI);
+      CI->eraseFromParent();
+    }
+  } else {
+    auto *True = ConstantInt::getTrue(M.getContext());
+    for (Use &U : make_early_inc_range(PublicTypeTestFunc->uses())) {
+      auto *CI = cast<CallInst>(U.getUser());
+      CI->replaceAllUsesWith(True);
+      CI->eraseFromParent();
+    }
+  }
+}
 
 /// If whole program visibility asserted, then upgrade all public vcall
 /// visibility metadata on vtable definition summaries to linkage unit
 /// visibility in Module summary index (for ThinLTO).
 void updateVCallVisibilityInIndex(
     ModuleSummaryIndex &Index, bool WholeProgramVisibilityEnabledInLTO,
     const DenseSet<GlobalValue::GUID> &DynamicExportSymbols) {
   if (!hasWholeProgramVisibility(WholeProgramVisibilityEnabledInLTO))