Page MenuHomePhabricator
Feed Advanced Search

Jun 11 2020

chrisjackson abandoned D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.

Abandoned in favour of the alternative D79863.

Jun 11 2020, 5:26 AM · Restricted Project, debug-info
chrisjackson committed rG4707bc217780: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues (authored by chrisjackson).
[DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues
Jun 11 2020, 3:37 AM
chrisjackson closed D79863: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues.
Jun 11 2020, 3:37 AM · debug-info, Restricted Project

Jun 10 2020

chrisjackson closed D78369: [DebugInfo] Reduce SalvageDebugInfo() functions.

This revision was closed with commit c6c65164. I'm uncertain why the commit's 'differential revision' tag failed to trigger the autoclose.

Jun 10 2020, 7:04 AM · Restricted Project, debug-info

Jun 8 2020

chrisjackson committed rGc6c65164af9b: [DebugInfo] Reduce SalvageDebugInfo() functions (authored by chrisjackson).
[DebugInfo] Reduce SalvageDebugInfo() functions
Jun 8 2020, 11:36 AM

May 28 2020

chrisjackson added a project to D79863: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues: debug-info.
May 28 2020, 4:50 AM · debug-info, Restricted Project
chrisjackson added a watcher for debug-info: chrisjackson.
May 28 2020, 4:18 AM
chrisjackson added a comment to D79863: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues.

Are there any more suggestions please?

May 28 2020, 3:44 AM · debug-info, Restricted Project

May 26 2020

chrisjackson committed rG9eacda51fa23: [debuginfo] Fix broken tests from MachineLICM salvaging fix (authored by chrisjackson).
[debuginfo] Fix broken tests from MachineLICM salvaging fix
May 26 2020, 3:18 PM
chrisjackson committed rGbd7ff5d94f0f: [DebugInfo] Correct debuginfo for post-ra hoist and sink in Machine LICM (authored by chrisjackson).
[DebugInfo] Correct debuginfo for post-ra hoist and sink in Machine LICM
May 26 2020, 1:39 PM
chrisjackson closed D79868: [DebugInfo] Correct debuginfo for post-ra hoist and sink in Machine LICM.
May 26 2020, 1:38 PM · Restricted Project

May 22 2020

chrisjackson updated the diff for D78369: [DebugInfo] Reduce SalvageDebugInfo() functions.

Removed unused return value from SalvageDebugInfo().

May 22 2020, 11:15 AM · Restricted Project, debug-info
chrisjackson updated the diff for D79863: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues.
  • Corrected comment
  • Removed unnecessary reference in iteration
  • Added missing pointer dereference
May 22 2020, 9:38 AM · debug-info, Restricted Project

May 19 2020

Herald added a project to D78369: [DebugInfo] Reduce SalvageDebugInfo() functions: Restricted Project.
May 19 2020, 7:00 AM · Restricted Project, debug-info
chrisjackson added a comment to D79863: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues.

Polite ping :)

May 19 2020, 1:03 AM · debug-info, Restricted Project

May 14 2020

chrisjackson updated the diff for D79868: [DebugInfo] Correct debuginfo for post-ra hoist and sink in Machine LICM.
  • Removed some excess debug information from post-ra test
  • Made sink Filecheck test more strict
May 14 2020, 11:23 AM · Restricted Project

May 13 2020

chrisjackson created D79868: [DebugInfo] Correct debuginfo for post-ra hoist and sink in Machine LICM.
May 13 2020, 8:37 AM · Restricted Project
chrisjackson added a comment to D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.

Assuming D78369 lands as it is right now; IMO it might make sense to push the OrUndef behaviour of salvageDebugInfo into salvageDebugInfoForDbgValues, and have salvageDebugInfo simply call salvageDebugInfoForDbgValues(I, findDbgUsers(I)).

This is pretty similar to what happens pre-D78369, except that salvageDebugInfo and salvageDebugInfoForDbgValues both MarkUndef if they fail to salvage. I'm fairly confident you could then simplify the instcombine sink salvaging with this setup.

What do you both think?

May 13 2020, 7:32 AM · Restricted Project, debug-info
chrisjackson created D79863: [DebugInfo] Refactor SalvageDebugInfo and SalvageDebugInfoForDbgValues.
May 13 2020, 7:32 AM · debug-info, Restricted Project

May 7 2020

chrisjackson abandoned D47540: [lld] Enable Visual Studio compatible output.

This work was completed in D58484.

May 7 2020, 4:29 AM

Apr 27 2020

chrisjackson updated the diff for D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.
  • Removed the call to reverse() in InstCombine.cpp and updated the debuginfo_add.ll test accordingly.
  • Moved some comments in InstCombine to a more appropriate spot.
  • Simplified an iteration in InstCombine as suggested.
Apr 27 2020, 8:34 AM · Restricted Project, debug-info
Herald added a project to D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine: Restricted Project.
Apr 27 2020, 1:33 AM · Restricted Project, debug-info

Apr 22 2020

chrisjackson updated the diff for D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.

Applied clang-format-diff.

Apr 22 2020, 7:33 AM · Restricted Project, debug-info

Apr 20 2020

chrisjackson added a comment to D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.
In D78398#1989878, @vsk wrote:

I personally find the existing code easier to read/understand. Is there a strong motivation for hiding salvageDebugInfoForDbgValues?

Apr 20 2020, 8:05 AM · Restricted Project, debug-info

Apr 17 2020

chrisjackson updated the summary of D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.
Apr 17 2020, 12:58 PM · Restricted Project, debug-info
chrisjackson created D78398: [DebugInfo] Factor out SalvageDebugInfoForDbgValues() from InstCombine.
Apr 17 2020, 12:58 PM · Restricted Project, debug-info
chrisjackson created D78369: [DebugInfo] Reduce SalvageDebugInfo() functions.
Apr 17 2020, 7:32 AM · Restricted Project, debug-info

Apr 6 2020

chrisjackson closed D76930: [DebugInfo] Ensure dead store elimination can mark an operand value as undefined.

Closed by commit 135709aa9013565b9d02000bb0ca744f7585e6a5.

Apr 6 2020, 5:54 AM

Mar 30 2020

chrisjackson committed rGf6b2c003f360: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not… (authored by chrisjackson).
[DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not…
Mar 30 2020, 8:04 AM
chrisjackson closed D76854: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not result in an incorrect debuginfo variable value.
Mar 30 2020, 8:04 AM · Restricted Project, debug-info
chrisjackson committed rG135709aa9013: [DebugInfo] Ensure dead store elimination can mark an operand value as undefined (authored by chrisjackson).
[DebugInfo] Ensure dead store elimination can mark an operand value as undefined
Mar 30 2020, 7:01 AM

Mar 27 2020

chrisjackson created D76930: [DebugInfo] Ensure dead store elimination can mark an operand value as undefined.
Mar 27 2020, 8:12 AM

Mar 26 2020

chrisjackson updated the diff for D76854: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not result in an incorrect debuginfo variable value.

Removed unnecessary CHECK line. It was erroneous anyway!

Mar 26 2020, 10:18 AM · Restricted Project, debug-info
chrisjackson updated the diff for D76854: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not result in an incorrect debuginfo variable value.

Simplified the CHECK expression to match only the component of the dbg.value call that is significant for this test.

Mar 26 2020, 10:18 AM · Restricted Project, debug-info
chrisjackson added reviewers for D76854: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not result in an incorrect debuginfo variable value: vsk, aprantl, djtodoro.
Mar 26 2020, 9:45 AM · Restricted Project, debug-info
chrisjackson added projects to D76854: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not result in an incorrect debuginfo variable value: debug-info, Restricted Project.
Mar 26 2020, 9:45 AM · Restricted Project, debug-info
chrisjackson created D76854: [DebugInfo] Ensure that a demanded bits optimisation in InstCombine does not result in an incorrect debuginfo variable value.
Mar 26 2020, 8:38 AM · Restricted Project, debug-info

Feb 25 2020

chrisjackson abandoned D58300: [lld] Enable Visual Studio compatible output.

The diagnostics were implemented in D58484.

Feb 25 2020, 3:21 AM · lld

Feb 19 2020

chrisjackson abandoned D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.
Feb 19 2020, 1:46 AM · Restricted Project
chrisjackson abandoned D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.
Feb 19 2020, 1:46 AM · Restricted Project

Feb 18 2020

chrisjackson added a comment to D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.

This revision will soon be abandoned along with D66426, unless there are any further comments.

Feb 18 2020, 4:20 AM · Restricted Project
chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

It would seem that there just isn't sufficient interest in this, so I'm going to mark this revision as abandoned if there are no further comments.

Feb 18 2020, 4:05 AM · Restricted Project

Jan 24 2020

chrisjackson added a comment to D62148: [llvm-nm] Omit the symbol table entry at index 0 when --debug-syms is enabled for ELF files.

@chrisjackson, I assume this got committed, and just didn't get closed? Can you confirm?

Jan 24 2020, 10:51 AM

Jan 17 2020

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

ping

Jan 17 2020, 5:41 AM · Restricted Project

Jan 14 2020

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

Apologies for the late reply.

http://lists.llvm.org/pipermail/llvm-dev/2019-November/137108.html does not evoke many responses. I take the lack of responses on the RFC and questions from @JonChesterfield and @ruiu as people are still doubting whether this feature will be generic enough to benefit the community, rather than a feature used in a very specific scenario, which can be easily replaced by another tool.

Jan 14 2020, 6:26 AM · Restricted Project

Jan 10 2020

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

ping

Jan 10 2020, 6:28 AM · Restricted Project

Jan 8 2020

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

Hello @ruiu and @MaskRay , can I help with any more technical queries concerning this proposal? Also with respect to the associated D70316 that enables printing of watermark note sections and computing of watermarks in readobj?

Jan 8 2020, 10:59 AM · Restricted Project

Jan 6 2020

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

Ping

Jan 6 2020, 4:07 AM · Restricted Project

Dec 11 2019

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

As I understand it, the scenario is:

  1. Do link; 2) Run the watermark tool to append .note.llvm.watermark; 3) Release SDK; 4) Downstream vendors modify .data and ship to end users; 5) End users verify that .note.llvm.watermark does not match computed watermark of loadable contents.

The build process before 3) are all controlled. The process should ensure there is no modification to .data between 1) and 2). How do you guarantee a linker side feature can prevent modification? How can you prevent the following:

  1. Do link and generate .note.llvm.watermark in one step 1.5) Modify .data 2) Run the watermark tool to update .note.llvm.watermark

This is a motivation to not have an external tool on the watermarking. That being said, as @chrisjackson has said on more than one occasion, this isn't intended to be a security feature so we are not attempting to detect a malicious attacker. It could be possible for downstream LLD producers to add a local salt to the watermark to make it more secure, should they so choose, I suppose.

This is actually going to be an interesting problem. Do your users make post-link modifications to executables by intention or by accident? If it's intentional, you are raising a bar of an arms race, and they'll catch up by adding --update-watermark option or something to their tool, so that they'll update a watermark when a binary is modified, which nullifies the point of this change.

Dec 11 2019, 6:27 AM · Restricted Project

Dec 9 2019

chrisjackson updated the diff for D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

Modified extractSegments() in watermark.h.

Dec 9 2019, 5:18 AM · Restricted Project

Dec 5 2019

chrisjackson updated the summary of D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.
Dec 5 2019, 11:31 AM · Restricted Project
chrisjackson updated the diff for D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.

Removed library test accidentally added in last update.

Dec 5 2019, 11:30 AM · Restricted Project
chrisjackson updated the diff for D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.

Added --compute-watermark, additional tests, use of library from update to D66426.

Dec 5 2019, 11:30 AM · Restricted Project
chrisjackson updated the diff for D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

Watermark functionality now placed in separate source and header in the object library. This is used by Writer and SyntheticSections.

Dec 5 2019, 11:21 AM · Restricted Project

Dec 4 2019

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

A post-link modification could recalculate and update the hash, but this would only occur in a deliberate attempt to subvert the watermark mechanism

I think it follows that this patch only detects accidental modifications to the binary that occur after linking. That seems to put it in the realm of network transmission errors, disk bit rot, optical media errors and so forth.

In which case, why only guard a subset of the binary, instead of computing a sha256 of all the compiled artifacts and checking that at install/network copy time? Then there is again no linker patch required.

Unless this is intended to catch people who deliberately change the binary, but lack the skills to then update the hash, which is surely vanishingly few people. Fewer when provided with convenient tools to recalculate the hash.

Dec 4 2019, 2:17 AM · Restricted Project

Nov 27 2019

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

I sympathize with the requirement to tell whether a binary has been edited after the link step. E.g. one could then raise an error from the loader.

Writing 8 bytes to a known location in the binary can't achieve that. Whatever post link modification is performed will recalculate and update the hash in the binary. If I understand correctly, the plan is to enhance a llvm binary utility to conveniently perform this updating, or at least to provide the 8 bytes to be written into the known location by any other tool.

So I see the cost - lld and other tools get more complicated - and I see the requirement - but I can't see how the proposed change meets the requirement.

Nov 27 2019, 8:27 AM · Restricted Project

Nov 20 2019

chrisjackson added inline comments to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.
Nov 20 2019, 8:43 AM · Restricted Project
chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

This patch introduces a new ELF extension with a new tag: NT_LLVM_WATERMARK. Last times such ELF extensions were made accompanying RFCs were posted:

https://lists.llvm.org/pipermail/llvm-dev/2019-February/130583.html
https://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html

I still have some concerns about the extension, especially its generality on the widely used ELF platforms.

We would like to have two PT_NOTEs, one for use by the OS and another for use by tooling. This is achieved by linker scripts. The second PT_NOTE is outside of any PT_LOAD and this is where the watermark would be housed. As described above, a required property of the watermark is that it can be recalculated by an external tool to infer whether or not the loadable parts of the ELF have been modified post-link. By having the watermark outside of any PT_LOAD, it is simpler for the external tool to recalculate. For a similar reason, it would actually be better in our case to have the watermark calculated after the build ID value has been "filled-in", as the build ID is inside a PT_LOAD.

Sorry, I am not following. Do you have a concrete readelf -Sl dump to help my understanding?
By linking a trivial executable with ld.lld a.o --watermark -o a, what I see is:

 PHDR           0x000040 0x0000000000200040 0x0000000000200040 0x000118 0x000118 R   0x8
 LOAD           0x000000 0x0000000000200000 0x0000000000200000 0x000174 0x000174 R   0x1000
 GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0
 NOTE           0x000158 0x0000000000200158 0x0000000000200158 0x00001c 0x00001c R   0x4

Section to Segment mapping:
 Segment Sections...
  00
  01     .note.llvm.watermark
  02
  03     .note.llvm.watermark

.note.llvm.watermark is still included in a PT_LOAD segment. I think you probably meant:

  • .note.gnu.build-id is included in a PT_LOAD and a PT_NOTE
  • .note.llvm.watermark is included in another PT_NOTE

There is also a question why .note.llvm.watermark should be flagged SHF_ALLOC if it is not supposed to be inspected at runtime.

Nov 20 2019, 6:23 AM · Restricted Project

Nov 19 2019

chrisjackson updated the diff for D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

@ruiu observed that the watermark computation was reliant on the segment ordering in the ELF file. This ordering can change without affecting the loadable image. Therefore, we now apply an ordering based on the segment's virtual address when calculating the watermark.

Nov 19 2019, 6:02 AM · Restricted Project

Nov 15 2019

chrisjackson updated the summary of D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.
Nov 15 2019, 9:29 AM · Restricted Project
chrisjackson created D70316: [llvm-readobj] Allow printing of the watermark note section proposed in D66426.
Nov 15 2019, 8:27 AM · Restricted Project

Sep 13 2019

chrisjackson added a comment to D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

It seems the main reason why you guys wanted to avoid an external tool is that it is too easy to forget to run the tool after link. But that can be fixed easily by writing a shell script as ld.lld which invokes the real ld.lld and a watermarking tool. Or, you could make a change to lld so that, after creating an executable file and before existing, lld invokes a watermarking tool on a file that the linker just created (in this configuration, the watermarking tool can either run in the same process or as a child process). Have you considered that approach? I think it is fine to add this feature directly to lld if it is convenient, but I'd like to explore other possibilities before we make a decision.

Besides that, there are a few technical concerns in this patch as below:

  • lld guarantees that the same build id will be computed only when the resulting output file (except the build id part itself) and the linker version are the same. We didn't guarantee that different versions of lld compute the build id in the same way. Actually, we have tweaked hash functions and the strategy for tree hash several times. This contract seems too weak for your use case -- for your use case, we need to guarantee that the way how we compute a hash value doesn't change over time. So, we need to make sure that the current way of hash computation is something that we want to maintain like forever.

    If watermarking doesn't have to be fast (e.g. users only have to do this for release binaries), consider using a simple non-tree hash function.
  • Maybe you should add a version field or something to the note section so that we can change a hash function or something in the future.
  • The watermark feature is to make sure that the program image loaded to memory hasn't changed since its file is created. In that sense, the hash function seems a bit too fragile. If you move a segment within an ELF file, you'd have to change the file offset field of a program header, but the memory image won't change by doing that, so in the sense of watermarking, I don't think it should be considered a change.
Sep 13 2019, 6:53 AM · Restricted Project

Sep 10 2019

chrisjackson added a comment to D62955: [llvm-nm] Additional lit tests for command line options.

This was closed with commit 7b39513302, svn r363248.

If your description included Differential Revision: https://reviews.llvm.org/D62955 (see other commits' descriptions), the revision would have been closed automatically when you committed.

Sep 10 2019, 6:32 AM

Sep 6 2019

chrisjackson updated the diff for D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.

This update applies suggested source corrections, enables readobj to output the note and adds logic to writeWatermark() to exclude the ELF Header and Program Header Table (PHT) from the watermark. In this diff the PHT can only be excluded if it is the first or last segment. If it is neither, an error is emitted.

Sep 6 2019, 3:42 AM · Restricted Project

Sep 3 2019

chrisjackson closed D63340: [llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined binding does not match class assigned by GNU nm.

This was closed with commit 41e20d210, svn r364559.

Sep 3 2019, 2:55 AM
chrisjackson closed D62955: [llvm-nm] Additional lit tests for command line options.

This was closed with commit 7b39513302, svn r363248.

Sep 3 2019, 2:47 AM

Aug 30 2019

chrisjackson closed D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

This was closed with revision r370458, commit fa1fe93789.

Aug 30 2019, 3:28 AM · Restricted Project
chrisjackson committed rGfa1fe937893c: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add… (authored by chrisjackson).
[llvm-objcopy] Allow the visibility of symbols created by --binary and --add…
Aug 30 2019, 3:17 AM

Aug 29 2019

chrisjackson updated the diff for D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

Update to apply the corrections pointed out by @MaskRay .

Aug 29 2019, 6:43 AM · Restricted Project

Aug 28 2019

chrisjackson updated the diff for D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

The diff has been updated to apply the visibility to symbols created with --add-symbol where a visibility flag is not specified. The flag name has also been changed from --binary-symbol-visibility to --new-symbol-visibility

Aug 28 2019, 9:29 AM · Restricted Project
chrisjackson retitled D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility from [llvm-objcopy] Allow the visibility of the start, end and size symbols created by --binary to be specified with --news-symbol-visibility to [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.
Aug 28 2019, 9:26 AM · Restricted Project
chrisjackson retitled D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility from [llvm-objcopy] Allow the visibility of the start, end and size symbols created by --binary to be specified with --binary-symbol-visibility to [llvm-objcopy] Allow the visibility of the start, end and size symbols created by --binary to be specified with --news-symbol-visibility.
Aug 28 2019, 9:26 AM · Restricted Project

Aug 20 2019

chrisjackson updated the summary of D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.
Aug 20 2019, 2:37 AM · Restricted Project

Aug 19 2019

chrisjackson added reviewers for D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section: jhenderson, edd, andrewng.
Aug 19 2019, 10:00 AM · Restricted Project
chrisjackson created D66426: [lld] Enable a watermark of loadable sections to be generated and placed in a note section.
Aug 19 2019, 9:56 AM · Restricted Project

Aug 15 2019

chrisjackson added a comment to D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

Is your GNU objcopy patched to use STV_HIDDEN or STV_PROTECTED for _binary_a_{start,end,size}?

objcopy -I binary -O elf64-x86-64 a b => Symbols in b are STV_HIDDEN/STV_PROTECTED?

Aug 15 2019, 6:26 AM · Restricted Project
chrisjackson added a comment to D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

I agree that the generalized version is useful and I have been looking at a patch for this. However, in this specific case the symbol names are automatically generated based on the path of the binary, so it makes it difficult to specify these specific symbol names on the command line.

The symbol names are generated but that is unavoidable. You also have to specify the name in the source: extern const char _binary_a_start[];

With regards to the expected semantics, we anticipate standard ELF semantics when combining different visibilities. Our toolchain operates using a default visibility of STV_HIDDEN for definitions unless explicitly exported, in which case they are STV_PROTECTED. This means that declarations need not be explicitly marked with a visibility in order to be exported or not. The new switch allows the replication of the compiler's -fvisibility=hidden switch in llvm-objcopy. As an alternative to the name --binary-symbol-visibility, how about --symbol-visibility? This would affect the visibility of all the new symbols generated by llvm-objcopy, not just those symbols added for binary input.

There are no other options that add new symbols. --add-symbol has its own visibility flags, --symbol-visibility will not help that feature. --new-symbol-visibility might be a better name but people need to be taught it will be overridden by --add-symbol visibility flags. I am still not sure the little amount of saved labor (__attribute__((visibility("protected"))) marks) justifies a new option, given that Keil embedded development tools already require other markers like __declspec(dllexport) and __declspec(dllimport). And another question: how did you survive with GNU objcopy? It doesn't allow you to change the visibility of _binary_xxx_start...

In D65893#1620711, @MaskRay wrote:
Quote gABI:

Second, if any reference to or definition of a name is a symbol with a non-default visibility attribute, the visibility attribute must be propagated to the resolving symbol in the linked object. If different visibility attributes are specified for distinct references to or definitions of a symbol, the most constraining visibility attribute must be propagated to the resolving symbol in the linked object. The attributes, ordered from least to most constraining, are: STV_PROTECTED, STV_HIDDEN and STV_INTERNAL.

You can achieve the same with: attribute((visibility("hidden"))) extern char _binary_a_start[]; So I'm not sure why another option is needed (I think my another concern is that the option name --binary-symbol-visibility can make people puzzled if they add/delete/change symbols). See D55682.

Aug 15 2019, 3:00 AM · Restricted Project
chrisjackson committed rGe5cdfbc65cac: [llvm-objcopy] Allow 'protected' visibility to be set when using add-symbol (authored by chrisjackson).
[llvm-objcopy] Allow 'protected' visibility to be set when using add-symbol
Aug 15 2019, 2:46 AM

Aug 14 2019

chrisjackson added a comment to D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.
  • For our use cases it is more convenient to change the symbol's visibility with an option than to have users alter their source code.

I accepted D65891 because:

  • It deals with an existing option: --add-symbol. protected is a natural extension to the existing default/hidden visibility. Nobody will get puzzled by this addition.
  • According to Changes to symbol visibility between RVCT v3.1 for µVision and RVCT v4.0 for µVision you linked, there are some use cases. I guess it is used for preventing inadvertent preemption in static linking.

The first bullet is sufficient for me to accept it, but I hope you can elaborate more on the second bullet. How do you use STV_PROTECTED in object files and shared objects? What semantics do you expected when linking STV_DEFAULT/STV_PROTECTED undefined/defined together?

For our use cases it is more convenient to change the symbol's visibility with an option than to have users alter their source code.

--binary-symbol-visibility can cause confusion. As suggested by rupprecht, it should be generalized to --redefine-visibility= if we do want the option.

Aug 14 2019, 6:35 AM · Restricted Project

Aug 9 2019

chrisjackson added a comment to D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

Quote gABI:

Second, if any reference to or definition of a name is a symbol with a non-default visibility attribute, the visibility attribute must be propagated to the resolving symbol in the linked object. If different visibility attributes are specified for distinct references to or definitions of a symbol, the most constraining visibility attribute must be propagated to the resolving symbol in the linked object. The attributes, ordered from least to most constraining, are: STV_PROTECTED, STV_HIDDEN and STV_INTERNAL.

You can achieve the same with: __attribute__((visibility("hidden"))) extern char _binary_a_start[]; So I'm not sure why another option is needed (I think my another concern is that the option name --binary-symbol-visibility can make people puzzled if they add/delete/change symbols). See D55682.

Three symbols per embedding in an object results in bloated output.

I don't understand this point. How can the visibility change de-bloat the output?

Preventing this with many attribute statements seems like a clumsy approach.

I don't understand this, either. Your are going to reference _binary_a_start, (_binary_a_end or _binary_a_size) - two undefined symbols. Restrict the undefined symbol with a visibility attribute, then the linked exe/dso will change the symbol bindings accordingly. What's the problem?

With regards to the confusing option, we can just decide on a better name.

We can do that once we decide this option is useful.

Aug 9 2019, 9:22 AM · Restricted Project
chrisjackson added a comment to D65891: [llvm-objcopy] Allow 'protected' visibility to be set when using add-symbol.

It makes sense from the perspective of completeness but STV_PROTECTED is problematic and can hardly find a justified use case.

People use STV_PROTECTED when they want to avoid GOT/PLT costs in PIC code. However,

A protected STT_OBJECT prevents you from doing copy relocation. ld.bfd silently accepts this, which can cause ODR violations.
A protected STT_FUNC prevents you from making a canonical PLT (-fno-pic code in executable references an external function). If the linker fails to stop you, you'll get pointer equality problems. (It can be argued this case can be fixed by letting the compiler go through GOT when the address of a protected function is taken, but then the GOT indirection will likely break people's expectation. A symbol lookup change is also required on ld.so side.)

I think lld and gold reject both cases. So at the best, a protected symbol can be seen as a promise of the toolchain that the symbol will not be interposed.

Actually, in a DSO, a STV_PROTECTED use case can be easily replaced with the hidden+alias idiom:

__attribute__((visibility("hidden"))) void internal() {} // internal use
__attribute__((alias("internal"))) void external(); // external API

The reservation of STV_INTERNAL was requested by SGI when gABI was formulated. All other parties appear to treat STV_INTERNAL the same as STV_HIDDEN.

Aug 9 2019, 4:54 AM · Restricted Project
chrisjackson added a comment to D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility.

Quote gABI:

Second, if any reference to or definition of a name is a symbol with a non-default visibility attribute, the visibility attribute must be propagated to the resolving symbol in the linked object. If different visibility attributes are specified for distinct references to or definitions of a symbol, the most constraining visibility attribute must be propagated to the resolving symbol in the linked object. The attributes, ordered from least to most constraining, are: STV_PROTECTED, STV_HIDDEN and STV_INTERNAL.

You can achieve the same with: __attribute__((visibility("hidden"))) extern char _binary_a_start[]; So I'm not sure why another option is needed (I think my another concern is that the option name --binary-symbol-visibility can make people puzzled if they add/delete/change symbols). See D55682.

Aug 9 2019, 4:20 AM · Restricted Project

Aug 7 2019

Herald added a reviewer for D65893: [llvm-objcopy] Allow the visibility of symbols created by --binary and --add-symbol to be specified with --new-symbol-visibility: alexshap.
Aug 7 2019, 10:26 AM · Restricted Project
Herald added a reviewer for D65891: [llvm-objcopy] Allow 'protected' visibility to be set when using add-symbol: alexshap.
Aug 7 2019, 10:05 AM · Restricted Project

Jul 25 2019

chrisjackson added a comment to D65213: [ELF] With --vs-diagnostics, print a separate message for each location of a duplicate symbol..

I don't think we should change the original error message, as the original message seems to make more sense than the new one for users.

The original error message, when --vs-diagnostics is not used, is mostly the same. There is only a small improvement for a case when only one section, either for an old or new definition, is present.

On the other hand, one of the reasons for --vs-diagnostics is to provide a user with means to jump to the mentioned location. That is achieved by placing the location in front of the error message. Thus, as there are usually two source locations, there should be two separate error messages.

Have you considered changing the regexp to parse this error message?

Jul 25 2019, 7:44 AM · Restricted Project

Jul 18 2019

chrisjackson committed rG0b03429a9111: [lld] Fix vs-diagnostics-version-script test. NFC. (authored by chrisjackson).
[lld] Fix vs-diagnostics-version-script test. NFC.
Jul 18 2019, 2:19 AM

Jul 17 2019

chrisjackson committed rG87886299b468: [lld] Add Visual Studio compatible diagnostics (authored by chrisjackson).
[lld] Add Visual Studio compatible diagnostics
Jul 17 2019, 7:56 AM

Jul 15 2019

chrisjackson added a comment to D58484: [DO NOT SUBMIT] Add -vs-diagnostics..

Does anyone have any thoughts on these additions?

Jul 15 2019, 5:25 AM · Restricted Project

Jun 27 2019

chrisjackson committed rG41e20d21015b: [llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined… (authored by chrisjackson).
[llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined…
Jun 27 2019, 9:29 AM

Jun 26 2019

chrisjackson updated the diff for D58484: [DO NOT SUBMIT] Add -vs-diagnostics..

Additional regexes to match more messages containing source:line information. Also added a test.

Jun 26 2019, 8:22 AM · Restricted Project

Jun 25 2019

chrisjackson updated the diff for D63340: [llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined binding does not match class assigned by GNU nm.

Tidied up the test and added some additional cases. Added an assert explaining the tested binding values.

Jun 25 2019, 6:59 AM

Jun 14 2019

chrisjackson updated the summary of D63340: [llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined binding does not match class assigned by GNU nm.
Jun 14 2019, 10:05 AM
chrisjackson updated the summary of D63340: [llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined binding does not match class assigned by GNU nm.
Jun 14 2019, 9:41 AM
chrisjackson created D63340: [llvm-nm] Fix for BZ41711 - Class character for a symbol with undefined binding does not match class assigned by GNU nm.
Jun 14 2019, 9:34 AM

Jun 13 2019

chrisjackson committed rG7b395133029a: [llvm-nm] Additional lit tests for command line options (authored by chrisjackson).
[llvm-nm] Additional lit tests for command line options
Jun 13 2019, 3:37 AM

Jun 6 2019

chrisjackson added reviewers for D62955: [llvm-nm] Additional lit tests for command line options: MaskRay, rupprecht, mtrent, evgeny777.
Jun 6 2019, 7:43 AM
chrisjackson added a comment to D62296: [Object] object::ELFObjectFile::symbol_begin(): skip symbol index 0.

Have you made sure that there aren't any uses which would result in an undesired change in behaviour because of this? I feel like it's something that people wouldn't expect to change, and therefore wouldn't test explicitly necessarily.

There should also be a test case somewhere (not that I have a good suggestion for where).

I've checked the call sites:

In RuntimeDyld, if (Flags & SymbolRef::SF_Undefined) continue; filters out index 0.
In llvm-objdump, the filtering code is removed in this patch.
In lib/Object/SymbolSize.cpp, I think the code doesn't notice index 0 is included... but it doesn't seem to matter.
In llvm-nm.cpp, I'll leave the test to @chrisjackson in D62148

Jun 6 2019, 7:27 AM · Restricted Project
chrisjackson created D62955: [llvm-nm] Additional lit tests for command line options.
Jun 6 2019, 7:24 AM

May 21 2019

chrisjackson updated subscribers of D62148: [llvm-nm] Omit the symbol table entry at index 0 when --debug-syms is enabled for ELF files.
May 21 2019, 4:25 AM

May 20 2019

chrisjackson created D62148: [llvm-nm] Omit the symbol table entry at index 0 when --debug-syms is enabled for ELF files.
May 20 2019, 8:30 AM