TableGen backend to generate an index in ctags(1) format.

[kpschoedel]

Summary

TableGen backend to generate an index in ctags(1) format.

Example use:

$ cd $SRCDIR/lib/Target/$TARGET
$ llvm-tblgen -I=../../../include --gen-ctags $TGT.td >tags

To use this in conjunction with exuberant ctags to generate a single
combined tags file, run tblgen first and then

$ ctags --append [...]

Since some identifiers have corresponding definitions in C++ code,
it can be useful (if using vim) to also use cscope, and

:set cscopetagorder=1

so that

:tag X

will preferentially select the tablegen symbol, while

:cscope find g X

will always find the C++ symbol.

[silvas]

I doesn't seem reasonable to be invoking this by hand. Maybe also add a utility script or something that will index all the tablegen files in the project?

[kpschoedel]

Thanks for the feedback.

There doesn't seem to be a good way to identify which files to run llvm-tblgen on without involving the build system (or replicating information from it); that is not done for normal C++ tags, so I didn't want to do it here. (This is not a problem from ctags(1), which merely scans individual source files and does not require them to be buildable.) The added script tries to guess reasonably well, and can be told to shut tblgen up in the cases it gets wrong.

Tested on Linux and OS X.

[paul.redmond]

ping?

I've played with this and it works really well.

[silvas]

utils/TableGen/CTagsEmitter.cpp

11

Cross-reference the helper script here.

utils/TableGen/tdtags

1–18 ↗

(On Diff #1308)

TBQH this is probably the best example of shell scripting I've ever seen (i.e. it's very readable and organized). I honestly wish I could code shell like this.

Despite that, as far as maintainability is concerned, I really don't think that shell is the best language for this script. Pragmatically, I wouldn't trust myself to maintain this file and uphold the same code quality that you have set forth here. Are you familiar enough with Python to write this in Python? For these kinds of scripts, Python seems to be the de facto choice for LLVM. Additionally, Python is, broadly speaking, more portable than shell (for example, Python's platform abstraction extends to Windows as well, while shell is not a "first class citizen" there).

I'd like to emphasize that this script is masterfully written. My concern is that writing correct and robust shell is somewhat of a black art, which hampers the maintainability. I would be glad to point you in the right direction (which API's are best to use, etc.) for a Python script if you aren't familiar.

[muhammadtauqir]

@silvas
I know this has been discussed before (e.g. in this bug report: http://llvm.org/bugs/show_bug.cgi?id=12900) but python is still not being very "portably used" and I still have to manually link which python to which python2 to get the build system to work on my Archlinux where which python is linked to which python3 by default (and I am sure this problem exists on other distros like archbang etc.).

[kpschoedel]

Yeah, I'm stuck in the '90s, where ‘script’ automatically means ‘shell’. I'm not a Python expert, but I expect I can manage. Are there specific Python scripts in the tree that are particularly good examples of the ideal style?

Does the script functionality seem reasonable? We have been using tblgen tags so far only to navigate our back end, so I'm not so sure about other uses (e.g. in clang).

[sdt]

If the script works today, rewriting it in Python now seems unnecessary. Sounds like Kevin will use Python for future scripts, but since this is just a little utility, I think it's fine.

(I have to say I giggled when I saw the size and quality of this script given that it was in response to a one-liner request for improvement :))

[kpschoedel]

I'm certainly willing to redo the script, and it should be easy (a language with arrays?!? Luxury!), but I might not get to it for a week or two. I'm happy to entertain comments on the current or desired functionality in the interim.

[silvas]

utils/TableGen/tdtags

2 ↗

(On Diff #1308)

typo: s/tdtabs/tdtags/

[kpschoedel]

Addressed review comments:

• Reference the helper script here in the tblgen backend.
• Fix typo in tblgen backend.

[kpschoedel]

Rebased against r177639.

[silvas]

Committed as r177682. Please keep an eye on the bots and the commit email in case any issues come up. If possible be on IRC to reduce response latency.

utils/TableGen/CMakeLists.txt

   X86DisassemblerTables.cpp
   X86ModRMFilters.cpp
   X86RecognizableInstr.cpp
+  CTagsEmitter.cpp
   )

utils/TableGen/CTagsEmitter.cpp

+//===- CTagsEmitter.cpp - Generate ctags-compatible index ------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This tablegen backend emits an index of definitions in ctags(1) format.
+//
+//===----------------------------------------------------------------------===//
+
+#define DEBUG_TYPE "ctags-emitter"
+
+#include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/TableGen/Error.h"
+#include "llvm/TableGen/Record.h"
+#include "llvm/TableGen/TableGenBackend.h"
+#include <algorithm>
+#include <string>
+#include <vector>
+using namespace llvm;
+
+namespace llvm {
+extern SourceMgr SrcMgr;
+}
+
+namespace {
+
+class Tag {
+private:
+  const std::string *Id;
+  SMLoc Loc;
+public:
+  Tag(const std::string &Name, const SMLoc Location):
+    Id(&Name), Loc(Location)
+  {}
+  int operator<(const Tag &B) const {
+    return *Id < *B.Id;
+  }
+  void emit(raw_ostream &OS) const {
+    int BufferID = SrcMgr.FindBufferContainingLoc(Loc);
+    MemoryBuffer *CurMB = SrcMgr.getBufferInfo(BufferID).Buffer;
+    const char *BufferName = CurMB->getBufferIdentifier();
+    std::pair<unsigned, unsigned> LineAndColumn = SrcMgr.getLineAndColumn(Loc);
+    OS << *Id << "\t" << BufferName << "\t" << LineAndColumn.first << "\n";
+  }
+};
+
+class CTagsEmitter {
+private:
+  RecordKeeper &Records;
+public:
+  CTagsEmitter(RecordKeeper &R):
+    Records(R)
+  {}
+
+  void run(raw_ostream &OS);
+
+private:
+  static SMLoc locate(const Record *R);
+};
+
+} // End anonymous namespace.
+
+SMLoc CTagsEmitter::locate(const Record *R) {
+  ArrayRef<SMLoc> Locs = R->getLoc();
+  if (Locs.empty()) {
+    SMLoc NullLoc;
+    return NullLoc;
+  }
+  return Locs.front();
+}
+
+void CTagsEmitter::run(raw_ostream &OS) {
+  const std::map<std::string, Record*> &Classes = Records.getClasses();
+  const std::map<std::string, Record*> &Defs = Records.getDefs();
+  std::vector<Tag> Tags;
+  // Collect tags.
+  Tags.reserve(Classes.size() + Defs.size());
+  for (std::map<std::string, Record*>::const_iterator I = Classes.begin(),
+       E = Classes.end(); I != E; ++I)
+    Tags.push_back(Tag(I->first, locate(I->second)));
+  for (std::map<std::string, Record*>::const_iterator I = Defs.begin(),
+        E = Defs.end(); I != E; ++I)
+    Tags.push_back(Tag(I->first, locate(I->second)));
+  // Emit tags.
+  std::sort(Tags.begin(), Tags.end());
+  OS << "!_TAG_FILE_FORMAT\t1\t/original ctags format/\n";
+  OS << "!_TAG_FILE_SORTED\t1\t/0=unsorted, 1=sorted, 2=foldcase/\n";
+  for (std::vector<Tag>::const_iterator I = Tags.begin(), E = Tags.end();
+       I != E; ++I)
+    I->emit(OS);
+}
+
+namespace llvm {
+
+void EmitCTags(RecordKeeper &RK, raw_ostream &OS) {
+  CTagsEmitter(RK).run(OS);
+}
+
+} // End llvm namespace.

utils/TableGen/TableGen.cpp

   GenTgtIntrinsic,
   PrintEnums,
   PrintSets,
-  GenOptParserDefs
+  GenOptParserDefs,
+  GenCTags
 };
 
 namespace {

                                "Print expanded sets for testing DAG exprs"),
                     clEnumValN(GenOptParserDefs, "gen-opt-parser-defs",
                                "Generate option definitions"),
+                    clEnumValN(GenCTags, "gen-ctags",
+                               "Generate ctags-compatible index"),
                     clEnumValEnd));
 
   cl::opt<std::string>

     }
     break;
   }
+  case GenCTags:
+    EmitCTags(Records, OS);
+    break;
   }
 
   return false;

utils/TableGen/TableGenBackends.h

 void EmitSubtarget(RecordKeeper &RK, raw_ostream &OS);
 void EmitMapTable(RecordKeeper &RK, raw_ostream &OS);
 void EmitOptParser(RecordKeeper &RK, raw_ostream &OS);
+void EmitCTags(RecordKeeper &RK, raw_ostream &OS);
 
 } // End llvm namespace