Fix flags for compiling examples/Kaleidoscope/MCJIT

[jmed]

Summary

This patch contains minor changes I made to get examples/Kaleidoscope/MCJIT/initial/toy.cpp to compile, discussed in the following bug report:
http://llvm.org/bugs/show_bug.cgi?id=20329

[rengolin]

examples/Kaleidoscope/MCJIT/initial/Makefile

4

This should be ok, but again, people that have worked on the tutorial should have a better opinion.

examples/Kaleidoscope/MCJIT/initial/toy.cpp

794

It seems that all MCJIT Kaleidoscope files have this, as well as the documentation in docs/tutorial. If you're going to change, better change all.

Also, from other patterns, the DataLayoutPass has the following usage:

new DataLayoutPass(Module)

which is not what you've used. I'm not sure how the pass manager is doing this at the moment, but other people may comment on that. Either way, better change all other files while you're at it.

[jmed]

Added fixes for compiling all Kaleidoscope/MCJIT examples (patch made against r213970). Compiled these on x86-64 & ARMv7a and ran as described in the test plan. Summary of changes:

• Fixed compile flags to use llvm-config --cxxflags instead of llvm-config --cppflags
• Added -pthread -ldl -lcurses -lz linking flags (zlib is apparently a dependency of libLLVMSupport, should -lz be added to llvm-config?)
• API update: raw_fd_ostream constructor needs sys::fs::F_None as flags param (prev. value removed)
• API update: usage of MemoryBuffer::getFile changed
• API update: RTDyldMemoryManager::getPointerToNamedFunction deprecated; use getSymbolAddress
• API update: use DataLayoutPass(Module*) instead of DataLayout for FPM (now using the preferred signature which passes the module)

Additionally, the code that enabled object caching wasn't being triggered. I prefixed "IR:" to the module ID to cause the caching to happen. Additionally, -use-object-cache needs to be specified on the command line, which I added to the python scripts generating the timing in MCJIT/cached/genk-timing.py.

[rengolin]

I can't comment on the other changes, which seem to be variations of the same fix on different files (and why do we have that many similar files bothers me), but I'll let someone that has actually changed Kaleidoscope recently to reply as to its validity.

My concerns were addressed. Thanks!

[rengolin]

LGTM, but please get Andrew's, Lang's (or someone else closer to MCJIT) to approve.

cheers,
--renato

[rengolin]

Hi,

Has this gone in? Is it still required/desired?

If yes, please ping the appropriate people, if not, please drop it.

Thanks,
--renato

[lhames]

Apologies for the very tardy review.

This all looks good to me, with one quibble: Where the link lines all include "-pthread -ldl -lcurses -lz", it would be preferable to add the --system-libs option to the llvm-config invocation instead. That should pull in those libraries.

If you have commit access, please go ahead and commit after making that change. If not, just ping me and I will commit it for you.

Thanks very much for your work on this Jesse.

[rengolin]

Any progress in this?

examples/Kaleidoscope/MCJIT/cached/Makefile

 all: toy-mcjit toy-jit toy-ir-gen
 
 toy-mcjit : toy.cpp
-  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cppflags --ldflags --libs core mcjit native irreader` -o toy-mcjit
+  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cxxflags --ldflags --libs core mcjit native irreader` -pthread -ldl -lcurses -lz -o toy-mcjit
 
 toy-jit : toy-jit.cpp
-  clang++ toy-jit.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cppflags --ldflags --libs core jit native irreader` -o toy-jit
+  clang++ toy-jit.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cxxflags --ldflags --libs core jit native irreader` -pthread -ldl -lcurses -lz -o toy-jit
 
 # This is a special build for the purpose of converting Kaleidoscope input to an IR file
 toy-ir-gen : toy-jit.cpp
-  clang++ toy-jit.cpp -g -O3 -rdynamic -fno-rtti -DDUMP_FINAL_MODULE `llvm-config --cppflags --ldflags --libs core jit native irreader` -o toy-ir-gen
+  clang++ toy-jit.cpp -g -O3 -rdynamic -fno-rtti -DDUMP_FINAL_MODULE `llvm-config --cxxflags --ldflags --libs core jit native irreader` -pthread -ldl -lcurses -lz -o toy-ir-gen

examples/Kaleidoscope/MCJIT/cached/genk-timing.py

         self.shfile.write("echo \"With MCJIT\" >> %s\n" % self.timeFile)
         self.shfile.write("/usr/bin/time -f \"Command %C\\n\\tuser time: %U s\\n\\tsytem time: %S s\\n\\tmax set: %M kb\"")
         self.shfile.write(" -o %s -a " % self.timeFile)
-        self.shfile.write("./toy-mcjit < %s > %s-mcjit.out 2> %s-mcjit.err\n" % (filename, rootname, rootname))
+        self.shfile.write("./toy-mcjit -use-object-cache < %s > %s-mcjit.out 2> %s-mcjit.err\n" % (filename, rootname, rootname))
         self.shfile.write("echo \"\" >> %s\n" % self.timeFile)
         self.shfile.write("echo \"With JIT\" >> %s\n" % self.timeFile)
         self.shfile.write("/usr/bin/time -f \"Command %C\\n\\tuser time: %U s\\n\\tsytem time: %S s\\n\\tmax set: %M kb\"")

examples/Kaleidoscope/MCJIT/cached/toy-jit.cpp

 
   // Set up the optimizer pipeline.  Start with registering info about how the
   // target lays out data structures.
-  OurFPM.add(new DataLayout(*TheExecutionEngine->getDataLayout()));
+  OurFPM.add(new DataLayoutPass(TheModule));
   // Provide basic AliasAnalysis support for GVN.
   OurFPM.add(createBasicAliasAnalysisPass());
   // Promote allocas to registers.

examples/Kaleidoscope/MCJIT/cached/toy.cpp

         return;
       }
       std::string ErrStr;
-      raw_fd_ostream IRObjectFile(IRCacheFile.c_str(), ErrStr, raw_fd_ostream::F_Binary);
+      raw_fd_ostream IRObjectFile(IRCacheFile.c_str(), ErrStr, sys::fs::F_None);
       IRObjectFile << Obj->getBuffer();
     }
   }

         // This file isn't in our cache
         return NULL;
       }
-      std::unique_ptr<MemoryBuffer> IRObjectBuffer;
-      MemoryBuffer::getFile(IRCacheFile.c_str(), IRObjectBuffer, -1, false);
+      
+      ErrorOr<std::unique_ptr<MemoryBuffer>> IRObjectBuffer = 
+          MemoryBuffer::getFile(IRCacheFile.c_str(), -1, false);
+      if(IRObjectBuffer.getError()) {
+        fprintf(stderr, "Unable to read from cache\n");
+        return NULL;
+      }
+      
       // MCJIT will want to write into this buffer, and we don't want that
       // because the file has probably just been mmapped.  Instead we make
       // a copy.  The filed-based buffer will be released when it goes
       // out of scope.
-      return MemoryBuffer::getMemBufferCopy(IRObjectBuffer->getBuffer());
+      return MemoryBuffer::getMemBufferCopy(IRObjectBuffer.get()->getBuffer());
     }
 
     return NULL;

   /// Our implementation will attempt to find functions in other
   /// modules associated with the MCJITHelper to cross link functions
   /// from one generated module to another.
-  ///
-  /// If \p AbortOnFailure is false and no function with the given name is
-  /// found, this function returns a null pointer. Otherwise, it prints a
-  /// message to stderr and aborts.
-  virtual void *getPointerToNamedFunction(const std::string &Name,
-                                          bool AbortOnFailure = true);
+  virtual uint64_t getSymbolAddress(const std::string &Name);
 private:
   MCJITHelper *MasterHelper;
 };
 
-void *HelpingMemoryManager::getPointerToNamedFunction(const std::string &Name,
-                                        bool AbortOnFailure)
+uint64_t HelpingMemoryManager::getSymbolAddress(const std::string &Name)
 {
   // Try the standard symbol resolution first, but ask it not to abort.
-  void *pfn = SectionMemoryManager::getPointerToNamedFunction(Name, false);
-  if (pfn)
-    return pfn;
+  uint64_t faddr = SectionMemoryManager::getSymbolAddress(Name);
+  if (faddr)
+    return faddr;
 
-  pfn = MasterHelper->getPointerToNamedFunction(Name);
-  if (!pfn && AbortOnFailure)
+  faddr = (uint64_t)MasterHelper->getPointerToNamedFunction(Name);
+  if (!faddr)
     report_fatal_error("Program used external function '" + Name +
                         "' which could not be resolved!");
-  return pfn;
+  return faddr;
 }
 
 MCJITHelper::~MCJITHelper()

     return OpenModule;
 
   // Otherwise create a new Module.
-  std::string ModName = GenerateUniqueName("mcjit_module_");
+  std::string ModName = GenerateUniqueName("IR:mcjit_module_");
   Module *M = new Module(ModName, Context);
   Modules.push_back(M);
   OpenModule = M;

 
     // Set up the optimizer pipeline.  Start with registering info about how the
     // target lays out data structures.
-    FPM->add(new DataLayout(*NewEngine->getDataLayout()));
+    FPM->add(new DataLayoutPass(M));
     // Provide basic AliasAnalysis support for GVN.
     FPM->add(createBasicAliasAnalysisPass());
     // Promote allocas to registers.

examples/Kaleidoscope/MCJIT/complete/Makefile

 all: toy
 
 toy : toy.cpp
-  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cppflags --ldflags --libs core jit mcjit native irreader` -o toy
+  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cxxflags --ldflags --libs core jit mcjit native irreader` -pthread -ldl -lcurses -lz -o toy

examples/Kaleidoscope/MCJIT/complete/toy.cpp

         return;
       }
       std::string ErrStr;
-      raw_fd_ostream IRObjectFile(IRCacheFile.c_str(), ErrStr, raw_fd_ostream::F_Binary);
+      raw_fd_ostream IRObjectFile(IRCacheFile.c_str(), ErrStr, sys::fs::F_None);
       IRObjectFile << Obj->getBuffer();
     }
   }

         // This file isn't in our cache
         return NULL;
       }
-      std::unique_ptr<MemoryBuffer> IRObjectBuffer;
-      MemoryBuffer::getFile(IRCacheFile.c_str(), IRObjectBuffer, -1, false);
+      ErrorOr<std::unique_ptr<MemoryBuffer>> IRObjectBuffer = 
+          MemoryBuffer::getFile(IRCacheFile.c_str(), -1, false);
+      if(IRObjectBuffer.getError()) {
+        fprintf(stderr, "Unable to read from cache\n");
+        return NULL;
+      }
+      
       // MCJIT will want to write into this buffer, and we don't want that
       // because the file has probably just been mmapped.  Instead we make
       // a copy.  The filed-based buffer will be released when it goes
       // out of scope.
-      return MemoryBuffer::getMemBufferCopy(IRObjectBuffer->getBuffer());
+      return MemoryBuffer::getMemBufferCopy(IRObjectBuffer.get()->getBuffer());
     }
 
     return NULL;

 
     // Set up the optimizer pipeline.  Start with registering info about how the
     // target lays out data structures.
-    TheFPM->add(new DataLayout(*TheExecutionEngine->getDataLayout()));
+    TheFPM->add(new DataLayoutPass(TheModule));
     // Provide basic AliasAnalysis support for GVN.
     TheFPM->add(createBasicAliasAnalysisPass());
     // Promote allocas to registers.

   /// Our implementation will attempt to find functions in other
   /// modules associated with the MCJITHelper to cross link functions
   /// from one generated module to another.
-  ///
-  /// If \p AbortOnFailure is false and no function with the given name is
-  /// found, this function returns a null pointer. Otherwise, it prints a
-  /// message to stderr and aborts.
-  virtual void *getPointerToNamedFunction(const std::string &Name,
-                                          bool AbortOnFailure = true);
+  virtual uint64_t getSymbolAddress(const std::string &Name);
 private:
   MCJITHelper *MasterHelper;
 };
 
-void *HelpingMemoryManager::getPointerToNamedFunction(const std::string &Name,
-                                        bool AbortOnFailure)
+uint64_t HelpingMemoryManager::getSymbolAddress(const std::string &Name)
 {
   // Try the standard symbol resolution first, but ask it not to abort.
-  void *pfn = RTDyldMemoryManager::getPointerToNamedFunction(Name, false);
-  if (pfn)
-    return pfn;
+  uint64_t faddr = SectionMemoryManager::getSymbolAddress(Name);
+  if (faddr)
+    return faddr;
 
-  pfn = MasterHelper->getPointerToNamedFunction(Name);
-  if (!pfn && AbortOnFailure)
+  faddr = (uint64_t)MasterHelper->getPointerToNamedFunction(Name);
+  if (!faddr)
     report_fatal_error("Program used external function '" + Name +
                         "' which could not be resolved!");
-  return pfn;
+  return faddr;
 }
 
 MCJITHelper::~MCJITHelper()

     return CurrentModule;
 
   // Otherwise create a new Module.
-  std::string ModName = GenerateUniqueName("mcjit_module_");
+  std::string ModName = GenerateUniqueName("IR:mcjit_module_");
   Module *M = new Module(ModName, Context);
   Modules.push_back(M);
   CurrentModule = M;

 
     // Set up the optimizer pipeline.  Start with registering info about how the
     // target lays out data structures.
-    FPM->add(new DataLayout(*EE->getDataLayout()));
+    FPM->add(new DataLayoutPass(M));
     // Provide basic AliasAnalysis support for GVN.
     FPM->add(createBasicAliasAnalysisPass());
     // Promote allocas to registers.

examples/Kaleidoscope/MCJIT/initial/Makefile

 all: toy-mcjit
 
 toy-mcjit : toy.cpp
-  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cppflags --ldflags --libs core mcjit native` -o toy-mcjit
+  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cxxflags --ldflags --libs core mcjit native` -pthread -ldl -lcurses -lz -o toy-mcjit

examples/Kaleidoscope/MCJIT/initial/toy.cpp

   /// Our implementation will attempt to find functions in other
   /// modules associated with the MCJITHelper to cross link functions
   /// from one generated module to another.
-  ///
-  /// If \p AbortOnFailure is false and no function with the given name is
-  /// found, this function returns a null pointer. Otherwise, it prints a
-  /// message to stderr and aborts.
-  virtual void *getPointerToNamedFunction(const std::string &Name,
-                                          bool AbortOnFailure = true);
+  virtual uint64_t getSymbolAddress(const std::string &Name);
 private:
   MCJITHelper *MasterHelper;
 };
 
-void *HelpingMemoryManager::getPointerToNamedFunction(const std::string &Name,
-                                        bool AbortOnFailure)
+uint64_t HelpingMemoryManager::getSymbolAddress(const std::string &Name)
 {
   // Try the standard symbol resolution first, but ask it not to abort.
-  void *pfn = SectionMemoryManager::getPointerToNamedFunction(Name, false);
-  if (pfn)
-    return pfn;
+  uint64_t faddr = SectionMemoryManager::getSymbolAddress(Name);
+  if (faddr)
+    return faddr;
 
-  pfn = MasterHelper->getPointerToNamedFunction(Name);
-  if (!pfn && AbortOnFailure)
+  faddr = (uint64_t)MasterHelper->getPointerToNamedFunction(Name);
+  if (!faddr)
     report_fatal_error("Program used external function '" + Name +
                         "' which could not be resolved!");
-  return pfn;
+  return faddr;
 }
 
 MCJITHelper::~MCJITHelper()

 
     // Set up the optimizer pipeline.  Start with registering info about how the
     // target lays out data structures.
-    FPM->add(new DataLayout(*NewEngine->getDataLayout()));
+    FPM->add(new DataLayoutPass(OpenModule));
     // Provide basic AliasAnalysis support for GVN.
     FPM->add(createBasicAliasAnalysisPass());
     // Promote allocas to registers.

examples/Kaleidoscope/MCJIT/lazy/Makefile

 all: toy-mcjit toy-jit
 
 toy-mcjit : toy.cpp
-  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cppflags --ldflags --libs core mcjit native` -o toy-mcjit
+  clang++ toy.cpp -g -O3 -rdynamic -fno-rtti `llvm-config --cxxflags --ldflags --libs core mcjit native` -pthread -ldl -lcurses -lz -o toy-mcjit
 
 toy-jit : toy-jit.cpp
-  clang++ toy-jit.cpp -g -O3 -rdynamic `llvm-config --cppflags --ldflags --libs core jit native` -o toy-jit
+  clang++ toy-jit.cpp -g -O3 -rdynamic `llvm-config --cxxflags --ldflags --libs core jit native` -pthread -ldl -lcurses -lz -o toy-jit

examples/Kaleidoscope/MCJIT/lazy/toy-jit.cpp

 
   // Set up the optimizer pipeline.  Start with registering info about how the
   // target lays out data structures.
-  OurFPM.add(new DataLayout(*TheExecutionEngine->getDataLayout()));
+  OurFPM.add(new DataLayoutPass(TheModule));
   // Provide basic AliasAnalysis support for GVN.
   OurFPM.add(createBasicAliasAnalysisPass());
   // Promote allocas to registers.

examples/Kaleidoscope/MCJIT/lazy/toy.cpp

   /// Our implementation will attempt to find functions in other
   /// modules associated with the MCJITHelper to cross link functions
   /// from one generated module to another.
-  ///
-  /// If \p AbortOnFailure is false and no function with the given name is
-  /// found, this function returns a null pointer. Otherwise, it prints a
-  /// message to stderr and aborts.
-  virtual void *getPointerToNamedFunction(const std::string &Name,
-                                          bool AbortOnFailure = true);
+  virtual uint64_t getSymbolAddress(const std::string &Name);
 private:
   MCJITHelper *MasterHelper;
 };
 
-void *HelpingMemoryManager::getPointerToNamedFunction(const std::string &Name,
-                                        bool AbortOnFailure)
+uint64_t HelpingMemoryManager::getSymbolAddress(const std::string &Name)
 {
   // Try the standard symbol resolution first, but ask it not to abort.
-  void *pfn = SectionMemoryManager::getPointerToNamedFunction(Name, false);
-  if (pfn)
-    return pfn;
+  uint64_t faddr = SectionMemoryManager::getSymbolAddress(Name);
+  if (faddr)
+    return faddr;
 
-  pfn = MasterHelper->getPointerToNamedFunction(Name);
-  if (!pfn && AbortOnFailure)
+  faddr = (uint64_t)MasterHelper->getPointerToNamedFunction(Name);
+  if (!faddr)
     report_fatal_error("Program used external function '" + Name +
                         "' which could not be resolved!");
-  return pfn;
+  return faddr;
 }
 
 MCJITHelper::~MCJITHelper()

 
   // Set up the optimizer pipeline.  Start with registering info about how the
   // target lays out data structures.
-  FPM->add(new DataLayout(*NewEngine->getDataLayout()));
+    FPM->add(new DataLayoutPass(M));
   // Provide basic AliasAnalysis support for GVN.
   FPM->add(createBasicAliasAnalysisPass());
   // Promote allocas to registers.