Page MenuHomePhabricator

Create an lldb/third_party folder, and add Python module 'six' to it.
ClosedPublic

Authored by zturner on Oct 19 2015, 4:10 PM.

Details

Summary

six is a Python module licensed under MIT which provides a compatibility layer between Python 2 and Python 3. You can read about it at the project's homepage, here. https://pypi.python.org/pypi/six

six consists of only a single python file, so it's very easy to drop into an existing project, works with any Python version >= 2.6, and is permissively licensed under MIT so should present no problem being in our repo.

The immediate need for this is that many modules and functions were either renamed or deleted in Python 3. A few examples of this:
Renamed modules:

StringIO -> io
cPickle -> pickle
Queue -> queue

Renamed methods, literal syntax, etc:

unichr -> chr
u"foo" -> "foo"
0700 -> 0o700  # octal literal

There are other examples as well. If we have to these behind a runtime version check at every usage site, it's going to be ugly, error prone, and cause a maintenance burden. With six this becomes very transparent and should make it easy to have consistent behavior running under both supported versions of Python.

Note that this patch does not do anything to get six into lldb's site-packages folder or hook it up to anything. This CL is only to get approval for "yes you can add this to LLDB". Hooking it up will come later.

Diff Detail

Event Timeline

zturner updated this revision to Diff 37812.Oct 19 2015, 4:10 PM
zturner retitled this revision from to Create an lldb/third_party folder, and add Python module 'six' to it..
zturner updated this object.
zturner added reviewers: tfiala, clayborg, dberlin.
zturner updated this object.
zturner updated this object.
zturner added a subscriber: lldb-commits.

Todd, do you have any good ideas on how to organize this and make it play nicely with python's module / package system? dotest is just an arbitrary script under lldb/test and is not part of any installed package or anything. At the same time, third party code needs to be in a centralized place at the top-level in case it needs to be tracked or audited by legal people. So I don't think we can put six under test (nor would it even make sense to, since that wouldn't help people under lldb/scripts who also want to use it).

So somehow lldb/scripts and lldb/test both need to be able to import from lldb/third_party/Python/modules.

I know this isn't really the "pythonic" way, because it expects everything to be nicely organized packages with a nice hierarchy, but dotest is treated as a one-off script. I don't really want people to have to install six via setuptools or anything because it just adds an unnecessary extra step which is a regression from the status quo.

Do absolute imports help us? (https://docs.python.org/2.5/whatsnew/pep-328.html)

I think this question speaks to a more general problem of: "How do we re-use python code across tests and scripts?" Do you have any suggestions that can make this work cleanly?

dberlin edited edge metadata.Oct 20 2015, 9:51 AM

LGTM from a license perspective. It doesn't impose any interesting obligations.

Greg, do you have any concerns here?

tfiala edited edge metadata.Oct 20 2015, 1:51 PM

Todd, do you have any good ideas on how to organize this and make it play nicely with python's module / package system? dotest is just an arbitrary script under lldb/test and is not part of any installed package or anything. At the same time, third party code needs to be in a centralized place at the top-level in case it needs to be tracked or audited by legal people. So I don't think we can put six under test (nor would it even make sense to, since that wouldn't help people under lldb/scripts who also want to use it).

So somehow lldb/scripts and lldb/test both need to be able to import from lldb/third_party/Python/modules.

I know this isn't really the "pythonic" way, because it expects everything to be nicely organized packages with a nice hierarchy, but dotest is treated as a one-off script. I don't really want people to have to install six via setuptools or anything because it just adds an unnecessary extra step which is a regression from the status quo.

Do absolute imports help us? (https://docs.python.org/2.5/whatsnew/pep-328.html)

I think this question speaks to a more general problem of: "How do we re-use python code across tests and scripts?" Do you have any suggestions that can make this work cleanly?

Hmm, registering that I see the question. I need to think about it.

Generally I have an upper level script (such as dotest.py in our case) that gets called directly, and adds paths relative to its script directory to set up all the other bits it needs to find. One example of where I do this in lldb is a util that probably nobody else uses, utils/sync-source/syncsource.py. It imports os and sys, and from that, bootstraps its sys.path (PYTHONPATH) by adding the script-file-relative lib dir to the python path.

So the technique is find an anchor point (in the case above, dotest.py) and have it add the script-relative path to the python path.

I do the same thing to adjust the pylint .pylintrc python path, so that pylint can find not only the lldbtest path, but also the lldb module executable (via lldb[.exe] -P).

Not sure if that helps?

Todd, do you have any good ideas on how to organize this and make it play nicely with python's module / package system? dotest is just an arbitrary script under lldb/test and is not part of any installed package or anything. At the same time, third party code needs to be in a centralized place at the top-level in case it needs to be tracked or audited by legal people. So I don't think we can put six under test (nor would it even make sense to, since that wouldn't help people under lldb/scripts who also want to use it).

So somehow lldb/scripts and lldb/test both need to be able to import from lldb/third_party/Python/modules.

I know this isn't really the "pythonic" way, because it expects everything to be nicely organized packages with a nice hierarchy, but dotest is treated as a one-off script. I don't really want people to have to install six via setuptools or anything because it just adds an unnecessary extra step which is a regression from the status quo.

Do absolute imports help us? (https://docs.python.org/2.5/whatsnew/pep-328.html)

I think this question speaks to a more general problem of: "How do we re-use python code across tests and scripts?" Do you have any suggestions that can make this work cleanly?

Hmm, registering that I see the question. I need to think about it.

Generally I have an upper level script (such as dotest.py in our case) that gets called directly, and adds paths relative to its script directory to set up all the other bits it needs to find. One example of where I do this in lldb is a util that probably nobody else uses, utils/sync-source/syncsource.py. It imports os and sys, and from that, bootstraps its sys.path (PYTHONPATH) by adding the script-file-relative lib dir to the python path.

So the technique is find an anchor point (in the case above, dotest.py) and have it add the script-relative path to the python path.

I do the same thing to adjust the pylint .pylintrc python path, so that pylint can find not only the lldbtest path, but also the lldb module executable (via lldb[.exe] -P).

That example would be in test/lldb_pylint_helper.py (which also documents how to use it to adjust your global pylintrc to make this all work, which becomes a bootstrap process much like what I think you're asking about here, albeit in the context of getting the python path setup for pylint rather than for the file itself --- but each file could have a little loader that does something like that).

Not sure if that helps?

tfiala accepted this revision.Oct 20 2015, 1:56 PM
tfiala edited edge metadata.

All that aside, LGTM.

This revision is now accepted and ready to land.Oct 20 2015, 1:56 PM

Yea, I actually did something very similar to that. Check out D13906

zturner closed this revision.Oct 20 2015, 2:11 PM