Python reflection: how to list modules and inspect functions

Recently I have been playing with some ideas about applying static analysis to Python and building a Python editor in Jetbrains MPS.

To do any of this I would need to first build a model of Python code. Recently we have seen how to parse Python code, however we still need to consider all the packages our code use. Some of those could be builtin or be implemented through C extensions. That means we do not have python code for them. In this post I look into retrieving a list of all modules and then inspect their contents.

My strategy is to use reflection writing scripts in Python. I will then invoke those scripts from inside Jetbrains MPS (and so from Java code). However this is the topic of a future post.

Listing modules

Listing top modules is relatively easy if you know how to do it. This script prints a list of all top level modules:

Now we need to look inside modules to find sub-modules. For performance reasons I want to do that only when it is needed:

For example for xml I get:

 

Examining module contents and recognizing functions

Now given a module I need to list all its contents. I can load the module by name and iterate over it, printing information about the elements found.
I want to distinguish between classes, submodules (which I will ignore for now), functions and simple values.
Builtin functions need to be treated differently: to access their information I need to parse their documentation. Not cool, not cool at all.

This is what I get for the module os:

Of course for functions I want to build a model of its interface (which parameters it takes, which ones are optional, which ones are variadic and so on). We have the information needed here, it is just a matter of transforming it in a representable form.

Conclusions

I still need to build a model of the imported classes but I starting to have a decent model of the elements I can import in my Python code. This would permit to verify easily which import statements are valid. Of course this can be used in combination with virtualenvs and requirements files: given a list of requirements I would install them in a virtualenv and build the model of the modules available in that virtualenv. I could then statically verify which import would work in that context.

Leave a Reply

Your email address will not be published. Required fields are marked *