Posts

Observers for AST nodes in JavaParser

javaparser-observer-ast

We are getting closer to the first Release Candidate for JavaParser 3.0. One of the last features we added was support for observing changes to all nodes of the Abstract Syntax Tree. While I wrote the code for this feature I received precious feedback from Danny van Bruggen (a.k.a. Matozoid) and Cruz Maximilien. So I use “we” to refer to the JavaParser team.

What observers on AST nodes could be used for?

I think this is a very important feature for the ecosystem of JavaParser because it makes easier to integrate with JavaParser by reacting to the changes made on the AST. Possible changes that can be observed are setting a new name for a class or add a new field. Different tools could react to those changes in different ways. For example:

  • an editor could update its list of symbols, which could be used for things like auto-completion
  • some frameworks could regenerate source code to reflect the changes
  • validation could be performed to verify if the new change lead to an invalid AST
  • libraries like JavaSymbolSolver could recalculate the types for expressions

These are just a few ideas that come to mind but I think that most scenarios in which JavaParser is used could benefit from the possibility to react to changes.

The AstObserver

The JavaParser 3.0 AST is based on Nodes and NodeLists. A Node, like a TypeDeclaration for instance, can have different groups of children. When these groups can contain more than one node we use NodeLists. For example a TypeDeclarations can have multiple members (fields, methods, inner classes). So each TypeDeclaration has a NodeList to contain fields, one to contain methods, etc. Other children, like the name of a TypeDeclaration, are instead directly contain in a node.

We introduced a new interface named AstObserver. An AstObserver receive changes on the Nodes and NodeLists.

What to observe

Now we have an AstObserver and we need to decide which changes it should received. We thought of three possible scenarios:

  1. Observing just one node, for example a ClassDeclaration. The observer would receive notifications for changes on that node (e.g., if the class change name) but not for any of its descendants. For example if a field of the class change name the observer would not be notified
  2. For a node and all its descendants at the moment of registration of the observer. In this case if I register an observer for the ClassDeclaration I would be notified for changes to the class and all its fields and methods. If a new field is added and later modified I would not receive notifications for those changes
  3. For a node and all its descendants, both the ones existing at the moment of registration of the observer and the ones added later.

So a Node has now this method:

To distinguish these three cases we simply use an enum (ObserverRegistrationMode). Later you can see how we implemented the PropagatingAstObserver.

Implementing support for observers

If JavaParser was based on some meta-modeling framework like EMF this would be extremely simple to do. Given this is not the case I needed to add a notification call in all the setters of the AST classes (there are around 90 of those).

So when a setter is invoke on a certain node it notifies all the observers. Simple. Take for example setName in TypeDeclaration<T>:

Given we do not have a proper metamodel we have no definitions for properties. Therefore we added a list of properties in an enum, named ObservableProperty. In this way an Observer can check which property was changed and decide how to react.

Internal hierarchy of observers

For performance reasons each node has its own list of observers. When we want to observe all descendants of a node we simply add the same observer to all nodes and nodelists in that subtree.

However this is not enough, because in some cases you may want to observe also all nodes which are added to the subtree after you have placed your observers. We do that by using a PropagatingAstObserver. It is an AstObserver that when see a new node been attached to a node it is observing start to observe the new node as well. Simple, eh?

Observers in action

Let’s see how this works in practice:

Conclusions

I am quite excited about this new feature because I think it enables more cool stuff to be done with JavaParser. I think our work as committers is to enable other people to do things we are not foreseeing right now. We should just act as enablers and then get out of the way.

I am really curious to see what people will build. By the way, do you know any project using JavaParser that you want to make known to us? Leave a comment or open an issue on GitHub, we are looking forward to hearing from you!

Resolve method calls in Java code using the JavaSymbolSolver

javasymbolsolver

Resolve method calls in Java Code using JavaSymbolSolver

Why I created the java-symbol-solver?

A few years ago I started using JavaParser and then I started contributing. After a while I realized that many operations we want to do on Java code cannot be done just by using the Abstract Syntax Tree produced by a parser, we need also to resolve types, symbols and method calls. For this reason I have created the JavaSymbolSolver. It is now been used to produce static analysis tools by Coati.

One thing that is missing is documentation: people open issues on JavaParser asking how to answer a certain question and the answer is often “for this you need to use JavaSymbolSolver”. Starting from these issues I will show a few examples.

Inspired by this issue I will show how to produce a list of all calls to a specific method.

Learn advanced JavaParser

Javaparser_visited

Receive a chapter on the book JavaParser: Visited.

This chapter presents the JavaSymbolSolver, which you will need for all the advanced analysis and transformation of Java code

Powered by ConvertKit

How can we resolve method calls in Java using the java-symbol-solver?

It can be done in two steps:

  1. You use JavaParser on the source code to build your ASTs
  2. You call JavaSymbolSolver on the nodes of the ASTs representing method calls and get the answer

We are going to write a short example. At the end we will get an application that given a source file will produce this:

We are going to use Kotlin and Gradle. Our build file looks like this:

Building an AST is quite easy, you simply call this method:

What the hell is a Type Solver? It is the object which knows where to look for classes. When processing source code you will typically have references to code that is not yet compiled, but it is just present in other source files. You could also use classes contained in JARs or classes from the Java standard libraries. You have just to tell to your TypeSolver where to look for classes and it will figure it out.

In our example we will parse the source code from the JavaParser project (how meta?!). This project has source code in two different directories, for proper source code and code generated by JavaCC (you can ignore what JavaCC is, it is not relevant to you). We of course use also classes from the java standard libraries. This is how our TypeSolver looks like:

This is where we invoke JavaParserFacade, one of the classes provided by JavaSymbolSolver. We just take a method call at the time and we pass it to the method solve of the JavaParserFacade. We get a MethodUsage (which is basically a method declaration + the value of the parameter types for that specific invocation). From it we get the MethodDeclaration and we print the qualified signature, i.e., the qualified name of the class followed by the signature of the method. This is how we get the final output:

There is so plumbing to do but basically JavaSymbolSolver does all the heavy work behind the scene. Once you have a node of the AST you can throw it at the class JavaParserFacade and it will give you back all the information you may need: it will find corresponding types, fields, methods, etc.

The problem is… we need more documentation and feedback from users. I hope some of you will start using JavaSymbolSolver and tell us how we can improve it.

Also, last week the JavaSymbolSolver was moved under the JavaParser organization. This means that in the future we will work more closely with the JavaParser project.

The code is available on GitHub: java-symbol-solver-examples