Documentation generation with the Compiler API

Documentation of code has a tendency to become obsolete quickly. For that reason most often the only documentation is the code itself and if you are lucky some JavaDoc that is more or less up to date. For that reason I already started some time ago to generate additional documentation by doing static code analysis on my OpenPatrician project. So far this was done mainly with the Reflection API and it was sufficient to figure out what Spring beans there are and where they are used.

The next documentation task was to figure out which class posts what event type on which EventBus and which class handles the event. Or more simple what are the Event producers and what are the Event consumers and mapping them to see how the event messages flow. As the publishing of the events (in the terminology of Guava EventBus ‚post‘) happens within a method, simple reflection will not yield the desired results. Therefore I turned to the Compiler API that can be found in the tools.jar. As that API is poorly documented by JavaDoc and there are not that many examples, I decided to write this post as a means to provide another example.

I will not go into the details how the whole retrieval of all relevant information for the documentation task works, but focus on the most complex part, the figuring out which event types are posted on which EventBuses. First of let’s take a look at the class that we want to inspect:

import com.google.common.eventbus.EventBus;
import javafx.beans.property.DoubleProperty;
import javafx.beans.property.IntegerProperty;
import javafx.beans.property.SimpleDoubleProperty;
import javafx.beans.property.SimpleIntegerProperty;
import javafx.beans.property.SimpleLongProperty;

import java.util.Random;

public class SampleClass {
    private EventBus clientServerEventBus;

    public void directPost() {
        clientServerEventBus.post(new SimpleLongProperty(47L));
    }

    public void indirectPost() {
        IntegerProperty s = new SimpleIntegerProperty();
        DoubleProperty event = new SimpleDoubleProperty();
        clientServerEventBus.post(event);
    }

    public void undecidedInstancePost() {
        IntegerProperty event;
        Random rnd = new Random();
        if (rnd.nextBoolean()) {
            event = new SimpleIntegerProperty(42);
        } else {
            event = new SimpleIntegerProperty(4711);
        }
        clientServerEventBus.post(event);
    }
}

In this example class the event object is created in three different ways:

  1. through direct invocation of the constructor as in the method directPost
  2. through referencing a variable like in indirectPost
  3. through referencing a variable that is uninitialized when defined as in undecidedInstancePost

There are still other cases than the three above, however these were the main cases encountered in my code.

This excellent article started me of in the right direction as it explains, how

  • to set up a source file for compilation from code,
  • to attach a processor to the compilation process
  • to configure the processor with a scanner that follows the visitor pattern to analyse the compilation unit while compiling.

There are however some slight differences between the described solution and my own approach. Starting of with the processor:

import com.sun.source.util.TreePath;
import com.sun.source.util.TreeScanner;
import com.sun.source.util.Trees;

import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import java.util.Set;

@SupportedSourceVersion(SourceVersion.RELEASE_8)
@SupportedAnnotationTypes("*")
public class ASTProcessor extends AbstractProcessor {
    private final TreeScanner scanner;

    private Trees trees;

    public ASTProcessor(TreeScanner scanner) {
        this.scanner = scanner;
    }
    @Override
    public synchronized void init( final ProcessingEnvironment processingEnvironment ) {
        super.init( processingEnvironment );
        trees = Trees.instance( processingEnvironment );
    }

    public boolean process( final Set< ? extends TypeElement> types,
                            final RoundEnvironment environment ) {

        if( !environment.processingOver() ) {
            for( final Element element: environment.getRootElements() ) {
                TreePath p = trees.getPath(element);
                scanner.scan(p.getCompilationUnit(), null);
            }
        }

        return true;
    }
}

The difference here is that I use a TreeScanner instead of a TreePathScanner, even though both will work. The important difference is how the scan method is called: For what I need I require access to all the parts of the compilation unit. When calling it with

scanner.scan( trees.getPath( element ), trees );

certain visitXYZ methods on the scanner will not be called, as they are outside of the scope that is scanned. Mainly these are the CompilationUnit itself as well as all the imports.

The really interesting part is the scanner itself:

import com.google.common.collect.ArrayListMultimap;
import com.google.common.collect.Multimap;
import com.sun.source.tree.ExpressionStatementTree;
import com.sun.source.tree.ExpressionTree;
import com.sun.source.tree.IdentifierTree;
import com.sun.source.tree.ImportTree;
import com.sun.source.tree.MemberSelectTree;
import com.sun.source.tree.MethodInvocationTree;
import com.sun.source.tree.MethodTree;
import com.sun.source.tree.NewClassTree;
import com.sun.source.tree.Tree;
import com.sun.source.tree.Tree.Kind;
import com.sun.source.tree.VariableTree;
import com.sun.source.util.TreeScanner;
import com.sun.source.util.Trees;

import javax.lang.model.element.Name;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * @author Andi Hotz, (c) Sahits GmbH, 2016
 *         Created on Aug 04, 2016
 */
public class ASTScanner extends TreeScanner<Object, Trees> {
    private Map<String, String> imports = new HashMap<>();
    private ProducerContext context = null;

    private String eventBusName;
    private final Multimap<String, Class<?>> eventBusPostEvent = ArrayListMultimap.create();


    public ASTScanner(String eventBusName) {
        this.eventBusName = eventBusName;
    }

    public Multimap<String, Class<?>> getEventBusPostEvent() {
        return eventBusPostEvent;
    }

    @Override
    public Object visitImport(ImportTree node, Trees trees) {
        Tree qualifiedIdentifier = node.getQualifiedIdentifier();
        String qualifiedName = qualifiedIdentifier.toString();
        String classNameSubsting = qualifiedName.substring(qualifiedName.lastIndexOf('.') + 1);
        imports.put(classNameSubsting, qualifiedName);
        return super.visitImport(node, trees);
    }

    @Override
    public Object visitExpressionStatement(ExpressionStatementTree node, Trees trees) {
        if (context != null) {
            if (node.getKind() == Kind.EXPRESSION_STATEMENT) {
                ExpressionTree exprTree = node.getExpression();
                if (exprTree.getKind() == Kind.METHOD_INVOCATION) {
                    MethodInvocationTree methodInvocation = (MethodInvocationTree) exprTree;
                    ExpressionTree methodSelect = methodInvocation.getMethodSelect();
                    if (methodSelect.getKind() == Kind.MEMBER_SELECT) {
                        MemberSelectTree memberSelect = (MemberSelectTree) methodSelect;
                        ExpressionTree reference = memberSelect.getExpression();
                        Name methodName = memberSelect.getIdentifier();
                        if (methodName.contentEquals("post") && eventBusName != null) {
                            List<? extends ExpressionTree> arguments = methodInvocation.getArguments();
                            ExpressionTree firstExpr = arguments.get(0);
                            if (firstExpr.getKind() == Kind.NEW_CLASS) {
                                Class<?> eventType = getEventType((NewClassTree) firstExpr);
                                eventBusPostEvent.put(eventBusName, eventType);
                            }
                            if (firstExpr.getKind() == Kind.IDENTIFIER) {
                                Class<?> eventType = getEventType((IdentifierTree)firstExpr);
                                eventBusPostEvent.put(eventBusName, eventType);
                            }
                        }
                    }
                }
            }
        }
        Object result = super.visitExpressionStatement(node, trees);
        return result;
    }

    private Class<?> getEventType(NewClassTree newExpr) {
        ExpressionTree identifier = newExpr.getIdentifier();
        Name name = ((IdentifierTree) identifier).getName();
        return getClassByName(name);
    }
    private Class<?> getEventType(IdentifierTree identExpr) {
        Name name = identExpr.getName();
        return context.assignedCreatedObjects.get(name);
    }

    @Override
    public Object visitVariable(VariableTree node, Trees trees) {
        if (context != null) {
            if (node.getInitializer() != null) {
                if (node.getInitializer().getKind() == Kind.NEW_CLASS) {
                    addVariableDeclaration(node);
                } else if (node.getInitializer().getKind() == Kind.METHOD_INVOCATION) {
                    addVariableDeclaration(node);
                }
            } else if (node.getKind() == Kind.VARIABLE) {
                // only variable definition without assignment
                addVariableDeclaration(node);
            }
        }
        return super.visitVariable(node, trees);
    }

    private void addVariableDeclaration(VariableTree node) {
        Name name = node.getName();
        Tree type = node.getType();
        if (node.getType() instanceof IdentifierTree) {
            IdentifierTree idenfier = (IdentifierTree) type;
            Name typeName = idenfier.getName();
            try {
                Class<?> clazz = getClassByName(typeName);
                context.assignedCreatedObjects.put(name, clazz);
            } catch (NullPointerException e) {
                // Could not find class => not in the import statements so it cannot be a variable assignment we are interested in.
            }
        }
    }

    private Class<?> getClassByName(Name name) {
        // Match to the imports
        try {
            String className = imports.get(name.toString());
            return getClass().getClassLoader().loadClass(className);
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
            return null;
        }
    }
    @Override
    public Object visitMethod(MethodTree node, Trees trees) {
        String methodName = node.getName().toString();
        context = new ProducerContext(methodName);
        Object result = super.visitMethod(node, trees);
        context = null;
        return result;
    }

    private static class ProducerContext {
        Map<Name, Class<?>> assignedCreatedObjects = new HashMap<>();
        String methodName;

        public ProducerContext(String methodName) {
            this.methodName = methodName;
        }
    }
}

There are three visitXYZ methods that are overridden.

The method visitImport takes care of all the imports and maps the  simple class name to the fully qualified class name. This becomes important when matching Names representing classes to the actual class.

The method visitMethod sets up the context upon which other visit methods operate.

The method visitExpressionSatement is the one that does the actual matching. What we are looking for is a method invocation on an instance of the member field with the name ‚eventBusName‘, that we have passed in as an argument to the constructor. On this field the method ‚post‘ must be called. So far we know that the class we are inspecting makes a post call on that EventBus. What we want to know in addition is which event type is posted. The most simple case from the perspective of the inspection is the direct creation of a new instance within the post method invocation. In that case we can deduce the type right there as the argument is a NewClassExpression from which we can learn the type. However most often the argument to the post method is an identifier referencing a local variable. To match the identifer to the type we have to keep track of the variable declarations.

This is done with visitVariable. That visitor distinguishes between three different cases (though there are more):

  • The variable is defined and instantiated with a NewClassExpression.
  • The variable is defined and instantiated by a method invocation
  • The variable is only defined while not initialized.

Each of these cases provide the type of the variable and the name of the variable. The name is then matched in visitExpressionStatement against the argument.

All this so far only gives us Names, what we actually require however are Classes. Names are basically a String literal. These are retrieved by matching the Names to their fully qualified names and from them load the class from the classloader. For exactly that purpose we went through the import statements. There are two cases that will fail with this approach and are not covered here:

  1. The Name represents a class in the package java.lang which is imported automatically
  2. The Name represents a class in the same package and therefore does not require an import statement.

To bind all this together there is a main class:

import com.google.common.collect.Multimap;

import javax.tools.DiagnosticCollector;
import javax.tools.JavaCompiler;
import javax.tools.JavaCompiler.CompilationTask;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.StandardLocation;
import javax.tools.ToolProvider;
import java.io.File;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collection;


public class ClassAnalyzer {

    public static void main(String[] args) {
        ASTScanner scanner = new ASTScanner("clientServerEventBus");
        ASTProcessor processor = new ASTProcessor(scanner);

        final DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<>();
        final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();

        final File file = new File("src/SampleClass.java");
        try (final StandardJavaFileManager manager =
                     compiler.getStandardFileManager(diagnostics, null, null)) {
            final Iterable<? extends JavaFileObject> sources = manager.getJavaFileObjectsFromFiles(Arrays.asList(file));
            String outputDir = System.getProperty("java.io.tmpdir");
            manager.setLocation(StandardLocation.CLASS_OUTPUT, Arrays.asList(new File(outputDir)));
            final CompilationTask task = compiler.getTask(null, manager, diagnostics, null, null, sources);
            task.setProcessors(Arrays.asList(processor));
            task.call();

        } catch (IOException e) {
            e.printStackTrace();
        }
        Multimap<String, Class<?>> postedEventsOnEventBus = scanner.getEventBusPostEvent();
        for (String eventBus : postedEventsOnEventBus.keySet()) {
            System.out.println("On EventBus '"+eventBus+"' the following event types are posted:");
            Collection<Class<?>> col = postedEventsOnEventBus.get(eventBus);
            for (Class<?> clazz : col) {
                System.out.println("\t- " + clazz.getName());
            }
        }
    }
}

This class will generate the following output:

On EventBus 'clientServerEventBus' the following event types are posted:
	- javafx.beans.property.SimpleLongProperty
	- javafx.beans.property.DoubleProperty
	- javafx.beans.property.IntegerProperty

This is exactly what we expected.

2 Gedanken zu „Documentation generation with the Compiler API“

  1. Pingback: Y-O-Y? (Part I) |

Schreibe einen Kommentar