{"id":2531,"date":"2016-08-06T20:29:42","date_gmt":"2016-08-06T19:29:42","guid":{"rendered":"http:\/\/sahits.ch\/blog\/?p=2531"},"modified":"2016-08-06T20:29:42","modified_gmt":"2016-08-06T19:29:42","slug":"documentation-generation-with-the-compiler-api","status":"publish","type":"post","link":"http:\/\/sahits.ch\/blog\/blog\/2016\/08\/06\/documentation-generation-with-the-compiler-api\/","title":{"rendered":"Documentation generation with the Compiler API"},"content":{"rendered":"<p>Documentation of code has a tendency to become obsolete quickly. For that reason most often the only documentation is the code itself and if you are lucky some JavaDoc that is more or less up to date. For that reason I already started some time ago to generate additional documentation by doing static code analysis on my <a href=\"https:\/\/sourceforge.net\/projects\/openpatrician\/\">OpenPatrician<\/a> project. So far this was done mainly with the Reflection API and it was sufficient to figure out what Spring beans there are and where they are used.<\/p>\n<p>The next documentation task was to figure out which class posts what event type on which <a href=\"https:\/\/github.com\/google\/guava\/wiki\/EventBusExplained\">EventBus<\/a> and which class handles the event. Or more simple what are the Event producers and what are the Event consumers and mapping them to see how the event messages flow. As the publishing of the events (in the terminology of Guava EventBus &#8218;post&#8216;) happens within a method, simple reflection will not yield the desired results. Therefore I turned to the <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/javax\/tools\/JavaCompiler.html\">Compiler API <\/a>that can be found in the tools.jar. As that API is poorly documented by JavaDoc and there are not that many examples, I decided to write this post as a means to provide another example.<\/p>\n<p><!--more--><\/p>\n<p>I will not go into the details how the whole retrieval of all relevant information for the documentation task works, but focus on the most complex part, the figuring out which event types are posted on which EventBuses. First of let&#8217;s take a look at the class that we want to inspect:<\/p>\n<pre class=\"brush:java\">import com.google.common.eventbus.EventBus;\r\nimport javafx.beans.property.DoubleProperty;\r\nimport javafx.beans.property.IntegerProperty;\r\nimport javafx.beans.property.SimpleDoubleProperty;\r\nimport javafx.beans.property.SimpleIntegerProperty;\r\nimport javafx.beans.property.SimpleLongProperty;\r\n\r\nimport java.util.Random;\r\n\r\npublic class SampleClass {\r\n    private EventBus clientServerEventBus;\r\n\r\n    public void directPost() {\r\n        clientServerEventBus.post(new SimpleLongProperty(47L));\r\n    }\r\n\r\n    public void indirectPost() {\r\n        IntegerProperty s = new SimpleIntegerProperty();\r\n        DoubleProperty event = new SimpleDoubleProperty();\r\n        clientServerEventBus.post(event);\r\n    }\r\n\r\n    public void undecidedInstancePost() {\r\n        IntegerProperty event;\r\n        Random rnd = new Random();\r\n        if (rnd.nextBoolean()) {\r\n            event = new SimpleIntegerProperty(42);\r\n        } else {\r\n            event = new SimpleIntegerProperty(4711);\r\n        }\r\n        clientServerEventBus.post(event);\r\n    }\r\n}\r\n<\/pre>\n<p>In this example class the event object is created in three different ways:<\/p>\n<ol>\n<li>through direct invocation of the constructor as in the method <em>directPost<\/em><\/li>\n<li>through referencing a variable like in <em>indirectPost<\/em><\/li>\n<li>through referencing a variable that is uninitialized when defined as in <em>undecidedInstancePost<\/em><\/li>\n<\/ol>\n<p>There are still other cases than the three above, however these were the main cases encountered in my code.<\/p>\n<p>This <a href=\"https:\/\/www.javacodegeeks.com\/2015\/09\/java-compiler-api.html\">excellent article<\/a> started me of in the right direction as it explains, how<\/p>\n<ul>\n<li>to set up a source file for compilation from code,<\/li>\n<li>to attach a processor to the compilation process<\/li>\n<li>to configure the processor with a scanner that follows the visitor pattern to analyse the compilation unit while compiling.<\/li>\n<\/ul>\n<p>There are however some slight differences between the described solution and my own approach. Starting of with the processor:<\/p>\n<pre class=\"brush:java\">import com.sun.source.util.TreePath;\r\nimport com.sun.source.util.TreeScanner;\r\nimport com.sun.source.util.Trees;\r\n\r\nimport javax.annotation.processing.AbstractProcessor;\r\nimport javax.annotation.processing.ProcessingEnvironment;\r\nimport javax.annotation.processing.RoundEnvironment;\r\nimport javax.annotation.processing.SupportedAnnotationTypes;\r\nimport javax.annotation.processing.SupportedSourceVersion;\r\nimport javax.lang.model.SourceVersion;\r\nimport javax.lang.model.element.Element;\r\nimport javax.lang.model.element.TypeElement;\r\nimport java.util.Set;\r\n\r\n@SupportedSourceVersion(SourceVersion.RELEASE_8)\r\n@SupportedAnnotationTypes(\"*\")\r\npublic class ASTProcessor extends AbstractProcessor {\r\n    private final TreeScanner scanner;\r\n\r\n    private Trees trees;\r\n\r\n    public ASTProcessor(TreeScanner scanner) {\r\n        this.scanner = scanner;\r\n    }\r\n    @Override\r\n    public synchronized void init( final ProcessingEnvironment processingEnvironment ) {\r\n        super.init( processingEnvironment );\r\n        trees = Trees.instance( processingEnvironment );\r\n    }\r\n\r\n    public boolean process( final Set&lt; ? extends TypeElement&gt; types,\r\n                            final RoundEnvironment environment ) {\r\n\r\n        if( !environment.processingOver() ) {\r\n            for( final Element element: environment.getRootElements() ) {\r\n                TreePath p = trees.getPath(element);\r\n                scanner.scan(p.getCompilationUnit(), null);\r\n            }\r\n        }\r\n\r\n        return true;\r\n    }\r\n}\r\n<\/pre>\n<p>The difference here is that I use a <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/jdk\/api\/javac\/tree\/com\/sun\/source\/util\/TreeScanner.html\">TreeScanner<\/a> instead of a <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/jdk\/api\/javac\/tree\/com\/sun\/source\/util\/TreePathScanner.html\">TreePathScanner<\/a>, even though both will work. The important difference is how the scan method is called: For what I need I require access to all the parts of the compilation unit. When calling it with<\/p>\n<pre class=\"brush:java\">scanner.scan( trees.getPath( element ), trees );<\/pre>\n<p>certain visitXYZ methods on the scanner will not be called, as they are outside of the scope that is scanned. Mainly these are the <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/jdk\/api\/javac\/tree\/com\/sun\/source\/tree\/CompilationUnitTree.html\">CompilationUnit<\/a> itself as well as all the imports.<\/p>\n<p>The really interesting part is the scanner itself:<\/p>\n<pre class=\"brush:java\">import com.google.common.collect.ArrayListMultimap;\r\nimport com.google.common.collect.Multimap;\r\nimport com.sun.source.tree.ExpressionStatementTree;\r\nimport com.sun.source.tree.ExpressionTree;\r\nimport com.sun.source.tree.IdentifierTree;\r\nimport com.sun.source.tree.ImportTree;\r\nimport com.sun.source.tree.MemberSelectTree;\r\nimport com.sun.source.tree.MethodInvocationTree;\r\nimport com.sun.source.tree.MethodTree;\r\nimport com.sun.source.tree.NewClassTree;\r\nimport com.sun.source.tree.Tree;\r\nimport com.sun.source.tree.Tree.Kind;\r\nimport com.sun.source.tree.VariableTree;\r\nimport com.sun.source.util.TreeScanner;\r\nimport com.sun.source.util.Trees;\r\n\r\nimport javax.lang.model.element.Name;\r\nimport java.util.HashMap;\r\nimport java.util.List;\r\nimport java.util.Map;\r\n\r\n\/**\r\n * @author Andi Hotz, (c) Sahits GmbH, 2016\r\n *         Created on Aug 04, 2016\r\n *\/\r\npublic class ASTScanner extends TreeScanner&lt;Object, Trees&gt; {\r\n    private Map&lt;String, String&gt; imports = new HashMap&lt;&gt;();\r\n    private ProducerContext context = null;\r\n\r\n    private String eventBusName;\r\n    private final Multimap&lt;String, Class&lt;?&gt;&gt; eventBusPostEvent = ArrayListMultimap.create();\r\n\r\n\r\n    public ASTScanner(String eventBusName) {\r\n        this.eventBusName = eventBusName;\r\n    }\r\n\r\n    public Multimap&lt;String, Class&lt;?&gt;&gt; getEventBusPostEvent() {\r\n        return eventBusPostEvent;\r\n    }\r\n\r\n    @Override\r\n    public Object visitImport(ImportTree node, Trees trees) {\r\n        Tree qualifiedIdentifier = node.getQualifiedIdentifier();\r\n        String qualifiedName = qualifiedIdentifier.toString();\r\n        String classNameSubsting = qualifiedName.substring(qualifiedName.lastIndexOf('.') + 1);\r\n        imports.put(classNameSubsting, qualifiedName);\r\n        return super.visitImport(node, trees);\r\n    }\r\n\r\n    @Override\r\n    public Object visitExpressionStatement(ExpressionStatementTree node, Trees trees) {\r\n        if (context != null) {\r\n            if (node.getKind() == Kind.EXPRESSION_STATEMENT) {\r\n                ExpressionTree exprTree = node.getExpression();\r\n                if (exprTree.getKind() == Kind.METHOD_INVOCATION) {\r\n                    MethodInvocationTree methodInvocation = (MethodInvocationTree) exprTree;\r\n                    ExpressionTree methodSelect = methodInvocation.getMethodSelect();\r\n                    if (methodSelect.getKind() == Kind.MEMBER_SELECT) {\r\n                        MemberSelectTree memberSelect = (MemberSelectTree) methodSelect;\r\n                        ExpressionTree reference = memberSelect.getExpression();\r\n                        Name methodName = memberSelect.getIdentifier();\r\n                        if (methodName.contentEquals(\"post\") &amp;&amp; eventBusName != null) {\r\n                            List&lt;? extends ExpressionTree&gt; arguments = methodInvocation.getArguments();\r\n                            ExpressionTree firstExpr = arguments.get(0);\r\n                            if (firstExpr.getKind() == Kind.NEW_CLASS) {\r\n                                Class&lt;?&gt; eventType = getEventType((NewClassTree) firstExpr);\r\n                                eventBusPostEvent.put(eventBusName, eventType);\r\n                            }\r\n                            if (firstExpr.getKind() == Kind.IDENTIFIER) {\r\n                                Class&lt;?&gt; eventType = getEventType((IdentifierTree)firstExpr);\r\n                                eventBusPostEvent.put(eventBusName, eventType);\r\n                            }\r\n                        }\r\n                    }\r\n                }\r\n            }\r\n        }\r\n        Object result = super.visitExpressionStatement(node, trees);\r\n        return result;\r\n    }\r\n\r\n    private Class&lt;?&gt; getEventType(NewClassTree newExpr) {\r\n        ExpressionTree identifier = newExpr.getIdentifier();\r\n        Name name = ((IdentifierTree) identifier).getName();\r\n        return getClassByName(name);\r\n    }\r\n    private Class&lt;?&gt; getEventType(IdentifierTree identExpr) {\r\n        Name name = identExpr.getName();\r\n        return context.assignedCreatedObjects.get(name);\r\n    }\r\n\r\n    @Override\r\n    public Object visitVariable(VariableTree node, Trees trees) {\r\n        if (context != null) {\r\n            if (node.getInitializer() != null) {\r\n                if (node.getInitializer().getKind() == Kind.NEW_CLASS) {\r\n                    addVariableDeclaration(node);\r\n                } else if (node.getInitializer().getKind() == Kind.METHOD_INVOCATION) {\r\n                    addVariableDeclaration(node);\r\n                }\r\n            } else if (node.getKind() == Kind.VARIABLE) {\r\n                \/\/ only variable definition without assignment\r\n                addVariableDeclaration(node);\r\n            }\r\n        }\r\n        return super.visitVariable(node, trees);\r\n    }\r\n\r\n    private void addVariableDeclaration(VariableTree node) {\r\n        Name name = node.getName();\r\n        Tree type = node.getType();\r\n        if (node.getType() instanceof IdentifierTree) {\r\n            IdentifierTree idenfier = (IdentifierTree) type;\r\n            Name typeName = idenfier.getName();\r\n            try {\r\n                Class&lt;?&gt; clazz = getClassByName(typeName);\r\n                context.assignedCreatedObjects.put(name, clazz);\r\n            } catch (NullPointerException e) {\r\n                \/\/ Could not find class =&gt; not in the import statements so it cannot be a variable assignment we are interested in.\r\n            }\r\n        }\r\n    }\r\n\r\n    private Class&lt;?&gt; getClassByName(Name name) {\r\n        \/\/ Match to the imports\r\n        try {\r\n            String className = imports.get(name.toString());\r\n            return getClass().getClassLoader().loadClass(className);\r\n        } catch (ClassNotFoundException e) {\r\n            e.printStackTrace();\r\n            return null;\r\n        }\r\n    }\r\n    @Override\r\n    public Object visitMethod(MethodTree node, Trees trees) {\r\n        String methodName = node.getName().toString();\r\n        context = new ProducerContext(methodName);\r\n        Object result = super.visitMethod(node, trees);\r\n        context = null;\r\n        return result;\r\n    }\r\n\r\n    private static class ProducerContext {\r\n        Map&lt;Name, Class&lt;?&gt;&gt; assignedCreatedObjects = new HashMap&lt;&gt;();\r\n        String methodName;\r\n\r\n        public ProducerContext(String methodName) {\r\n            this.methodName = methodName;\r\n        }\r\n    }\r\n}\r\n<\/pre>\n<p>There are three visitXYZ methods that are overridden.<\/p>\n<p>The method <em>visitImport<\/em> takes care of all the imports and maps the\u00a0 simple class name to the fully qualified class name. This becomes important when matching <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/api\/javax\/lang\/model\/element\/Name.html\">Name<\/a>s representing classes to the actual class.<\/p>\n<p>The method <em>visitMethod<\/em> sets up the context upon which other visit methods operate.<\/p>\n<p>The method <em>visitExpressionSatement<\/em> is the one that does the actual matching. What we are looking for is a method invocation on an instance of the member field with the name &#8218;eventBusName&#8216;, that we have passed in as an argument to the constructor. On this field the method &#8218;post&#8216; must be called. So far we know that the class we are inspecting makes a post call on that EventBus. What we want to know in addition is which event type is posted. The most simple case from the perspective of the inspection is the direct creation of a new instance within the post method invocation. In that case we can deduce the type right there as the argument is a <a href=\"https:\/\/docs.oracle.com\/javase\/8\/docs\/jdk\/api\/javac\/tree\/com\/sun\/source\/tree\/NewClassTree.html\">NewClassExpression<\/a> from which we can learn the type. However most often the argument to the post method is an identifier referencing a local variable. To match the identifer to the type we have to keep track of the variable declarations.<\/p>\n<p>This is done with <em>visitVariable<\/em>. That visitor distinguishes between three different cases (though there are more):<\/p>\n<ul>\n<li>The variable is defined and instantiated with a NewClassExpression.<\/li>\n<li>The variable is defined and instantiated by a method invocation<\/li>\n<li>The variable is only defined while not initialized.<\/li>\n<\/ul>\n<p>Each of these cases provide the type of the variable and the name of the variable. The name is then matched in <em>visitExpressionStatement<\/em> against the argument.<\/p>\n<p>All this so far only gives us Names, what we actually require however are Classes. Names are basically a String literal. These are retrieved by matching the Names to their fully qualified names and from them load the class from the classloader. For exactly that purpose we went through the import statements. There are two cases that will fail with this approach and are not covered here:<\/p>\n<ol>\n<li>The Name represents a class in the package java.lang which is imported automatically<\/li>\n<li>The Name represents a class in the same package and therefore does not require an import statement.<\/li>\n<\/ol>\n<p>To bind all this together there is a main class:<\/p>\n<pre class=\"brush:java\">import com.google.common.collect.Multimap;\r\n\r\nimport javax.tools.DiagnosticCollector;\r\nimport javax.tools.JavaCompiler;\r\nimport javax.tools.JavaCompiler.CompilationTask;\r\nimport javax.tools.JavaFileObject;\r\nimport javax.tools.StandardJavaFileManager;\r\nimport javax.tools.StandardLocation;\r\nimport javax.tools.ToolProvider;\r\nimport java.io.File;\r\nimport java.io.IOException;\r\nimport java.util.Arrays;\r\nimport java.util.Collection;\r\n\r\n\r\npublic class ClassAnalyzer {\r\n\r\n    public static void main(String[] args) {\r\n        ASTScanner scanner = new ASTScanner(\"clientServerEventBus\");\r\n        ASTProcessor processor = new ASTProcessor(scanner);\r\n\r\n        final DiagnosticCollector&lt;JavaFileObject&gt; diagnostics = new DiagnosticCollector&lt;&gt;();\r\n        final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();\r\n\r\n        final File file = new File(\"src\/SampleClass.java\");\r\n        try (final StandardJavaFileManager manager =\r\n                     compiler.getStandardFileManager(diagnostics, null, null)) {\r\n            final Iterable&lt;? extends JavaFileObject&gt; sources = manager.getJavaFileObjectsFromFiles(Arrays.asList(file));\r\n            String outputDir = System.getProperty(\"java.io.tmpdir\");\r\n            manager.setLocation(StandardLocation.CLASS_OUTPUT, Arrays.asList(new File(outputDir)));\r\n            final CompilationTask task = compiler.getTask(null, manager, diagnostics, null, null, sources);\r\n            task.setProcessors(Arrays.asList(processor));\r\n            task.call();\r\n\r\n        } catch (IOException e) {\r\n            e.printStackTrace();\r\n        }\r\n        Multimap&lt;String, Class&lt;?&gt;&gt; postedEventsOnEventBus = scanner.getEventBusPostEvent();\r\n        for (String eventBus : postedEventsOnEventBus.keySet()) {\r\n            System.out.println(\"On EventBus '\"+eventBus+\"' the following event types are posted:\");\r\n            Collection&lt;Class&lt;?&gt;&gt; col = postedEventsOnEventBus.get(eventBus);\r\n            for (Class&lt;?&gt; clazz : col) {\r\n                System.out.println(\"\\t- \" + clazz.getName());\r\n            }\r\n        }\r\n    }\r\n}\r\n<\/pre>\n<p>This class will generate the following output:<\/p>\n<pre class=\"brush:plain\">On EventBus 'clientServerEventBus' the following event types are posted:\r\n\t- javafx.beans.property.SimpleLongProperty\r\n\t- javafx.beans.property.DoubleProperty\r\n\t- javafx.beans.property.IntegerProperty<\/pre>\n<p>This is exactly what we expected.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Documentation of code has a tendency to become obsolete quickly. For that reason most often the only documentation is the code itself and if you are lucky some JavaDoc that is more or less up to date. For that reason I already started some time ago to generate additional documentation by doing static code analysis &hellip; <a href=\"http:\/\/sahits.ch\/blog\/blog\/2016\/08\/06\/documentation-generation-with-the-compiler-api\/\" class=\"more-link\"><span class=\"screen-reader-text\">\u201eDocumentation generation with the Compiler API\u201c <\/span>weiterlesen<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[307,308,301],"class_list":["post-2531","post","type-post","status-publish","format-standard","hentry","category-java","tag-compiler-api","tag-documentation","tag-java"],"_links":{"self":[{"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/posts\/2531","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/comments?post=2531"}],"version-history":[{"count":4,"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/posts\/2531\/revisions"}],"predecessor-version":[{"id":2535,"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/posts\/2531\/revisions\/2535"}],"wp:attachment":[{"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/media?parent=2531"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/categories?post=2531"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/sahits.ch\/blog\/wp-json\/wp\/v2\/tags?post=2531"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}