days
-4
-3
hours
-1
-4
minutes
-5
-2
seconds
-1
-2
search
Innovative language features for Java - Part 1

Manifold: The reinvention of code generators

Scott McKinney
manifold
© Shutterstock / gonin

This is the first in a series of articles covering Manifold, a unique, open source technology you can use in any Java project to enable cutting edge language features such as type-safe metaprogramming, extension methods, templating, and structural typing. In this segment, Scott McKinney discusses Manifold’s reinvention of code generators, namely Type Manifolds.

☠ Death to Code Generators ☠

While Manifold provides a broad host of high-level features, its primary focus is to eliminate the gap separating source code from metadata. For decades code generators have served to bridge this gap, but not without sometimes debilitating drawbacks. If a project you develop involves one or more code generators, perhaps you know what I mean.  Read on for a more productive alternative.

The Metadata Disconnect

Our modern lives are replete with structured information, or metadata.  It is everywhere and it is produced by near everything with a power cord. As a consequence the software industry has become much less code-centric and much more information-centric. Despite this transformation the means by which our software consumes metadata has remained virtually unchanged for half a century. Whether it’s JSON, XSD/XML, WSDL, CSV, DDL, SQL, JavaScript, XLS, or any one of a multitude of other metadata sources, most modern languages, including Java, do very little to connect them with your code:

../abc/Person.json

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string"
    },
    "lastName": {
      "type": "string"
    },
    "age": {
      "type": "integer",
      "minimum": 0
    }
  },
  "required": ["firstName", "lastName"]
}

As Java developers we want to use metadata in a type-safe manner — we want to use the Person JSON schema file as a Person Java class:

class Foo {
  private abc.Person person; // ERROR: Cannot resolve symbol 'abc.Person'
} 

SEE ALSO: Manifold: Alien technology

Of course the Java compiler has no idea what to do with abc.Person, so we resort to running a code generator in a separate build step to generate all our JSON classes beforehand so the compiler can readily use them. The effects of this build step on the development lifecycle range from mild irritation to utter devastation, depending on the rate of metadata change, number and size of metadata files, density of usage in project, number of developers, etc. The problems include:

  • stale generated classes
  • long build times
  • code bloat esp. with large metadata domain
  • changes to structured data don’t invalidate generated code
  • no support for incremental compilation, all or nothing
  • can’t navigate from code reference to corresponding element in the structured data
  • can’t find code usages of elements from the structured data
  • can’t refactor / rename structured data elements
  • complicated custom class loader issues, generated classes loaded in separate loader
  • concurrency problems with the shared thread context loader
  • generated code is often cached and shared, which leads to stale cache issues
  • customers often need to change metadata, which requires access to code generators

The Manifold Framework

The Manifold framework represents a rethinking of code generators. It altogether avoids the many disadvantages often involved with them by directly integrating with the Java compiler via the javac plug-in mechanism.

Implementors of the Type Manifold API, called type manifolds, establish a type supplier relationship with the Java compiler — the Manifold framework hooks into the compiler so that as the compiler encounters type names the type manifolds contribute toward resolving them, generating code in memory as needed. As such your application code can reference metadata sources directly by name as Java types, effectively enabling the prior example to work:

class Foo {
  private abc.Person person; // OK <img draggable="false" class="emoji" alt="🙂" src="https://s.w.org/images/core/emoji/2.3/svg/1f642.svg">
}

Think of a type manifold as a new domain of types for the compiler to access. As such the Manifold framework serves as a gateway between javac and type manifolds, effectively expanding Java’s type system to include whole new domains of types. Any number of type manifolds can operate in concert; they can also cooperate so that the types contributed from one can feed into the next and so on, forming a type building pipeline.

manifold

As the diagram illustrates a type manifold contributes to the definition of types in its domain. For example, the JSON type manifold produces types defined in JSON files. A type manifold can contribute toward a type in different ways. Most often a type manifold registers as a primary contributor — it supplies the main body of the type. The JSON type manifold is a primary contributor because it supplies the full type definition according to a JSON Schema file or JSON sample file.

SEE ALSO: Code generation with Xtend

Alternatively, a type manifold can be a partial or supplementary contributor. The extension type manifold, for instance, is a supplementary contributor because it augments an existing type with methods, interfaces, and other features. Thus both the JSON and Extension type manifolds can contribute to the same type, where the JSON manifold supplies the main body of the type and the Extension type manifold contributes methods and other features provided by extension classes. (I’ll cover extensions in a later article.)

Altogether this strategy eliminates many problems plaguing conventional code generation and metadata access in general. In essence the Type Manifold API redefines what it means to be a code generator. Benefits include:

  • Zero turnaround – live, type-safe access to metadata; make, discover, and use changes instantly
  • Lightweight – direct integration with standard Java, requires no special compilers, annotation processors, class loaders, or runtime agents
  • Efficient, dynamic – Manifold only produces types as they are needed by the compiler
  • Simple, open API – you can build your own type manifolds
  • No code generation build step – eliminates code generators from your development build process
  • IntelliJ IDEA – comprehensive IDE support: code completion, navigation, usage searching, refactoring, debugging, etc.

Further, the Type Manifold API unifies code generation architecture by providing much needed structure and consistency for developers writing code generators. It puts an end to “lone wolf” code generator projects only one developer fully understands. Moreover, you don’t have to invest in one-off IDE integration projects; the Manifold plugin for IntelliJ handles everything for you, from incremental compilation to usage searching to refactoring. Finally, even if you’ve already invested in an existing code generator, you can still recycle it as a wrapped type manifold — the wrapper can delegate source production to your existing framework. Learn more about implementing type manifolds here.

Synergy

Perhaps the most refreshing benefit from using Manifold is the synergy resulting from its presence. With Manifold, a developer can define and use metadata that best suits the needs of a project without having to worry about build implications or IDE integration; he can create a metadata file, use it directly as a type, modify it, and access the changes immediately in his code. No compilation necessary, no build steps to invoke. With comprehensive IDE support, he can readily navigate to and from metadata elements, find usages from metadata, refactor, etc. Finally metadata has first-class representation in the Java development lifecycle! View it in action.

SEE ALSO: Gosu — Hey look! It’s a pragmatic language for the JVM. A simple language

Using Manifold

Setup

Using Manifold in your Java project is easy:

  • Add the Manifold jar[s] to your classpath (and tools.jar if you’re using Java 8)
  • Add -Xplugin:Manifold as an argument to javac (for compilation only)

That’s all.

Manifold fully supports Java 8, Java 9, and Java 10.

Manifold works well with Maven and Gradle too. Learn more about adding Manifold to your project here.

Working with IntelliJ

Manifold is best experienced using IntelliJ IDEA.

Install

Get the Manifold plugin for IntelliJ IDEA directly from IntelliJ via:

Settings | Plugins | Browse Repositories | Manifold

manifold

Creating a new project with Manifold support is also very easy; there’s a demo video available here. Alternatively, Manifold can also be added to modules of an existing project:

manifold

Anyone who wants to experiment with Manifold should take a look at the sample project here.

SEE ALSO: Fresh out of the oven: Torus is a Docker-based toolkit for machine learning projects

New Project

Creating a new project with Manifold support is easy. Check it out.

Add Manifold to Existing Module

You can add manifold to module[s] of an existing project too. Check it out.

Sample Project

Experiment with the Manifold Sample Project.

SEE ALSO: “Good code generators will be the most helpful and useful tools for coding by 2040”

Conclusion

As a long time Java developer I’ve personally worked on several projects involving heavy code generation. I’ve seen the sometimes devastating effects of its use: build times measured in hours at customer sites, dev lifecycle demoralization, code generator development and maintenance consuming precious time, etc. It’s about time for a better solution and I think Manifold makes good progress toward that goal. Type Manifolds offer an inviting and productive development experience. With no code generators to invoke and no separate build steps to integrate, metadata just works.

There’s much more to cover, I’ve only scratched the surface with the Type Manifold API. Future articles in this series will cover:

As a bonus for reading this far, I’ll touch on one of Manifold’s latest features…

A String template lets you use the $ character to embed a Java expression directly into a String. You can use $ to embed a simple variable:

int hour = 8;
String time = "It is $hour o'clock";  // prints "It is 8 o'clock" 

Or you can embed an expression of any complexity in curly braces:

LocalTime localTime = LocalTime.now();
String ltime = "It is ${localTime.getHour()}:${localTime.getMinute()}"; // prints "It is 8:39"

Learn more here.

 

    DevOpsCon Whitepaper 2018

    Free: BRAND NEW DevOps Whitepaper 2018

    Learn about Containers,Continuous Delivery, DevOps Culture, Cloud Platforms & Security with articles by experts like Michiel Rook, Christoph Engelbert, Scott Sanders and many more.

manifold

Author

Scott McKinney

Scott McKinney is the founder and principle engineer at Manifold Systems. Previously, he was a staff engineer at Guidewire where he designed and created Gosu. He currently pounds code by the truckload while listening to way too much retro synthwave.


Leave a Reply

Be the First to Comment!

avatar
400
  Subscribe  
Notify of