The story of OSGi enRoute

Introducing enRoute 1.0 – the new OSGi Framework | Part 2

Peter Kriens
@enroute.osgi.org

In the first part of this article, enRoute inventor and OSGi mastermind Peter Kriens explained the role of OSGi enRoute. In the second part, he will offer further clarifications regarding version 1.0 of OSGi enRoute framework.

OSGi enRoute’s goal is to make it easy to get started with OSGi using best practices. It is an API, a toolchain (bnd, Bndtools, Gradle, Git,and Travis), and a distro based on open source bundles to quickly build OSGi (web) applications for different target environments. It is an excellent environment to learn how OSGi is intended to be used. The best way to get your hands dirty is the OSGi enRoute Quick Start tutorial.

OSGi enRoute Base API

OSGi enRoute provides a JAR which defines a minimal environment to develop simple Web Applications, potentially even on IoT devices. It can of course be the base for larger web applications since it is a component system. It requires Java 8, OSGi R6, and several service contracts from the OSGi R6 release as well as services developed specifically for OSGi enRoute.

SEE ALSO: Introducing enRoute 1.0 – the new OSGi Framework

All service contracts in the OSGi enRoute Base API are explained in the OSGi enRoute service catalog. This catalog is intended to be the starting point when you encounter a new service.

Though the OSGi specifications are very thorough, they do focus on the implementer of a specification; the service catalog focuses instead on the user of a specification. The OSGi specifications can sometimes be too formal and detailed if you just want to use it. The intention is that each entry in the service catalog will have an example application and some snippets to copy. That said, the service catalog still requires some additional work.

We took the following services from the OSGi R6 Compendium in alphabetical order:

table1

We then added a number of new services that were crucial for web based applications but were not part of the OSGi compendium yet. Most of these services have been submitted as RFPs to standardize them via the OSGi specification process.

Screen Shot 2016-03-21 at 4.35.58 PM
The OSGi enRoute Base API also contains a number of annotations that require the web resources needed for web applications. These annotations create the proper requirements that are then encoded in the bundle’s manifest.

table2
In certain cases OSGi got so much into POJOs that the assembly phase was not aware of any dependencies and did not include a component that was still relevant. A good example is the Declarative Services runtime. Most components do not have any coupling to the component API. In this case, the tooling (bnd) automatically adds a requirement to the Declarative Services when the component annotations are used. However, in other cases it is required to explicitly require a subsystem. OSGi enRoute provides the following subsystem annotations:

Screen Shot 2016-03-21 at 4.35.27 PM
OSGi enRoute Base Distro

I hope it is abundantly clear now that the OSGi enRoute Base API is specification only. Some OSGi members made it clear that they did not want a ‘blessed’ runtime from OSGi, they wanted to provide an application server with their own runtime. This largely drove the model to where there is a base API used to create bundles that can then be resolved against a distro into an application. Clearly, this model allows many different distros; we actually hope that popular environments like Karaf and other OSGi aware application servers and products will provide a distro out of the box.

However, we also want the user to be able to download the tooling, create a workspace, and debug an app without having to chase the parts. This obviously requires a default distro. We therefore include a base distro of open source components. This distro satisfies all of the requirements from the OSGi enRoute Base API but certain parts might not be industrialized.

Tooling

The OSGi enRoute tooling is build around bnd. This is a small library that has a lot of knowledge about OSGi and is excellent at creating working bundles. (Or otherwise telling you what is wrong.) As its input, it takes bnd files, which are actually just property files. These properties describe the desired bundle and then bnd assembles a JAR file with a valid OSGi manifest according to this specification. bnd goes out of its way to simplify the developer’s life.

For example, it calculates the imports from the references in the Java byte codes. It has special annotations that can generate requirements or capabilities in the manifest; these annotations can even be used indirectly on other annotations. (These annotations will likely be specified by the OSGI Alliance in R7.) And it has generators to generate Declarative Services and Metatype metadata according to R6 specifications.

External dependencies in bnd are managed by pluggable repositories. This allows bnd builds to integrate with maven repositories or other types, like for example jpm4j.org. Repositories can provide the requirements and capabilities of remote artifacts. A bnd workspace usually has a number of repositories. Repositories often reflect an aggregate of artifacts. For example, Bndtools uses a repository on S3 with all the Eclipse bundles it is built against.

In the release process, the artifacts are written to the release repository; repositories can be read and written. This is a flexible model that allows the build to adapt to the myriad of release models that exist.

A significant amount of the work to develop OSGi enRoute went into enhancing bnd and Bndtools.

bnd has no user interface by itself, it must therefore live in an IDE or a build tool. In the case of OSGi enRoute we use Eclipse with Bndtools for the IDE and Gradle for the continuous integration and command line aficionados.

Yes, no maven. See the future section if you want/must hang on to maven (which does have a bnd plugin). For Intellij, they are also integrating bnd but results may vary since they are doing this integration (unfortunately) in stealth mode. However, we explicitly picked one toolchain to keep it as simple as possible to get started with OSGi.

Bndtools

I like to think that Bndtools sets a standard for Java development. The coding phase is very well supported with lots of OSGi-specific errors reported on the exact place they are made. The best part is, however, the edit/debug cycle; it is extremely short. And it achieves this speed without cutting any corners. Bndtools generates the real bundles and uses the dynamic nature of OSGi to update these bundles. Using OSGi instead of something like JRebel allows us to debug the same way on your local machine as a cloud based application server or even a Raspberry Pi.

Obviously we inherited all the goodies from the unsurpassed Eclipse Java Development Tool. Refactoring, incremental compiles, templates, navigation, etc. Lots of features for OSGi development have been integrated with proper shortcuts, quick fixes, and reporting.

OSGi enRoute added a project template for typical OSGi project setups. The template reacts to the project name by nudging the use of consistent names. Since naming is important, it ensures that all template package names and class names are derived from the project name. The following types are supported:

  • *.application – A web based application project. Out of the box you get a REST server and an Angular GUI that uses the REST service.
  • *.api – Defines a service contract.
  • *.provider, *.adapter – For implementation bundles.
  • *.webresource – Web resources.
  • *.example – API, provider, and Gogo shell commands bundles.
  • *.test – An OSGi test project.

One of the unique features we added is support for versioning because humans are notoriously bad with versions. Although bnd goes out of its way to simplify and automate versions, it is all too easy to violate the semantic versioning rules. To prevent these errors from reaching the runtime, bnd verifies the versions against the latest released artifact using the Java rules of binary compatibility and OSGi semantic versioning. If it finds an inconsistency, it will report this at the place where the version is defined and where the Java culprit source code resides.

For example, when you add a method foo() to an exported interface, you will clearly break all implementers of that interface. Such a change requires a major alteration in the package’s version. Therefore bnd will mark the foo() method with a shortcut to do a major bump on the version. It is kind of scary how often one runs into this error.

External dependencies are an inevitable part of the developer’s life. Bndtools includes a view on the bnd repositories to manage these external dependencies. We provide a Jar editor that can view the repository entries in detail directly from the repository. The repositories can be searched for text patterns or requirements for packages, services, or even general. Bundles can be directly dropped on a repository to add them. OSGi enRoute also includes the JPM repository for external dependencies. This JPM repository provides all Maven Central and more artifacts. From Bndtools you can search this repository and you can use drag and drop to add these artifacts to the build path.

As we discussed, the capability model is an intrinsic part of OSGi enRoute. One of the unique aspects of Bndtools is that it has a resolver tool to work with these capabilities to assemble an executable. For this, you can specify a bndrun file (also a properties file) describing the desired runtime environment. The bndrun file only specifies the initial requirements. The Bndtools resolver will then find a set of bundles from the repositories, ensuring that this set is self consistent. This means that each bundle in the set has all its requirements satisfied by bundles in the set or the target runtime environment (e.g. the VM and framework or application server).

A launcher plugin then uses this specification to launch the application. First this is done in the Eclipse debugger to test if all the parts are collaborating as expected. Any updates to the bundle are immediately deployed to the target.

I do a lot of javascript lately that gets served from a bundle. The workflow is really quite nice. I save the Javascript, switch to the browser, refresh with F5 and see what happens. The back-end REST or JSON RPC server is part of the same build so you have a complete debug environment for the back-end and front-end of your code. No artificial boundaries.

Debugging is also simplified by providing a special debug environment. The OSGi enRoute templates provide two bndrun files for this. The first bndrun file is the target application and the second is intended for debugging. The second bndrun file inherits all the settings from the target bndrun but then adds a set of standard debug bundles. This then provides the runtime with Gogo shell, Webconsole, Xray, and other goodies that make the life of a debugger less gruesome.

The bnd launcher is pluggable; two launchers are currently provided out of the box. The default launcher launches the application on the local machine. The second launcher launches remotely using another machine that is reachable on the network. This machine should run a special command (bndremote) or have a framework with an agent bundle. With the bndremote command you can specify the framework, class path JARs, and framework properties. The agent already runs in a configured framework so those aspects are already decided. A local cache is used for the bundles which makes it surprisingly fast, I often find that I was debugging remote while thinking it was local.

The bndrun file and its features are also used to define JUnit testing. Testing can be done without a framework (normal JUnit) and it can be done inside an OSGi framework. It is possible to set this up so that every change in the IDE is immediately tested.

Once the application is good enough, bnd can export the bndrun file to an executable format. Since the bndrun file is closely associated with the launcher, it is this plugin that generates the executable bits. Currently we support executable JARs but we will soon support a model to export to common Java execution formats like OSGi subsystem, Karaf KARs, WARs,Docker images, and others.

Gradle

IDEs are good for interactive development but command line tools are better when you can automate a task. Gradle is an excellent command line build tool based on the newest insights in build tools and it uses Groovy (which is a superset of Java) to extend it.

In Bndtools bnd manages the JAR build and we use the same library in Gradle, The bnd Gradle plugin setups Gradle to create identical artifacts as those that are created in Bndtools. All the functions, and then some, that are available from the IDE are also available in Gradle. This makes it very easy to automate repeating tasks using the same settings as the IDE uses.

Github & Travis

In OSGI enRoute we standardized on Github and Travis for the source control and continuous integration servers.

Github because, well you know. Github is quite a lot of bang for no buck. (At least if you’re in an open source project.) The workspace template for OSGi enRoute is maintained on Github as are all workspaces for the APIs, bundles, project templates, and examples.

Travis because it is so well integrated with Github. I set up my first continuous integration server coupled to CVS in 1994. Since then, I’ve not worked on a single project without one, it is probably the greatest bang for the project’s buck. Fortunately, since 1994 it has become a lot easier. Easier than with Travis (when your project is on Github) is hard to envision.

How To Get Started

That was a long description. I hope you now find it worthwhile to get your hands dirty and try this all out. The enroute.osgi.org website contains a number of tutorials to get you going:

  • The quick start tutorial is the fastest way to get a project up and running. It shows you how to create a workspace and a web application project.
  • The base tutorial is a more extensive variation of the quick start. It shows how to setup a workspace properly with API projects,providers, test, and applications. it contains a lot of practical guidance.
  • The IoT tutorial shows how to use a Raspberry Pi for OSGi enRoute development. It shows remote debugging and deployment as well how the IoT API in the OSGi enRoute Base API works.

Future

We released OSGi enRoute 1.0 in September 2015. Though you can build simple web applications there are some glaring omissions — the greatest is the absence of a persistence API. Though JPA is the logical candidate, some tests showed that it was very hard to switch providers because the specification for JPA is quite weak in many areas. There are also a few glaring omissions in the OSGi specification to make JPA work really well in OSGi. Since there is no obvious alternative we decided to postpone persistence support in enRoute. However, this has the highest priority. Recent work in Apache Aries and Karaf could provide a proper solution in the near future.

The other omission is the lack of maven integration. Maven has really become the primary build tool in the Java world. Though there are good technical arguments to use Gradle, unfortunately there are no longer such good arguments if you want to lower the threshold for people to start with OSGi. Since we started with OSGi enRoute, m2eclipse has become a lot more usable, addressing some of the concerns we had with maven.

In the bnd team a two pronged plan has been laid out. bnd(tools) users will get seamless access to maven repositories and will be able to share the local maven repository with maven builds. This will support all repository features for dependencies as well as releasing that maven supports. Maven users (m2eclipse) will get plugins that integrate the bndrun features in maven. This will bring the resolver in reach of maven users. With these plugins it will be possible to create capability repositories purely for maven and use the Bndtools launchers and resolver to create and debug OSGi runtime environments.

Further plans are to focus on IoT. Obviously there is a lot of action in IoT but it is also clear that OSGi for IoT is more than a perfect union. OSGi was made for the embedded world and it shows; combine this with the use of OSGi in enterprise and you have a match made in heaven. There are a number of activities inside the OSGi Alliance around IoT including the recently formed IoT Expert Group that you can get involved with. In OSGi enRoute we want to provide an easy playground for these initiatives.

Last but not least, we want to work with the myriad of OSGi open source communities. The current distro already contains bundles from each of the open source communities, a proof of how interoperable OSGi is. We will finish the OSGi enRoute support for distros on application servers. This requires exporters. We plan a Karaf export because this appears to be one of the popular OSGi open source distributions but we’d like to support OSGi subsystems as any other format out there. We would also like to share the service catalog and tutorials. We also want to work with commercial products to make it easy for them to run OSGi enRoute applications and share the education and tooling efforts.

Participate

OSGi enRoute is run by the OSGi Alliance as an open source project, unlike the OSGi specification process which is a member only activity for contribution. Though the current Github source repositories contain non-standardized service contracts and special implementation bundles, the intention is to remove those once they can be obtained from one of the open source projects that provide OSGi bundles. We’re not interested in becoming the dominant OSGi distro in the world. OSGi enRoute was developed to create a beacon of best practices with OSGi. A beacon that is not encumbered by backward compatibility constraints that are necessary for any project but too often make it impossible to fulfill the rich promises that OSGi offers. That is the task put on the open source communities as well as commercial providers.

Our primary goal is to make it easier to get started with OSGi, which should be in the interest of all open source organizations, commercial entities, and the developers that are involved with OSGi. To reach this goal, we are open to any input. Since OSGi enRoute is not a specification we have a lot of freedom to use it as incubator for specifications. We hope that Eclipse, Karaf, Felix, and others will provide distros for their environments so that the applications are easily transportable between these environments. We hope that the special OSGi enRoute services can be replaced by standardized OSGi services and that the bundles can be replaced by open source implementations.

All sources are on Github, including the web site. We welcome feedback through issues or Pull Requests.

Acknowledgements

OSGi enRoute was made possible through the work of many individuals. First, we thank the OSGi Alliance board for making this possible, and especially the president Dan Bandera (IBM) and Susan Schwarze (ProSyst) who have been instrumental to enable this project.

OSGi enRoute stands on the shoulders of several different great products. Neil Bartlett (Paremus) has been the primary author of Bndtools and is largely responsible for the user friendliness. BJ Hargrave (IBM) strengthened the bnd(tools) development process by adopting the bnd Gradle plugin and make us eat our own dog food at all cost. Tim Ward (Paremus), David Jencks (IBM), Marcel Offermans (Amdatu), Paul Bakker (Amdatu), Ray Auge (Liferay), Greg Amerson (Liferay), PK Sörelde, and David Bosschaert (Adobe) are committers on bnd and Bndtools and thus contributed as did countless others that took the effort to provide a PR when they discovered a bug or wanted a feature. And let’s not forget all the open source projects out there that provided those wonderful OSGi bundles on which the enRoute distro stands. Thanks.

Author
Peter Kriens
Peter Kriens is an independent consultant since 1990.He currently works for the OSGi Allianceand jpm4j. During the eighties he developed advanced distributed systems for newspapers based on microcomputers based on, at the time very novel, object oriented technologies. For this experience in Objects he was hired by a number of international companies, including Adobe, Intel, Ericsson, IBM, and many others. During his work at Ericsson Research in 1998 he got involved with the OSGi specification; Later he became the primary editor for these specifications. In 2005 he was awarded the OSGi Fellows title. After taking a sabbatical in 2012 to develop jpm4j he returned to the OSGi Alliance to help increasing adoption. He is Dutch but decided to live in France.

Comments
comments powered by Disqus