days
1
3
hours
1
4
minutes
3
8
seconds
2
5
search
A NetBeans world

Particle Accelerating NetBeans: Docker and ROOT

Thomas Kruse, Karsten Sitterberg
root
Wooden toolbox on the table image via Shuterstock

This article illustrates the possibilities using docker in conjunction with NetBeans to aid the development of C++ applications and is intended to stimulate readers’ creativity for their own experiments.

Docker deserves its space in every developer’s toolbox. Those who want to know more about docker should study the docker documentation and consider attending a docker training.

All about Speed

Setting up and using C++-Frameworks can be cumbersome, especially when dealing with parallel and maybe conflicting installations of libraries, header files and sources. In this article, the CERN ROOT framework will be set up for the development of a small project, introducing techniques to avoid the above-mentioned pitfalls. If testing and support are needed not only for different versions, but also different environments – like Ubuntu Linux, Scientific Linux or Fedora – the resulting matrix of systems necessary to test will be huge. Besides using physically different machines, one approach for facing these issues has been the virtualization of machines. In particular Oracle’s Virtual Box combined with HashiCorp’s Vagrant have proven to be a real boost for productivity.

Virtualization made it possible to run different isolated environments at the same time even on a laptop. They may even be used to create well-defined states via snapshots or configuration management (e.g. Puppet).

If the development environment of choice provides good support for remote builds, a nearly perfect development process can be reached. However, speed and efficiency when using the operating systems’ resources (i.e. RAM, CPU etc.) are somewhat limited on a virtual machine. Therefore, something more lightweight would be desirable. This criterion is perfectly met with containers.

Docker Container

A container differs from a full virtualization in the sense that a container starts as a lightweight operating-system process. It is then isolated from the rest of the system in a way that can be best described as an extended chroot. However, the higher flexibility in comparison chroot and a better performance than a virtual machine come at a price: lower system-isolation and the necessity to use a Linux-based system as environment. OS X and Windows users can still use Docker, but within the container a Linux environment has to be used. Invisible to the user, VirtualBox and Vagrant are used as Docker runtime environment on Windows or OS X.

ROOT

ROOT is a software framework based on the programming language C[]+ and was developed at the CERN Facility for data-analysis purposes in high-energy physics experiments. It is a good choice for adoption in the big data sector because of its abilities in saving, analyzing and displaying the data. The User can leverage the features of ROOT either in the form of ROOT-scripts, which are executed in the Cling C+ interpreter, or in the form of compiled programs. Furthermore, ROOT is able to integrate with other languages like Python or R.

Set-Up

In the following section, the set-up of the docker environment and the CERN ROOT Docker container as an example for remote C++ development with help of the NetBeans IDE is described. For this project, a new folder in should be created in the file system. Into this folder, the sample project-archive you can find in the appendix of this article can be extracted. Also from this folder, all command line instructions within this article are executed. (As an alternate — and especially for Windows users — a fixed path can be used instead of the pwd command.)

Docker

Setting up Docker on Linux is as simple as installing it via the package manager of choice. User of Mac and Windows simply install the Docker toolbox from https://www.docker.com/toolbox

With Ubuntu, the docker.io package has to be installed. One has to give oneself the access rights to interact with Docker, then. Caution: Every Person within the docker-group can give oneself root access to the system via Docker. (So much for reduced isolation, but each developer should have root access to his workstation anyways).

sudo apt-get install docker.io
sudo usermod -a -G docker <username>

It is best to reboot the system after this, so the docker service will be started and the access rights will be properly applied.

ROOT Docker Image

Docker uses Images as basis for the run time instances, which are called Container. After an image is built, it will not be changed. Therefore, images are ideally suited for creating reproduce able or even identical environments for development and testing.

To create an image, a ‘recipe’ for the image is written into a Dockerfile. Docker uses a layered file system, so some kind of inheritance on image-level is possible. This inheritance is realized in that each docker image can have a base-image, which is declared in the dockerfile via FROM.

FROM gcc

To keep the process of generating the docker images maintainable, it is recommended to work with placeholders, e.g. for concrete version numbers or for artifacts to be installed. With Docker, parameters can be exported and then be used in all commands during image creation. In our example, the used version of the ROOT framework and the download URL are exported into such parameters:

ENV USER_DIR      /home/user
ENV ROOT_VERSION  v6.04.02
ENV DOWNLOAD_URL  https://root.cern.ch/download/root_${ROOT_VERSION}.source.tar.gz

This parameters then can be used throughout the actual installation, as can be seen below. Also it should emphasized that the operating system’s environment can be employed within the container: For example the number of CPU cores can be obtained with the well-known command nproc. The number of cores can be used for example to set the number of compiler threads.

RUN mkdir root-cern \
  && echo "download ${DOWNLOAD_URL}" \
  && curl -L --silent "${DOWNLOAD_URL}" | tar -xz --strip=1 -C "${USER_DIR}/root-cern" \
  && cd "${USER_DIR}/root-cern/build" \
  && cmake ../ \
  && cmake --build . -- -j$(($(nproc) + 1)) \
  && cmake --build . --target install \
  && ldconfig
Figure 1. CPU diagram showing CPU usage during a multi threaded build

Figure 1. CPU diagram showing CPU usage during a multi threaded build

For SSH — used by NetBeans for remote-access — being successful, the container has to be configured to start the SSH deamon on container start. Additionally the TCP port 22 has to be declared to the outer environment.

EXPOSE 22

Building the docker images is initiated via docker build. Also the image can be given a meaningful name and the execution directory can be specified, for example like this:

docker build -t cern-root .

The container utilizes the official gcc-docker base image. If this is not already locally available, for example when starting it the first time, the base image will b fetched from a central repository, called the Docker Hub. Subsequently the sources for ROOT, about 100 MB, are loaded from CERN and compiled.

On a notebook with a weak CPU and four cores, the complete build takes about one hour. On a desktop computer with 16 threads the build takes about 15 minutes.

Within the image, the CERN ROOT framework is installed to /usr/local.

Netbeans

Even though there already are NetBeans docker-images, the best way for obtaining a smooth and stable NetBeans is to load the right version from the official website, https://www.netbeans.org/. For the C[]+ version of NetBeans, a separate Java-installation is not needed because of the Netbeans IDE shipping its own installation since NetBeans version 8.1. If not only C+ should be developed, but for example C++ in conjunction with Java, a current version of the JDK and a corresponding edition of the NetBeans IDE have to be installed.

Test Run

This step is optional, but does shed some light onto what happens under the hood. After generating the docker image, it serves as a template for one or more instances derived from this image. During normal operation, a container (instance) is generated from the image. This container can be started and stopped or removed if the instance is no longer needed. For the test run this steps are merged into one and additionally, an interactive shell is started to be able to act within the container.

docker run --name run_root -ti --rm cern-root /bin/bash

From within the instance, everything can be handled like usual. As a test, one can assure oneself that GCC is installed and that its version can be printed to the command prompt. To exit the session, the exit command works like in every normal shell. In this particular case, the container is stopped after this. The container doesn’t have to be deleted due to the parameter --rm, which indicates that the container is deleted after each run automatically by docker. Without this parameter or without manually deleting the docker container, a new instance would remain after each execution with own and isolated files and file-changes. All saved changes are deleted along with the instance, which is a consequence of the isolation. But what if there are files which should not be deleted, or not be isolated at all? For this, docker introduces the concept of Volumes. These volumes exist in parts of the memory which are independent from the life cycle of the containers. With this, Volumes can be used even between multiple containers at the same time. In the dockerfile, the path/home/user/data/sample was declared to contain such a persistent memory area:

VOLUME /home/user/data/sample

Volumes allow to connect a container with its surrounding environment so that files generated within the container can be made transparently visible to the outside world. Configuration of this feature can be accomplished via the parameter -v, which interconnects the local directory to the path within the container. (For Mac and Windows systems, this is not as easy to obtain but can be done also. At this point, we refer to the docker documentation for these platforms.)
If the example project was prepared within a folder, the container should be started also from this folder. This allows for connecting the sample directory to the container without usage of an absolute path. It is instead connected via a relative path, which is combined with a determination of the current directory.

Just like filesystem paths, network ports can be connected with help of the parameter -p. Along with this parameter, the local and corresponding container ports have to be provided.

Following is the command line for creating an appropriately configured container named run_root.

docker create --name run_root -v `pwd`/sample:/home/user/data/sample -p 1022:22  cern-root

The corresponding instance can now be started this way:

docker start run_root

Because the SSH service is started as specified in the dockerfile, and the container SSSH port 22 is mapped to the local port 1022, a login to the container can be performed via SSH. For testing purposes this can be done form the command prompt – the password being netbeans.

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 1022 user@localhost

If you want to stop the container, you can use the stop command:

docker stop run_root

A stopped container can be deleted together with the persistent volume via rm -v:

docker rm -v run_root

If the image should be deleted this can also be done subsequently:

docker rmi cern-root

After these preparations, the NetBeans IDE can be configured to connect via SSH to the running container instance.

Integration

First, the container is – if not already done in the previous steps – created and started. It is available for use per SSH afterwards.

docker create --name run_root -v `pwd`/sample:/home/user/data/sample -p 1022:22  cern-root
docker start run_root

Usage with Netbeans

To use the docker environment with NetBeans, a new C[]+ build host has to configured in the first place. This host allows NetBeans to access the ROOT environment of the docker container. The build hosts themselves can be configured in the NetBeans menu “Services”. Here, a new entry can be added by right-clicking “C+ Build Hosts”:

Figure 2. Adding a new C/C++ build host in the NetBeans-"Services"

Figure 2. Adding a new C/C++ build host in the NetBeans-“Services”

For the set-up of the new hosts it is necessary to provide a host name and the port under which the SSH service is reachable. Netbeans is looking for reachable hosts automatically, but doesn’t include ports other the port 22. Therefore the docker host has to be configured as localhost with the port number 1022, which was introduced above.

Figure 3. Set-up of name und port of the host

Figure 3. Set-up of name und port of the host

Next the necessary access data have to be specified, for example by providing username and password. Here, “user” is used as login, as it was used previously for the docker container.

Figure 4. Remote Access Configuration

Figure 4. Remote Access Configuration

After setting up the configuration, NetBeans tries to connect to the new remote host and requests the user to enter the associated password. Just like the username, the password is configured in the dockerfile and is set to ‘netbeans’ in the sample project.

Figure 5. Inputting the password

Figure 5. Inputting the password

When connecting was successful, development of new ROOT/C++ projects can begin.

Starting a New ROOT Project

To start a new C++ project, the usual NetBeans dialog is used. In this sample, a project with already existing source files will be created.

Figure 6. Creating a C++ project

Figure 6. Creating a C++ project

During the next step, the directory containing the already existing source code has to be chosen. Additionally, the build host has to be configured. Of course, the project can also be configured to use different build hosts at a later point. Especially if more than one docker instance is used, this becomes important for providing different environments.

Figure 7. Configuring of the build host and the source code directory

Figure 7. Configuring of the build host and the source code directory

To ensure NetBeans is accessing the correct files on the remote host, the local directory structure has to be aligned to the structure on the remote host during the first run. This corresponds to the volume mapping of docker (in this case). .Assigning Local and Remote Directories image::07_createProject-pathMapping.png[width=”350″, height=”200″]

The local path will be set to the source directory of the project – sample in this case – and /home/user/data/sample will be used as the remote path.

Figure 8. Choosing the remote directory

Figure 8. Choosing the remote directory

With this, the set-up of the project is completed and the development cycle can begin.

Executing the ROOT project

In this sample, a simple ROOT program will be written. Every time the program is called, it will plot the graph sin(x)/x in a random interval. The graph will additionally be saved to the file ex.pdf.
When first running the ROOT program, NetBeans will ask to specify the executable to run and its path. NetBeans provides a suggestion which normally can be used.

Figure 9. Choosing the Executable

Figure 9. Choosing the Executable

If now the generated pdf-file is viewed in evince, evince will refresh the PDF-view every time the program is executed. This is due to the embedded volume directly exposing updates so Evince can react to the changes.

Figure 10. Source code, displaying the resulting graph and the command line output of NetBeans.

Figure 10. Source code, displaying the resulting graph and the command line output of NetBeans.

Also debugging with NetBeans is possible this way. Debugging includes the insertion of breakpoints, stepwise code execution as well as the displaying the contents of variables. Development work is therefore as comfortable as it would be without the docker layer.

Figure 11. Remote-Debugging in Netbeans with ROOT and C++

Figure 11. Remote-Debugging in Netbeans with ROOT and C++

Outlook

Working with different versions is simple with docker – the version is specified in the dockerfile and for every ROOT version an own image is created. With these images the corresponding container instances can be created. For other Linux distributions the procedure is the same then.

What NetBeans theoretically is also supporting, but didn’t work with at least the 8.1rc2 version, is using X11 forwarding. Development of graphical applications in a docker container would be possible with this, if Linux is used as the environment.

This article illustrated the possibilities using docker in conjunction with NetBeans to aid the development of C++ applications and is intended to stimulate creativity for own experiments. Besides virtualization docker deserves room in the toolbox of every developer. Those who want to know further details about docker should absolutely take their time to study the docker documentation and consider attending a docker training.

For NetBeans, a direct docker support can be foreseen in the future, as there already are first ideas for this topic in the NetBeans Wiki: http://wiki.netbeans.org/Docker

A nice-to-have feature in this scenario would be to be able to use a docker exec, or some other command, instead of SSH. This would reduce the complexity of the setup. Also it would remove the dependency to the SSH which is built into NetBeans.

Author

Thomas Kruse, Karsten Sitterberg

Thomas Kruse studied computer science and works as an architect, consultant and coach for the trion development GmbH. In his free time he takes part in the open source community and is leader of the Java Usergroup in Münster.

Karsten Sitterberg was involved in data analysis with the ROOT framework during his studies at the university of Münster, especially during his masters thesis. Karsten is an Oracle certified Java developer and works as freelance developer and trainer for Java and the Web.


Leave a Reply

Be the First to Comment!

avatar
400
  Subscribe  
Notify of