days
1
4
hours
0
8
minutes
4
0
seconds
0
7
search
Java 9 & Java EE 8: They are finally here!
Part two

Spring Boot tutorial: Microservices and Kubernetes (part 2)

Michael Gruczel
Kubernetes
© Shutterstock /  Unconventional

Moving from the monolith to microservices has a lot of advantages. In part two of this tutorial, Michael Gruczel finishes his step-by-step tutorial teaching developers how to implement microservices architecture in Kubernetes and Pivotal Cloud Foundry with Spring Boot.

This is part two of a series. If you missed part one, it can be found here!

Service discovery in Kubernetes

You can use internal DNS in Kubernetes or you can use the same service discovery approach like in PCF (via Eureka) in Kubernetes as well. since Kubernetes offers another option to do service discovery, I will show that one in my example. We will deploy an instance of our weather app in Kubernetes in Listing 26 with some prebuilt containers. We will use the description (weather-app-v1.yaml) from Listing 27.

Listing 26

kubectl create -f weather-app-v1.yaml
kubectl get deployment
kubectl describe deployment weather-app
kubectl expose deployment weather-app --type="LoadBalancer"
kubectl get pods -l app=weather-app
kubectl get service weather-app 

Listing 27

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: weather-app
spec:
  replicas: 1
  template: 
    metadata:
      labels:
        app: weather-app
    spec:
      containers:
      - name: weather-app
        image: mgruc/weather-app:v1
        ports:
        - containerPort: 8090 

You can call the weather service now by ‘http://:8090/weather?place=springfield’. The weather service discovery information will be available inside Kubernetes under WEATHER_APP_SERVICE_HOST and WEATHER_APP_SERVICE_PORT. That means we will remove the service discovery annotation from our application and our rest controller and just use the variables.

Listing 28 shows how you could use that in a Spring Boot app.

Listing 28

@RestController
public class ConcertInfoController {

    @Value("${WEATHER_APP_SERVICE_HOST}")
    private String weatherAppHost;
    
    @Value("${WEATHER_APP_SERVICE_PORT}")
    private String weatherAppPort;
    

    @Bean
    RestTemplate restTemplate(){
        return new RestTemplate();
    }

    @Autowired
    RestTemplate restTemplate;

    @RequestMapping("/concerts")
    public ConcertInfo concerts(@RequestParam(value="place", defaultValue="")String place) {
        
        
        // retrieve weather data in order to add it to the concert infos 
        Weather weather = restTemplate.getForObject(
                         "http://" + weatherAppHost + ":" + weatherAppPort + "/weather?place=" 
                        + place, Weather.class);
        ...
        
    }
} 

So let us deploy the concert app as well (Listing 29, concert-app-v1.yaml) by executing ‘kubectl create -f concert-app-v1.yaml ’.

Listing 29

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: concert-app
spec:
  replicas: 1
  template: 
    metadata:
      labels:
        app: concert-app
    spec:
      containers:
      - name: concert-app
        image: mgruc/concert-app:v1
        ports:
        - containerPort: 8100 

The concert app is able to find the weather app, because the vars WEATHER_APP_SERVICE_HOST and WEATHER_APP_SERVICE_PORT are automatically injected by Kubernetes. That is just one of the options to implement service discovery within Kubernetes. If you want to try it from outside, check the public available IP (see Listing 30). The service is now reachable from outside by for example:’curl http://:8100/concerts?place=hamburg’.

Listing 30

kubectl expose deployment concert-app --type="LoadBalancer"    
kubectl get service concert-app 

Treat backing services as attached resources

This principle also originates from the twelve-factor app and comparable principles are known since a long time. Any resilient system should support graceful degradation. That means if a service dependency is missing, the service should not crash. Instead it should offer as much functionality as possible. If our service only works after a login, but the user database is not available, we will maybe have difficulties to offer something useful, but there are other use cases where we can follow the principle.

A great and lean library to support this in Java is for example Hystrix. Hystrix is a circuit breaker. If a service is not responding any more correctly, he will open the circuit for a given time and not propagate requests to that service any more.

Instead we can define a fallback method. By using that principle, errors or loops will not be propagated to next service layer, instead he will break a possible loop or error propagation. Listing 31 is an example for this.

Listing 31

@RequestMapping("/hello")
@HystrixCommand(fallbackMethod = "hellofallback")
public String hello() {
    String aName = .... complex logic to find a name
    return "Hello " + aName;
}
    
public String hellofallback() {
    return "Hello world";
}

In my example, if the weather service it offline, then the concert app can still provide concert information, see here for more information.

Execute the app as one or more stateless processes

Another principle from the twelve-factor app is important for microservices and the reason is obvious. If you want to scale at anytime and if you expect your service instance to be moved from one host to another anytime (e.g. auto failover), then you cannot have a state in your application. Any data that needs to be persisted must be stored in a backing service.

There are 2 major design mistakes which often result in a state:

  • Sessions managed by the application server
  • Local file persistence

Moving the session from the server into a database like Redis is simple. Apart from some dependencies, you often need just one annotation to do this in Spring Boot. If you use spring security and have web security enabled (@EnableWebSecurity) and you want to store your http sessions in a database, then Listing 32 is all you need to store your sessions in Redis.

Listing 32

 @EnableRedisHttpSession
public class HttpSessionConfig {
} 

To demonstrate this, I will add another app to our weather-app and concert-app. It is a chat-app. In order to use the chat app, you have to login. Additionally, you can ask in the chat window for concert or weather information. This data will then be retrieved from the weather and concert-app. Image 3 is the UI to showcase the functionality.

Kubernetes

Image 3: chat app UI

If you want to try it locally, then you could install Redis locally or better use Docker. Another option is to start a vagrant box with Redis. I have prepared a Redis vagrant box for that use case. If you want to use that one, go to folder vms/redis of my repository on GitHub and execute ‘vagrant up’. The source code for the chat-app is in source/chat-app.

If you have still eureka, the weather-app and the concert-app running from the last example, start now additionally the chat app by executing ‘./gradlew bootRun’. Login as ‘homer’ with password ‘beer’ on http://localhost:8080. You can now enter messages which will be stored in Redis. The session itself is stored in Redis as well and you can call other services indirectly if they are available e.g. by submitting ‘/weather springfield’ in the text field of the app. If the services are not available, the command will not work, but everything else will still continue to work.

Getting rid of the local file persistence for existing apps can be difficult. One solution is to use an object storage like AWS S3. There are other object storage solutions which use the same API. Take a look into Minio (https://github.com/minio/minio) if you want to install a S3 like storage locally. Instead of writing or reading a file, you can use an AWS S3 java client to write or read files from buckets. The next code snippet (Listing 33) should illustrate the idea, but I will not use it in my examples.

Listing 33

@RestController
@RequestMapping("/filehandling")
public class SampleS3Controller {

private AmazonS3Client client;
private String bucketName;

@RequestMapping(value = "upload", method = RequestMethod.POST)
    public void uploadFile(@RequestParam("name") String name,
                            @RequestParam("file") MultipartFile file) {
            try {
                ObjectMetadata objectMetadata = new ObjectMetadata();
                objectMetadata.setContentType(file.getContentType());
                client.putObject(new PutObjectRequest(bucketName, name, file.getInputStream(), 
                   objectMetadata));
            } catch (Exception e) {
            ...
        }
    }
} 

Execute the app as one or more stateless processes in PCF

Let us push our chat server to PCF and connect it to a Redis database (which is available as service in PCF) to use it with our already deployed weather-app and concert-app. The name of the Redis service must be Redis or the binding data has to fit to the scheme, as seen here.

Listing 34 shows how to do it in PCF.

Listing 34

# artefact is in artefacts 
cf push mgruc-pcf-chat-app -p chat-app-0.0.1.jar --random-route --no-start
cf bind-service mgruc-pcf-chat-app mgruc-service-registry
cf create-service rediscloud 30mb redis
# in case of an internal installation it would be: cf create-service p-redis shared-vm redis 
cf bind-service mgruc-pcf-chat-app redis
cf start mgruc-pcf-chat-app
cf app mgruc-pcf-chat-app 

You can now login to the chat service (url you can get from ‘cf app mgruc-pcf-chat-app’) and use it. It will store the sessions and messages in the Redis database and use the weather and concert service.

Execute the app as one or more stateless processes in Kubernetes

Let us deploy Redis and out chat service. Additionally we will make the service available from outside. We will deploy a simple Redis setup for our demo (Listing 35, see redis-master.yaml). In a real life example you would probably define a clustered setup.

Listing 35

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: redis
        role: master
        tier: backend
    spec:
      containers:
      - name: master
        image: gcr.io/google_containers/redis
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379 

Let’s deploy it.

Listing 36

kubectl create -f redis-master.yaml
kubectl expose deployment redis --type="LoadBalancer"

And now we deploy the chat app. I’ve added the connection information (using env variables) to the application properties for this. Apart from that our Spring Boot app is the same like in PCF or locally (Listing 37, see chat-app-v1.yaml).

Listing 37

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: chat-app
spec:
  replicas: 1
  template: 
    metadata:
      labels:
        app: chat-app
    spec:
      containers:
      - name: chat-app
        image: mgruc/chat-app:v1
        ports:
        - containerPort: 8080  

Let’s now deploy our chat app and make it available in Listing 38.

Listing 38

kubectl create -f chat-app-v1.yaml    
kubectl expose deployment chat-app --type="LoadBalancer"    
kubectl get service chat-app 

Open it in your browser http://:8080/. I will skip the file storage example here. Since Kubernetes is based on Docker, the files can be mounted to the host if needed or the S3 solution can be applied as well. The source for the Kubernetes examples is available under /source/docker-… in my repo.

Run admin/management tasks as one-off processes

This useful principle is one more time originated in the 12 factor apps manifest (XII. Admin processes). Admin tasks are often not part of the pipeline, because the administration is in a lot of companies done by another department. Software changes are in the most companies versioned and running through a pipeline.

Database changes, config changes, file cleanups are treated as exceptions, even if the risk is often bigger than a change in software. The exception I see the most are database changes. Some companies have dedicated DBAs which are not requested to commit their changes in the same pipelines. They are handled in a different way and not rolled out with the application change. That’s why I will just give one example how database changes can be integrated into the software.

Flyway is one of the libraries which is designed to execute versioned database changes. Spring Boot will automatically autowire Flyway with its DataSource and invoke it on startup if the dependency is given. Flyway supports versioned database migrations (unique version – applied exactly once) or repeatable migrations (re-applied every time their checksum changes). Let’s say you would have a folder named db/migration in you classpath (e.g. resource folder of application), then you could add sql files in that folder with the names ‘V1__lorem.sql’, ‘V2__ipsum.sql’ and so on. Flyway would execute all sql files which were not yet executed on the database in the given order (V[order]__[any name]) if you deploy an application.

Treat logs as event streams

If you accept that your application might be started on a different host any time (see autofailover, autoscaling, …), that it should be easy to scale it, than local file systems are not reliant. That means writing your logs to files is not reliant and your logging capabilities has to scale as well. The consequence is that you understand your logs as streams (12 factor apps – XI. Logs) and pipe your logs as a stream to different channels.

Treat logs as event streams in PCF

Cloud foundry collects the system out and all requests into one big stream. You can link yourself to the stream by the command ‘cf logs mgruc-pcf-chat-app [–recent]’. A better solution is to stream your PCF application logs to your external logging solution. PCF supports log forwarding compatible to syslog out of the box. Let’s say we want to forward our logs from the pcf-chat-app to an external logstash agent, then we could follow Listing 39.

Listing 39

cf cups my-logstash -l syslog://<ip_of_your_logstah_host>:<your_logstah_port>
cf bind-service mgruc-pcf-chat-app my-logstash
cf restage mgruc-pcf-chat-app 

The same binding will work with splunk, papertrail or other services which support syslog.

Treat logs as event streams in Kubernetes

There are several ways to access your logs in Kubernetes. One way is to use the cli e.g. ‘kubectl logs deployment/chat-app’. You can use kubectl logs to retrieve logs but you want to stream your logs to an external solution. The out-of-the box support for Kubernetes is not as straight forward as in PCF, you have to decide yourself on a strategy and implement it.

Some of your options are:

  • Push logs directly to a backend from within your application, means you could use a log appender which sends directly to splunk or other tools.
  • You can use a streaming sidecar container, means you link the logs from you application container is a second container. The second container contains a log-forwarder.
  • You can use a log forwarder on the host and mount the logfiles from the Docker container to the host.

Dynamic Tracing and monitoring

If you have hundreds of services which call each other and your software is changing dynamically and fast, then you want to know where you lose your time in order to troubleshoot latency problems and strange behavior. That means additionally to the monitoring of the services itself, you have to add monitoring for the network of microservices. There are several tools and cloud provider like App Dynamics, New Relic and Dynatrace available to monitor distributed apps, only to name a few. Everyone can easily fill an article.

I will show instead two open source options:

  • Hystrix with Turbine
  • Zipkin

I introduced Hystrix already as circuit breaker. Additionally Hystrix is able to export some metrics. Hystrix needs this information in order to open and close the circuits (number of failed requests, response times, …) and the embedded Hystrix dashboard is able to visualize them for each service.

If you have hundreds of services and thousands of circuit breakers, you don’t want to monitor the different dashboards of the different services instead you want to collect them and visualize them as one dashboard. This is when Turbine comes in handy. It’s also open source from Netflix. Turbine is designed to aggregate Hystrix streams to a single dashboard.

The second framework I want to introduce is Zipkin. Zipkin is a distributed tracing system, that means Zipkin visualizes the trace of requests over several services. Zipkin is available as self running jar based on spring boot. You can easily start it locally as jar or as Docker container (Listing 40).

Listing 40

# as jar
wget -O zipkin.jar 'https://search.maven.org/remote_content?g=io.zipkin.java&a=zipkin-server&v=LATEST&c=exec'
java -jar zipkin.jar
# or as Docker
docker run -d -p 9411:9411 openzipkin/zipkin 

Or you can create your own spring boot application and add the annotation @EnableZipkinStreamServer to the main class to make it to a Zipkin server. Once you’ve started Zipkin, you can browse to http://your_host:9411 to find traces. If you now want to add your service to Zipkin, you have to add a sampler to send the information to the Zipkin server.
The easiest way (but not most performant) to do this is probably the AlwaysSampler (Listing 41).

Listing 41

@Bean
public AlwaysSampler defaultSampler() {
    return new AlwaysSampler();
} 

In a real use case, you would send the data to Zipkin aggregated after a delay in order to reduce the overhead. Image 4 and Image 5 are Zipkin samples from our example.

Kubernetes

Image 4: Zipkin Dependencies

 

Kubernetes

Image 5: Zipkin traces

Dynamic Tracing and monitoring in PCF

Let’s try it in PCF. You can use my pre-build artefacts (see Listing 42).

Listing 42

# folder artefacts
# new zipkin service
cf push mgruc-pcf-zipkin -p zipkin.jar --random-route --no-start
cf bind-service mgruc-pcf-zipkin mgruc-service-registry
# new version of apps which export data to zipkin and the hystrix dashboard
cf push mgruc-pcf-chat-app -p chat-app-with-zipkin-0.0.1.jar --random-route --no-start
cf push mgruc-pcf-weather-app -p weather-app-with-zipkin-0.0.1.jar --random-route --no-start
cf push mgruc-pcf-concert-app -p concert-app-with-zipkin-0.0.1.jar --random-route --no-start
# attach hystrix-turbine dashboard
cf create-service p-circuit-breaker-dashboard standard mgruc-circuit-breaker-dashboard 
cf bind-service mgruc-pcf-chat-app mgruc-circuit-breaker-dashboard 
cf bind-service mgruc-pcf-weather-app mgruc-circuit-breaker-dashboard
cf bind-service mgruc-pcf-concert-app mgruc-circuit-breaker-dashboard
# attach to zipkin 
cf app mgruc-pcf-zipkin
cf set-env mgruc-pcf-weather-app SPRING_ZIPKIN_BASE-URL http://<zipkin url>
cf set-env mgruc-pcf-concert-app SPRING_ZIPKIN_BASE-URL http://<zipkin url>
cf set-env mgruc-pcf-chat-app SPRING_ZIPKIN_BASE-URL http://<zipkin url>
# start the apps
cf start mgruc-pcf-zipkin
cf start mgruc-pcf-weather-app
cf start mgruc-pcf-concert-app
cf start mgruc-pcf-chat-app
cf apps 

If you now use the chat app for storing messages and request concert and weather data, Zipkin will visualize your dependencies and latencies, Eureka will show you all services and the Hystrix dashboard will help you to identify problems in the communication. Image 6 and 7 are screenshots of the PCF service discovery and the PCF Hystrix Dashboard.

 

Kubernetes

Image 6: PCF service discovery

 

Kubernetes

Image 7: PCF hystrix

If you want to have a look into the code which enables Zipkin, then check weather-app-with-zipkin folder, the concert-app-with-zipkin folder and chat-app-with-zipkin folder in my repo.

Dynamic Tracing and monitoring in Kubernetes

Let us use Zipkin as Docker container in Kubernetes.

Listing 43

kubectl delete deployment weather-app
kubectl delete deployment concert-app
kubectl delete deployment chat-app

kubectl create -f zipkin.yaml    
kubectl expose deployment zipkin --type="LoadBalancer"
kubectl get service zipkin
    
kubectl create -f weather-app-v2.yaml
kubectl create -f concert-app-v2.yaml
kubectl create -f chat-app-v2.yaml

kubectl get service chat-app
kubectl get service zipkin 

If you now use the chat app for storing messages and request concert and weather data, Zipkin will visualize your dependencies and latencies. I will skip the Hystrix-dashboard example for Kubernetes here since the concepts should be clear now, but with the help of Turbine you can of course create a Hystrix Dashboard in Kubernetes as well.

Clean up

Nothing in life is free, least of all the cloud. Cloud services cost money, so it’s important to clean up after you’re done.. The free account should be fully sufficient to execute the examples, but I highly recommend cleaning up after yourself once finished.

Clean up in PCF

In PCF the following commands in Listing 44 should delete the examples. Please verify that everything is removed after executing the commands.

Listing 44

cf delete updatedemo
cf delete-service autoscaler
cf delete mgruc-pcf-chat-app
cf delete mgruc-pcf-weather-app
cf delete mgruc-pcf-concert-app
cf delete mgruc-pcf-zipkin
cf delete-service redis
cf delete-service mgruc-circuit-breaker-dashboard 
cf delete-service mgruc-service-registry
cf delete-service my-logstash 

Clean up in Kubernetes

In Kubernetes the following commands in Listing 45 should delete the most things. Please verify that everything is removed after executing the commands.

Listing 45

kubectl delete service weather-app
kubectl delete deployment weather-app
kubectl delete service concert-app
kubectl delete deployment concert-app
kubectl delete service chat-app
kubectl delete deployment chat-app
kubectl delete service redis
kubectl delete deployment redis
kubectl delete service zipkin
kubectl delete deployment zipkin
gcloud container clusters delete example-cluster 

Summary

Moving from monolithic applications to microservices is a big shift of complexity. Luckily there are efficient ways for this. In this article I could of course only scratch the surface of some of the most important tools and options you can use to make your architecture resilient and scalable. I hope that I could mediate the underlying principles and show how a real microservice architecture could look like by using some simple but practical examples.

Author
Michael Gruczel
As Advisory Consultant at Dell EMC, Michael Gruczel advises companies in Application Modernization, DevOps, and BigData. Part of this consultancy work is the full stack including steps like Discovery, Software Development, Testing, Infrastructure, Release, Deployment and Operation of Software.

Comments
comments powered by Disqus