Spring Cloud Hystrix Example

In this tutorial, we will be discussing creating a self-healing and fault tolerance services with circuit breaker pattern using Netflix Hystrix. We will be discussing about failures in a distributed system and how spring cloud Netflix hystrix helps to create such a fault tolerance system using annotations such as @EnableCircuitBreaker, @HystrixCommand. At the end, we will enable hystrix dashboard within our example using @EnableHystrixDashboard.

In a distributed system, failure is inevitable. We have many micro-services that combine together to make a complete system. Sometimes, there can be a network error, or software failure or even hardware failure and this leads to a cascading failure in a distributed system.

In these scenarios, a common way to prevent cascading failure is to use circuit breaker pattern. It is a design pattern used in modern software development.

It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure or unexpected system difficulties and provide resilience to our microservice architecture based distributed system.

Spring cloud helps to build fault tolerance system by implementing Netflix Hystrix. It is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.

Hystrix implements circuit breaker pattern. It wraps service calls and watches for failures and provides a default rolling window for 10 seconds. For a 20 request volume, if the error rate is more than 50% then the service is tripped and no request is allowed through.

In this tutorial, we will be building a spring cloud app using Netflix Hystrix. This app will have a eureka discovery server that we built in our last example - Spring Cloud Netflix Eureka Discovery and a eureka service and eureka client.

All the Hystrix implementation will be in our client service and our client app will be a fault tolerance system against the unavailability of our service. During unavailability of our service, a fallback method will be invoked with the hystrix command.

Here, unavailability of any service can be understood as failure or time-out of a third party API call or a downstream microservice API call.

Another scenario could be an instance of a downstream microservice went down or all the instances of a particular microservice went down. These are all the scenarios which we need to care of while designing our system to provide resilience. We will try to create these scenarios and analyse how Hystrix behaves in all these scenarios.

Spring Cloud Discovery Server

We will be using our existing discovery server implementation on github. Following is the application.properties file and DiscoveryServerApplication.javathat we used while implementing discovery server in our last example.

spring.application.name=discovery-server
eureka.client.register-with-eureka=false
eureka.client.fetch-registry=false
server.port=8761

@EnableEurekaServer
@SpringBootApplication
public class DiscoveryServerApplication {

	public static void main(String[] args) {
		SpringApplication.run(DiscoveryServerApplication.class, args);
	}
}

First Microservice(3rd Party Server)

Now we will be creating a sample microservice that will register itself to discovery server. We will assume this to be our 3rd party API provider or a downstream microservice that will fail at some point. There will be multiple instances running of this microservice to reproduce our scenarios.

Let us generate a spring boot project from http://start.spring.io

This brings following maven dependencies.

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.cloud</groupId>
		<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
	</dependency>

	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-test</artifactId>
		<scope>test</scope>
	</dependency>
</dependencies>

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-dependencies</artifactId>
			<version>${spring-cloud.version}</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

Now, let us define our application.properties.spring.application.name is the unique identifier for this service and eureka.client.service-url.defaultZone is the url of service discvery server.

spring.application.name=netflix-service
eureka.client.service-url.defaultZone=http://localhost:8761/eureka
server.port=8085

Following is our spring boot application class.We want our service to be registered on discovery server and hence the annotation @EnableDiscoveryClient

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;

@SpringBootApplication
@EnableDiscoveryClient
public class NetflixServiceApplication {

	public static void main(String[] args) {
		SpringApplication.run(NetflixServiceApplication.class, args);
	}
}

Now we will have an endpoint exposed at /demo

@RestController
public class DemoController {

    @GetMapping("/demo")
    public String Demo(){
        return "demo";
    }
}

Second Microservice(Client App with Hystrix Configuration)

This will be our eureka client app where we will configure Hystrix and we will make a call to the First microservice APIs from here.

Let us generate it from http://start.spring.io

This brings following maven dependencies.

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-actuator</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.cloud</groupId>
		<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.cloud</groupId>
		<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.cloud</groupId>
		<artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
	</dependency>

	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-test</artifactId>
		<scope>test</scope>
	</dependency>
</dependencies>

<dependencyManagement>
	<dependencies>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-dependencies</artifactId>
			<version>${spring-cloud.version}</version>
			<type>pom</type>
			<scope>import</scope>
		</dependency>
	</dependencies>
</dependencyManagement>

Now let us define our spring boot application class.@EnableCircuitBreaker tells Spring Cloud that the Reading application uses circuit breakers and to enable their monitoring, opening, and closing (behavior supplied, in our case, by Hystrix).

@SpringBootApplication
@EnableCircuitBreaker
@EnableDiscoveryClient
public class NetflixClientApplication {

	public static void main(String[] args) {
		SpringApplication.run(NetflixClientApplication.class, args);
	}
}

Following is the application.properties

spring.application.name=netflix-client
eureka.client.service-url.defaultZone=http://localhost:8761/eureka
server.port=9090

Now, let us expose sample endpoint in the client project.

@RestController
public class ClientController {

    @Autowired
    private ClientService clientService;

    @GetMapping("/test")
    public String test(){
        return clientService.test();
    }
}

Now we will be using @HystrixCommand to define our fallback method for any failure of an API call. @HystrixCommand only works in a class marked with @Component or @Service and hence we following service defined. We have @HystrixCommand annotated on test() and hence when our API call fails the fallback method will be executed.

Here, we have injected the EurekaClient instance to get the registered instance details of eureka-service

@Service
public class ClientService {

    @Autowired
    private EurekaClient eurekaClient;

    @HystrixCommand(fallbackMethod = "failed")
    public String test(){
        InstanceInfo instanceInfo = eurekaClient.getNextServerFromEureka("netflix-service", false);
        String serviceBaseUrl = instanceInfo.getHomePageUrl();
        String result = new RestTemplate().getForObject(serviceBaseUrl + "demo", String.class);
        return result;
    }

    public String failed(){
        return "failed";
    }
}

Here, we can have multiple situations such as may be the instance of netflix-service is itself down(we are running a single instance now) or this particular API call might have thrown some exception or the API response time is more then the API response threshold time.

Test Spring Cloud netflix Hystrix

Now let us start our discovery server, netflix eureka service and client in a row and hit http://localhost:9090/test

Now we will stop the service and hit the url again and this time the fallback method will be invoked.

Now, let us discuss the different scenarios associated with the failure that we mentioned above.

Different Hystrix Filure Scenarios

Assume the instance itself is down. It is very straight forward in this case. The instance is down and hence the method test() will throw exception at the vary first line. Hence, our fallback method failed() will execute.

Next scenario could be the service API is taking too much time to provide the response. By default, Hystrix has a default timeout set to 500ms. Let us add below line in our demo() inside DemoController to deliberately delay the API response.

@GetMapping("/demo")
    public String Demo(){
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return "demo";
    }

In this case also our Hystrix fallback method will be executed. This timeout can be increased by adding custom commandProperties.

@HystrixCommand(fallbackMethod = "failed",  commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "2000")
    })

It can be also configured using .yaml file.

The Next scenario could be the exception condition where the API throws certain exception. The exception could be with status code 400 or 500.

If the exception status code is 500, we want to execute our fallback method for obvious reason but what if the exception status code is other then 500. But with the default configuration, for an error code 400, the fallback method will be executed which is not intended.

Let us configure our Hystrix not to invoke the fallback method for some custom exceptions which needs to be propagated to the client.

@HystrixCommand(fallbackMethod = "failed",  commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000")
    }, ignoreExceptions = {CustomException.class})

Hystrix Dashboard

Hystrix dashboard is a web application that provides a dashboard for monitoring applications using Hystrix. Hystrix dashboard is not intended to be deployed on untrusted networks, or without external authentication and authorization. To enable Hystrix dashboard, we only have to annotate our spring boot main class with @EnableHystrixDashboard. Following is the dependency information of Hystrix project.

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-starter-netflix-hystrix-dashboard</artifactId>
</dependency>

The Hystrix dashboard is avialable at http://localhost:9090/hystrix for client-service instance in our case.

Propagating the Security Context

One more important aspect of Hystrix is that by default, the methods with @HystrixCommand will be executed on a different thread because the default execution.isolation.strategy is ExecutionIsolationStrategy.THREAD. So, if we want some thread local context to propagate into a @HystrixCommand, we need to use execution.isolation.strategy=SEMAPHORE.

@HystrixCommand(fallbackMethod = "failed",  commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000"),
                    @HystrixProperty(name="execution.isolation.strategy", value="SEMAPHORE")
    }, ignoreExceptions = {CustomException.class})

The same thing applies if you are using @SessionScope or @RequestScope.

Next thing that we can explore is using of Hystrix with Fiegn client which we have discussed here.

Conclusion

In this article, we discussed about implementing spring cloud netflix hystrix to build a fault tolerance system using circuit breaker pattern.The source can be downloaded from here. If you have anything that you want to add or share then please share it below in the comment section