Circuit Breakers: Making Resilient Systems (Part-2)

In our last post, we got to know about the concept and significance of resilient systems. We also studied the concept of circuit breakers, with a great deal of theoretical explanations and standardizations. This post is going to get a little deeper into the execution phase, wherein we discuss the issues we need to cater while using a circuit breaker. I am also trying to implement a circuit breaker from scratch, which I will certainly put up in a different article, but for this post, we will be using the Hystrix-go library.

Hystrix is an open-source project, written initially by Netflix in JAVA, which provides developers to easily use the Circuit Breaker pattern. Hystrix-go is just a Golang implementation of Hystrix. So without any more delays, let us start from where we left, Circuit Breakers. 

States of a Circuit Breaker

We generally associate three states with a circuit breaker. Closed, Open and Half Open.

1. Closed:   The default state of any circuit breaker is Closed. This is the state in which any request is normally sent to the external service and the result is awaited. All the responses are monitored in this state and if the errors cross a certain limit, the circuit breakers change the state to Open.
2. Open:  This is the resilient state of the system. If the external service is giving too many errors, the circuit breakers switches to this state to prevent the external system from overloading.
3. Half-Open: This is the gateway for a Circuit Breaker in Open state to get back to the closed state. Periodically a circuit breaker checks if the external service has become healthy again. If it is healthy, the circuit breaker goes back to the closed state, else it remains open.

Time to Code

We will be using hystrix-go library which is the implementation of Netflix’s Hystrix Library in Golang. Please note that you need to have an understanding of go channels and select statements to understand this code completely. The library provides us with two APIs, Hystrix.Do(sync) and Hystrix.Go(async). For most of the external API calls, we use Hystrix.Go API. Before using a circuit breaker, we need to configure it. In case you are unaware of them, we discussed the configuration and parameters of a Circuit Breaker in the last post.
Let’s see how the parameters are configured.

hystrix.ConfigureCommand("circuitBreakerName", hystrix.CommandConfig{	
		Timeout: 50000,
		MaxConcurrentRequests: 300,
		RequestVolumeThreshold: 10,
		SleepWindow: 1000,
		ErrorPercentThreshold: 50,
	})

“circuitBreakerName” is the name of the circuit breaker and is the only identifier available. You can have one or more than one circuit breaker in your code. For example, you are allowed to have a separate circuit breaker for each of the endpoints you hit or you can have only one circuit breaker catering all the endpoints. This is just a design choice.

Let us see what we did in this snippet. We are just configuring the parameters of a circuit breaker named “circuitBreakerName”.  That is the only thing that this snippet does. Now let us use the circuit breaker.

func DoSomethingWithCB() {
        resultCh := make(chan int)
	errCh := hystrix.Go("circuitBreakerName", func() error {
		resp, err := doSomething()  //Do something is the function that makes the external call.
		if err != nil {
			return err
		}

		if err == nil {
			resultCh <- true
		}
		return err
	}, nil)

	select {
	case res := <-resultCh:
		log.Println("success to get response from sub-system:", string(res))
		w.WriteHeader(http.StatusOK)
	case err := <-errCh:
		log.Println("failed to get response from sub-system:", err.Error())
		w.WriteHeader(http.StatusServiceUnavailable)
	}
}

Hystrix.Go is an async API and thus we need to use the channels. We made a channel resultCh , to hold the response and Hystrix.Go returns a channel of errors. Inside the hystrix.Go we call the doSomething() function and capture the response and error and pass them to the respective channels. Outside the Hystrix.Go function we use a select statement we capture the response from channels into respective variables and handle them.

Designing a clean architecture involves a lot of practice and experience, and a bit of reading.  I was personally benefitted a lot from this amazing book, Hands-On Software Architecture with Golang . Clean explanation and relatable examples make it a must-read.
But only reading takes you nowhere, you have to get your hands dirty and start coding.
We will be revisiting Circuit Breakers and try to implement a Circuit Breaker on our own instead of using Hystrix-Go library. In the meanwhile, revisit your own projects and think of any fallback mechanisms you can apply to make them resilient.