Real-Time Configuration Updates in Microservices with Spring Cloud and RabbitMQ

Spring Cloud provides tools for developers to quickly implement common patterns in distributed systems, such as configuration management, service discovery, circuit breakers, intelligent routing, micro-proxy, control bus, short-lived microservices, and contract testing.

We will demonstrate the integration of Spring Cloud Config Server with Spring Cloud Bus to manage configurations for microservices efficiently. The Config Server centralizes configuration files stored in a GitHub repository, while the Bus leverages RabbitMQ (or similar brokers) to broadcast updates to all connected microservices.

1. Config Server: This will be a project or microservice that stores all configurations, whether sourced from Git, SVN, or local files.

2. Config Client: All other microservices will connect to the Config Server to retrieve configuration details.

   Config Server

Create a project and add these two dependencies :

1. Spring Cloud Config.

<dependency>
	<groupId>org.springframework.cloud</groupId>
	<artifactId>spring-cloud-config-server</artifactId>
</dependency>

2. Spring Boot Actuator

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Next, in the main class method, add this annotation @EnableConfigServer

@SpringBootApplication
@EnableConfigServer
public class ConfigServer {
  public static void main(String[] args) {
    SpringApplication.run(ConfigServer.class, args);
  }
}

We will continue with the option of reading configuration from a GitHub repository, as it is very common in real-world projects. Add the configuration for accessing files from your GitHub repository in the application.yml.

spring:
  application:
    name: "configserver"
  profiles:
    active: git
  cloud:
    config:
      server:
        git:
          uri: "provide the uri to your github repo"
          default-label: master
          timeout: 5 
          clone-on-start: true
          force-pull: true
server:
  port: 8071

Now, this will work. If you navigate to http://localhost:8071/<application-name>/<profile>, you will retrieve your configuration from GitHub.

Request URL Components

http://<config-server-host>:<port>/<application-name>/<profile>

For example, if the configuration repository contains a file named dataCatalog-prod.yaml, the URL to fetch the production configuration for the dataCatalog project would be: http://localhost:8071/dataCatalog/prod. To fetch the test configuration for the same project, the URL would be:http://localhost:8071/dataCatalog/test

When the endpoint is accessed, the Config Server:

1. Looks up the file <application-name>-<profile>.yml

   (e.g., dataCatalog-prod.yml) in the Git repository.

2. Reads the configuration properties.

3. Responds with the properties as a JSON object.

At this point, you have successfully fetched the configuration from your GitHub repository into the Config Server. Next, you need to send this configuration from the Config Server to the microservices.

Connect Config Server with Services.

We will connect Config Server with microservices. Add Config Client dependency in your microservice. But this will not get resolved. As it is not a spring cloud project. We need to add cloud dependencies.

Add Config Client in your POM.xml

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
</dependency>

Add Spring Cloud Version below your java version in your POM.xml

    <spring-cloud.version>2023.0.0</spring-cloud.version>

Add the dependency management just above your build in your POM.xml

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>${spring-cloud.version}</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

Add configure in your application.yml

  config:
    import: "optional:configserver:http://localhost:8071/"

configserver — This is the spring.application.name for the Config Server.

Optional — If the microservice is unable to connect to the Config Server, it can still start.

Add Actuator endpoints in your application.yml

management:
  endpoints:
    web:
      exposure:
        include: "*"

The reason for this setup is that when you make changes in your GitHub repository, the Config Server detects these changes. However, the microservices are not automatically notified about the updates. To address this, we have a few options:

1. Manually Invoke Actuator Refresh Endpoints

   All microservices have actuator endpoints. To reflect the changes, you need to invoke the refresh endpoint for each microservice. However, if you have 500 microservices, you would need to invoke the refresh endpoint 500 times, which is not practical.

2. Use Spring Cloud Bus

   With Spring Cloud Bus, configuration updates can be broadcast across services with a single bus-refresh invocation.The changes will then be broadcast to all other microservices through the message broker.

3. Use Webhook + Spring Cloud Bus

   With this approach, you don’t need to invoke the actuator refresh endpoint manually. As soon as a commit is made to the GitHub repository, the changes are automatically broadcast to all microservices via the message broker

Okay, now the basic connections are established.

Start the Config Server and then your microservices. If you make any changes to your GitHub repository, you will see those changes reflected in the Config Server (e.g., http://localhost:8071/dataCatalog/prod).

However, the microservices are not automatically notified about these changes. To see the changes in a microservice, invoke its refresh actuator endpoint (e.g., http://localhost:9000/actuator/refresh).

Spring Cloud Bus

Spring Cloud Bus links nodes of a distributed system with a lightweight message broker. This can then be used to broadcast state changes (e.g. configuration changes) or other management instructions.

With Spring Cloud, all properties for the microservices are stored in GitHub. If any configuration changes are made to those files at runtime, the Config Server will detect the updates. However, the microservices will not automatically be notified about the changes in the configuration files on GitHub.

To reflect the changes, you need to explicitly invoke the refresh actuator endpoint for each microservice. While this works, it becomes impractical when dealing with 500 microservices, as you would need to invoke the refresh endpoint for each one individually.

To resolve this, we use Spring Cloud Bus - it uses message broker(rabbitmq/others) to broadcast the changes to the microservices.

Install RabbitMQ

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.13-management

Add the Spring Starter Bus and RabbitMQ dependencies to all the microservices, including the Config Server.

  <dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-bus-amqp</artifactId>
  </dependency>

Expose the bus refresh

path of the actuator: /actuator/busrefresh

management:
  endpoints:
    web:
      exposure:
        include: "busrefresh"

Now, we need to establish a link between the microservices and the message queue by providing the RabbitMQ connection details. Add the following configuration to the application.yml file of your microservices.

  rabbitmq:
    host: "localhost"
    port: 5672 
    username: "guest" 
    password: "guest"

After this, everything is ready. If we make any changes to a property, we only need to invoke the bus-refresh endpoint once. The changes will then be propagated to all the microservices whose configurations have been updated.

Even after this, there is still one drawback: one needs to invoke the bus-refresh endpoint at least once to broadcast the changes to all the microservices.

Create a webhook in the GitHub repository using Hookdeck.

When running the Config Server and microservices locally, the localhost URL cannot be directly used for GitHub webhooks because GitHub requires publicly accessible URLs to send webhook requests. Hookdeck acts as an intermediary, exposing your hostport as a public URL that GitHub can use.

Add this dependency ONLY in the config server POM.xml.

  <dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-config-monitor</artifactId>
  </dependency>

Also add management and rabbitmq config in configserver → application.yml file.

  rabbitmq:
    host: "localhost"
    port: 5672 
    username: "guest" 
    password: "guest" 
    
   management:
   endpoints:
     web:
       exposure:
         include: "*"

If there is any change in the repository, it triggers the monitor API path. As soon as the monitor API path receives the webhook request, it invokes the refresh event behind the scenes with the help of Spring Cloud Bus and RabbitMQ.

hookdeck login
hookdeck listen [config server port] Source

After this, any commits made to the GitHub repository will trigger an event, and RabbitMQ will broadcast the changes to all the microservices.

Summary

This blog explores the integration of Spring Cloud Config Server with Spring Cloud Bus to manage configurations for microservices efficiently. The Config Server centralizes configuration files stored in a GitHub repository, while the Bus leverages RabbitMQ (or similar brokers) to broadcast updates to all connected microservices.

This setup ensures efficient configuration management for microservices, reducing manual effort and enabling seamless propagation of configuration updates across services.

1. Centralized Configuration

   1. Configurations for all microservices are stored in GitHub, enabling runtime updates without redeployments.

   2. Configuration files follow a standard naming convention (<application-name>-<profile>.yml).

2. Challenges with Manual Refresh

   1. Changes in GitHub are detected by the Config Server but not automatically propagated to microservices.

   2. Without automation, invoking the refresh actuator endpoint for each microservice becomes tedious, especially at scale.

3. Spring Cloud Bus Solution

   1. By adding Spring Starter Bus and RabbitMQ dependencies to all microservices, configuration updates can be broadcast across services with a single bus-refresh invocation.

4. Webhook Integration for Automation

   1. A GitHub webhook, paired with a monitor API, triggers the refresh event automatically upon repository changes.

   2. Tools like Hookdeck help handle local development challenges by exposing localhost to GitHub for webhook integration.

5. Seamless Broadcast

RabbitMQ handles broadcasting configuration updates to all impacted microservices, eliminating the need for manual refreshes.

Source Code

1. Event-Driven

Resources

1. Spring Cloud

2. Spring Cloud Config

3. RabbitMQ