Eureka never unregisters a service

Question

I'm currently facing an issue where Eureka does not unregister a registered service. I've pulled the Eureka server example straight from git hub and made only one change, eureka.enableSelfPreservation = false. My application.yml looks like this:

server:
  port: 8761
eureka:
  enableSelfPreservation: false
  client:
    registerWithEureka: false
fetchRegistry: false
  server:
    waitTimeInMsWhenSyncEmpty: 0

I've read that if 85% of the registered services stop delivering heartbeats within 15 minutes, Eureka assumes the issue is network related and does not de-register the services that are not responding. In my case I have only one service running, so I disabled self-preservation mode. I am abruptly killing the process and Eureka leaves the service registered for what seems like an indefinite amount of time.

My client's application.yml looks like this:

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 3
  client:
    healthcheck:
      enabled: true
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
  appInfo:
    replicate:
      interval: 3
    initial:
      replicate:
        time: 3
spring:
  rabbitmq:
    addresses: ${vcap.services.${PREFIX:}rabbitmq.credentials.uri:amqp://${RABBITMQ_HOST:localhost}:${RABBITMQ_PORT:5672}}

My goal is to create a demo where Eureka quickly detects the service is no longer running and another service that is started can quickly register itself.

As of now, once the eureka client is started, it registers in 3 seconds. It just never un-registers when the service is abruptly terminated. After I kill the service, the Eureka dashboard reads:

EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE.

How can I prevent this behavior?

Answer 1

I realized that self preservation mode was never actually being disabled. It turns out the actual property is

eureka.server.enableSelfPreservation=false

(See DefaultEurekaServerConfig Code), which I haven't found documented anywhere. This resolved my issue.

Answer 2

I made service de-registration work by setting the below values

Eureka server application.yml

eureka:
  server:
    enableSelfPreservation: false

Service application.yml

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 1
    leaseExpirationDurationInSeconds: 2

The full example is here https://github.com/ExampleDriven/spring-cloud-eureka-example

Answer 3

After struggling a lot, finally I got solution if any service unregistered from Eureka server due to some issue. It will notify to the Admin by extending the HealthCallback of Eureka-Server APIs. Let Say Service-A register with Eureka. Hence Eureka Client is integrate with Service-A and Implement following Callbacks in Service A.

Service-A [Eureka-Client] Add following properties in properties files.

#Eureka Configuration
eureka.client.eureka-server-port=8761
eureka.client.register-with-eureka=true
eureka.client.healthcheck.enabled=false
eureka.client.prefer-same-zone-eureka=true
eureka.client.fetchRegistry=true
eureka.client.serviceUrl.defaultZone=${eurekaServerURL1}, ${eurekaServerURL2}
eureka.client.eureka.service-url.defaultZone=${eurekaServerURL1}, ${eurekaServerURL2}
eureka.instance.hostname=${hostname}
eureka.client.lease.duration=30
eureka.instance.lease-renewal-interval-in-seconds=30
eureka.instance.lease-expiration-duration-in-seconds=30

Add following java files.

@Component
public class EurekaHealthCheckHandler implements HealthCheckHandler, ApplicationContextAware, InitializingBean {


    static Logger logger = LoggerFactory.getLogger(EurekaHealthCheckHandler.class);

    private static final Map<Status, InstanceInfo.InstanceStatus> healthStatuses = new HashMap<Status, InstanceInfo.InstanceStatus>() {{
        put(Status.UNKNOWN, InstanceInfo.InstanceStatus.UNKNOWN);
        put(Status.OUT_OF_SERVICE, InstanceInfo.InstanceStatus.OUT_OF_SERVICE);
        put(Status.DOWN, InstanceInfo.InstanceStatus.DOWN);
        put(Status.UP, InstanceInfo.InstanceStatus.UP);
    }};

@Autowired
ComunocationService comunocationService ;
    private final CompositeHealthIndicator healthIndicator;

    private ApplicationContext applicationContext;

    public EurekaHealthCheckHandler(HealthAggregator healthAggregator) {
        Assert.notNull(healthAggregator, "HealthAggregator must not be null");
        this.healthIndicator = new CompositeHealthIndicator(healthAggregator);

        Health health = healthIndicator.health();
        logger.info(" =========== Testing =========== {}", health.toString() );
    }

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }

    @Override
    public void afterPropertiesSet() throws Exception {

        final Map<String, HealthIndicator> healthIndicators = applicationContext.getBeansOfType(HealthIndicator.class);
        for (Map.Entry<String, HealthIndicator> entry : healthIndicators.entrySet()) {
            logger.info("======"+ entry.getKey() +"============= "+entry.getValue());
            healthIndicator.addHealthIndicator(entry.getKey(), entry.getValue());
        }
    }

    @Override
    public InstanceInfo.InstanceStatus getStatus(InstanceInfo.InstanceStatus instanceStatus) {
        logger.info("============== Custome Eureka Implementation ==================="+ getHealthStatus());
        return getHealthStatus();
    }

    protected InstanceInfo.InstanceStatus getHealthStatus() {
        final Status status = healthIndicator.health().getStatus();
        return mapToInstanceStatus(status);
    }

    protected InstanceInfo.InstanceStatus mapToInstanceStatus(Status status) {
        logger.info("============== Test Custome Eureka Implementation ==================={}", status);
        if(status.equals(InstanceInfo.InstanceStatus.UP)) {
            // Send mail after configured times
              comunocationService.sendEmail("ServiceName");
        }
        if(!healthStatuses.containsKey(status)) {
            return InstanceInfo.InstanceStatus.UNKNOWN;
        }
        return healthStatuses.get(status);
    }

    public void getstatusChangeListner() {
        ApplicationInfoManager.StatusChangeListener statusChangeListener = new ApplicationInfoManager.StatusChangeListener() {
            @Override
            public String getId() {
                return "statusChangeListener";
            }
            @Override
            public void notify(StatusChangeEvent statusChangeEvent) {
                if (InstanceStatus.DOWN == statusChangeEvent.getStatus() ||
                        InstanceStatus.DOWN == statusChangeEvent.getPreviousStatus()) {
                    // log at warn level if DOWN was involved
                    logger.warn("Saw local status change event {}", statusChangeEvent);
                } else {
                    logger.info("Saw local status change event {}", statusChangeEvent);
                }

            }
        };
    }


}

and

@Configuration
public class EurekaHealthCheckHandlerConfiguration {

    @Autowired(required = false)
    private HealthAggregator healthAggregator = new OrderedHealthAggregator();

    @Bean
    @ConditionalOnMissingBean
    public EurekaHealthCheckHandler eurekaHealthCheckHandler() {
        return new EurekaHealthCheckHandler(healthAggregator);
    }
}

This is absolutely working and well tested code