1

I have a spring cloud app acting as a turbine server and Hystrix dashboard. It has a Eureka client and the dashboard comes up ok and can see an individual Hystrix stream but I don't get anything in the turbine stream.

This is the application.yml:

eureka:
  client:
    serviceUrl:
      defaultZone: ${vcap.services.eureka-service.credentials.uri:http://10.12.2.99:8761}/eureka/

turbine:
  aggregator:
    clusterConfig: ITEMS-SERVICE
  appConfig: items-service

If I look in the logs when the Turbine/dashboard app starts up it looks like it finds my Hystrix services ok:

2015-03-10 14:45:35.256  INFO 1383 --- [        Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery  : Fetching instance list for apps: [items-service]
2015-03-10 14:45:35.257  INFO 1383 --- [        Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery  : Fetching instances for app: items-service
2015-03-10 14:45:35.257  INFO 1383 --- [        Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery  : Received instance list for app: items-service, size=2
2015-03-10 14:45:35.264  INFO 1383 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Retrieved hosts from InstanceDiscovery: 2
2015-03-10 14:45:35.265  INFO 1383 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Found hosts that have been previously terminated: 0
2015-03-10 14:45:35.265  INFO 1383 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Hosts up:2, hosts down: 0
2015-03-10 14:45:35.270  INFO 1383 --- [        Timer-0] c.n.t.monitor.instance.InstanceMonitor   : Url for host: http://nf-app1:8080/hystrix.stream ITEMS-SERVICE
2015-03-10 14:45:35.270  INFO 1383 --- [        Timer-0] c.n.t.handler.TurbineDataDispatcher      : 
Just added and starting handler tuple: ITEMS-SERVICE_agg_aggClusterEventHandler
2015-03-10 14:45:35.271  INFO 1383 --- [        Timer-0] c.n.turbine.data.AggDataFromCluster      : Per handler dispacher started for: ITEMS-SERVICE_agg_aggClusterEventHandler
2015-03-10 14:45:35.273  INFO 1383 --- [        Timer-0] c.n.t.monitor.instance.InstanceMonitor   : Url for host: http://nf-app2:8080/hystrix.stream ITEMS-SERVICE

If I hit the Turbine stream at 

http://localhost:8080/turbine.stream?cluster=ITEMS-SERVICE

I get:

: ping
data: {"timestamp":1426018670641,"name":"meta","reportingHostsLast10Seconds":0,"type":"meta"}

: ping
data:     {"timestamp":1426018673643,"name":"meta","reportingHostsLast10Seconds":0,"type":"meta"}

: ping
data: {"timestamp":1426018676645,"name":"meta","reportingHostsLast10Seconds":0,"type":"meta"}

and when I look in the Turbine/dashboard log I don't see any errors: 

2015-03-10 14:48:33.804  INFO 1383 --- [nio-8080-exec-1] c.n.t.s.servlet.TurbineStreamServlet     : FilterCriteria: []
2015-03-10 14:48:33.804  INFO 1383 --- [nio-8080-exec-1] c.n.t.s.servlet.TurbineStreamServlet     : StatsType filters: []
2015-03-10 14:48:33.808  INFO 1383 --- [nio-8080-exec-1] c.n.t.s.TurbineStreamingConnection       : Relevance config: []
2015-03-10 14:48:33.808  INFO 1383 --- [nio-8080-exec-1] c.n.t.s.TurbineStreamingConnection       : Relevance metrics config: {}
2015-03-10 14:48:33.809  INFO 1383 --- [nio-8080-exec-1] c.n.t.monitor.cluster.ClusterMonitor     : Registering event handler for cluster monitor: StreamingHandler_f1b7fc33-d0ed-4206-9976-fd4b2347bdd5
2015-03-10 14:48:33.809  INFO 1383 --- [nio-8080-exec-1] c.n.t.handler.TurbineDataDispatcher      : 

Just added and starting handler tuple: StreamingHandler_f1b7fc33-d0ed-4206-9976-fd4b2347bdd5
2015-03-10 14:48:33.809  INFO 1383 --- [nio-8080-exec-1] c.n.turbine.data.AggDataFromCluster      : Per handler dispacher started for: StreamingHandler_f1b7fc33-d0ed-4206-9976-fd4b2347bdd5
2015-03-10 14:48:33.809  INFO 1383 --- [nio-8080-exec-1] c.n.t.monitor.cluster.ClusterMonitor     : All event handlers for cluster monitor: [StaticListener_For_Aggregator, StreamingHandler_f1b7fc33-d0ed-4206-9976-fd4b2347bdd5]
2015-03-10 14:48:33.809  INFO 1383 --- [nio-8080-exec-1] c.n.t.monitor.cluster.ClusterMonitor     : Starting up the cluster monitor for ITEMS-SERVICE_agg
2015-03-10 14:48:35.265  INFO 1383 --- [        Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery  : Fetching instance list for apps: [items-service]
2015-03-10 14:48:35.266  INFO 1383 --- [        Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery  : Fetching instances for app: items-service
2015-03-10 14:48:35.266  INFO 1383 --- [        Timer-0] o.s.c.n.turbine.EurekaInstanceDiscovery  : Received instance list for app: items-service, size=2
2015-03-10 14:48:35.266  INFO 1383 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Retrieved hosts from InstanceDiscovery: 2
2015-03-10 14:48:35.266  INFO 1383 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Found hosts that have been previously terminated: 0
2015-03-10 14:48:35.267  INFO 1383 --- [        Timer-0] c.n.t.discovery.InstanceObservable       : Hosts up:2, hosts down: 0

I get valid Hystrix data if I hit one of the item streams directly or through the dashboard (eg http://nf-app1:8080/hystrix.stream ).

It looks like the Turbine app is finding the two items-services via Eureka and their Hystrix stream endpoints are valid and work so I can't figure out why the Turbine stream is empty.

Matt Reynolds
  • 81
  • 1
  • 7
  • The data you see is not "invalid" I would say. It looks like the circuits were never used in the app instances that turbine is pointing at though. Maybe you need to flex them a bit to make them send some data? Or maybe you are pointing at different physical instances with your http://nf-app1:8080/hystrix.stream experiment? – Dave Syer Mar 11 '15 at 09:41
  • Thanks for your response Dave. nf-app1 and nf-app2 are single physical instances and I've hit them both enough for their individual hystrix.stream endpoints to return full data and have that successfully display on the hystrix dashboard individually. I can get full data from nf1 & 2 hystrix.streams both before and after I try my Turbine end point which always returns the same type of response as previously shown. – Matt Reynolds Mar 11 '15 at 14:58
  • Maybe the "-" (hyphen) in the cluster name is a problem (URL encoding?)? It works for me but I'm not using any non alpha chars. – Dave Syer Mar 11 '15 at 16:16
  • I tried changing the clusterConfig to `ITEMSERVICE` with the same results. I'm a little confused by the Spring cloud docs as it says: "Value returned from eureka are uppercase, thus we expect this example to work if there is an app registered with Eureka called "customers": turbine: aggregator: clusterConfig: CUSTOMERS appConfig: customers - It looks like the doc seems to be saying the cluster name should match the Eureka name but is that just a suggested convention? – Matt Reynolds Mar 11 '15 at 16:48
  • I'm also confused when I read that. The sample I used to test was this one: https://github.com/spring-cloud-samples/tests/tree/master/turbine. It works for me (the cluster name is the same as the service name), so can you verify that it works for you? – Dave Syer Mar 11 '15 at 17:30
  • I'm not sure how to do that. How does that test project find the sample service? It doesn't seem to have any eureka setup. Do I have to run some of the other sub projects in test? – Matt Reynolds Mar 11 '15 at 17:56
  • There are 2 main classes in the "apps" package. Run those at the same time as the `TurbineApplication`. The `SimpleApplication` has a circuit breaker in the "/" and "/fail" endpoints. – Dave Syer Mar 11 '15 at 17:59
  • Thanks Dave, that test project does work for me so now I have something to compare against. SO is suggesting I move this discussion to chat but I don't have the reputation to do that so I may have to hit you up to do that if I hit more roadblocks. Otherwise I'll hopefully be back soon to post what the issue was... – Matt Reynolds Mar 12 '15 at 00:34
  • They don't actually force you to use chat. If you want to chat about Spring Cloud there's a gitter.im room #spring-cloud. – Dave Syer Mar 12 '15 at 08:00
  • Link to gitter: https://gitter.im/spring-cloud/spring-cloud – Dave Syer Mar 12 '15 at 11:06

1 Answers1

3

The problem was the machine Turbine was running on was not time synced to the machine(s) the Hysterix services were running on. This Answer on a different Turbine question indicates that Turbine will ignore "old" events by default. I tried disabling turbine.InstanceMonitor.eventStream.skipLineLogic in my application.yml and everything started working.

Community
  • 1
  • 1
Matt Reynolds
  • 81
  • 1
  • 7