How to solve Vertx blocking DNS issue

Question

TL;DR: is it possible to overcome blocking DNS issues with Vertx? Preferably with Vertx HttpClient?

I'm trying to use Vertx to scrape some URLs with different hosts. I'm currently using Vertx HttpClient to do so, I've recently encountered an issue with a blocking DNS call which sometimes blocks my HttpClient requests (and therefore also blocking the event loop). This issue is described here.

I can't use a blocking http client in an "executeBlocking" function since I went for the Vertx solution because it is asynchronous and should be able to handle the loads I'm expecting. Also I can't resolve the IP address myself because it only solves it for a specific host while I'm trying to fetch many different hosts

So my question is: is there any (relatively) elegant solution to this issue that doesn't require me to wait for future Netty versions? Preferably something that uses the native Vertx HttpClient.

Thanks in advance

score 3 · Answer 1 · answered Dec 10 '18 at 20:36

3

Other option is to disable dns resolver by adding -Dvertx.disableDnsResolver to commandline.

answered Dec 10 '18 at 20:36

Ruslan Akhundov

2,178
4
20
38

@ElliotVargas it works for me. Do you have any particular consern regarding that approach? – Ruslan Akhundov Aug 13 '20 at 19:23
No, only wanted to confirm this solution works. Thanks – Elliot Vargas Aug 17 '20 at 16:40

score 1 · Accepted Answer · answered Mar 01 '16 at 14:46

Ok so after digging about this issue for several days I found several possible solutions to this issue. I hope this might help someone

These are the possible solutions

Programmatically solving the DNS issue using an executeBlock - this requires first "fixing" the DNS issue using vertx.executeBlocking with a future handler and only when the handler is invoked invoking the actual HTTP client code. Some possible optimizations here might be:
- Saving an in memory map of all the hosts that the DNS already resolved and if a URL with a resolved host arrives just scrap it asynchronously.
- Setting ordered to false in the executeBlocking statement (if you really don't care about ordering) as it can improve the performance of that code
Waiting for Netty 4.1 to come out. I don't know exactly when it's going to happen but I obviously could not wait for it
Do nothing about blocking the event loop. This means the program will be sluggish until the DNS server starts getting cache hits. If you can accept some sluggishness, that just might be a good fit

All of these solutions are from the Vertx mailing list. You can see the full correspondence here (including some code examples)

Currently I'm running the program in a cluster and I didn't encounter this issue on it. If I will later need to solve this I will probably aim for the 1st solution (unless Netty 4.1 is released by then)

I hope I got all the solutions right, if someone sees this and has better ideas or corrections I'll highly appreciate it

Note: We still have the issue with Netty 4.1.1 and 4.1.5. – рüффп Oct 28 '16 at 16:04 — рüффп, Oct 28 '16 at 16:04

How to solve Vertx blocking DNS issue

2 Answers2

Linked