1

In my project, I need to download a html (about 50K-100K long when read into String, yes, quite fat), and fetch some contents using regular expressions.And then insert them into the database. The performance is quite bad, and I want to know why.

The process of the codes is like that (multithreaded):

  1. using httpcomponents to download the html file into String (String html)
  2. using Regular expressions to fetch the content,and insert (database is mysql)
Pattern p = Pattern.compile("<h.*</a></h.>",Pattern.CASE_INSENSITIVE);
Matcher m = p.matcher(html);
boolean result = m.find();
while (result) {
    //insert into database stuff
    //update database stuff
}

The string is very long, but if I split it into pieces, some matches may be missed. This is quite disturbing.

I added some print lines and find that after inserting into database, there are some delays, before updating operations, but I can't figure it out as the connection to the database isn't closed.

Mat
  • 202,337
  • 40
  • 393
  • 406
Jacob
  • 619
  • 1
  • 5
  • 17
  • 3
    Why do you think you need to use regular expressions to fetch the contents? Have you tried using an HTML parser instead? – Jon Skeet Mar 06 '12 at 11:05
  • 2
    have you tried to commit every 10 or 20 insertion and not at everyone? – rosco Mar 06 '12 at 11:08
  • Also note that your regex matches, for instance, the following: `

    Test

    link then comes `. I.e. it matches until the last occurrence of `` of the page, which might be a much larger string than you intended?     – Christoffer Mar 06 '12 at 11:09
  • @Christoffer:thanks,the regex in the codes is not what I am using,it's just an example. – Jacob Mar 06 '12 at 11:16
  • @JonSkeet:It is not just some tags,I think it may be complicate using html parser. – Jacob Mar 06 '12 at 11:22
  • 3
    @Rusty: If it's genuinely HTML, then parsing it with regular expressions is likely to be more complicated than parsing it with something which knows about the HTML structure. – Jon Skeet Mar 06 '12 at 11:24
  • @rosco:I need the auto generated id. – Jacob Mar 06 '12 at 11:24
  • @JonSkeet:I am looking into it,thanks. – Jacob Mar 06 '12 at 11:34

4 Answers4

2

  1. Try avoid Regex, use standard HTML Parser like JSoup, there are many. They might be more efficient, at least more than Regex, I would hope.

  2. If using regex, try not compiling regex each time. Can have a private static for the Pattern. But this ain't huge gain in performance, just good practice.

  3. Use connection pooling for Database. If possible do batch inserts.

Nishant
  • 54,584
  • 13
  • 112
  • 127
  • Does a private static for the Pattern support multithread?Thanks – Jacob Mar 06 '12 at 11:30
  • If thread safety is concerned, you can have Pattern object as instance variable instead of a class variable. But compiling each time the method is invoked is something I would not recommend. I also assume you do not have one thread per HTML page, rather you have a thread pool that polls, probably, a queue of downloaded HTML or queue of URLs to be processed. – Nishant Mar 06 '12 at 11:48
2

Stop right there.

You are committing one of the worst sins that it's possible to when performance tuning.

You are assuming that the performance problem is where you think it is in the code.

You do not know that, and until you have hard evidence, you could be optimizing the wrong thing - and may well be making the situation worse.

First of all, you need to confirm that the problem is application code. As this is a multithreaded application, which is downloading data (over a network) and inserting into a database (over a network), then you first need to rule out issues to do with thread monitors / locks and network / IO issues.

It's too early to even use a profiler. If you profile now, you could be missing things.

1) If you don't have the GC switches on, put them on now. Production Java applications should never run without GC logging.

2) Rerun your test case, with vmstat 1 running (if it's Unix) or Task Manager (if it's Windows).

3) Update your question with details of whether the CPU utilisation goes to 100% during the test run, and we can do the next step.

kittylyst
  • 5,640
  • 2
  • 23
  • 36
1

Use a profiler, such as VisualVM. It will show you exactly what method is taking up time.

In your case, it's a pretty safe bet that your approach of using a regex is not ideal.

Edit: I disagree it's too early for a profiler. You can monitor your threads, and see if they're waiting for locks. Also, the profiler will show memory statistics and CPU utilization - so you'll know that it is the application. A profiler is the perfect tool to use.

Matthijs Bierman
  • 1,720
  • 9
  • 14
  • Too early for a profiler - we don't know that application code is what's causing the bottleneck yet. – kittylyst Mar 06 '12 at 11:13
  • The profiler will tell you if it isn't. You can profile *everything*, including your libraries. At least it will tell you _what_ is being slow. – Matthijs Bierman Mar 06 '12 at 13:02
  • And suppose the problem is systemic, rather than being tied to any one particular method? E.g. process is swapping due to memory constraints, so all object allocations are slow. What then? A profiler is not a magic bullet. It can, and **will** give you totally wrong answers under some circumstances. – kittylyst Mar 06 '12 at 14:46
  • Won't that be visible as the entire application performing poorly, instead of only a handful of methods? Then you know to look for a systemic cause. Besides, if it really is systemic you will notice a lot more going wrong than just this particular job. – Matthijs Bierman Mar 06 '12 at 14:55
  • Not necessarily. If you took 5 minutes to assess the general health of the system (e.g. by looking at vmstat) then you wouldn't run into these conditions. Developers always want to reach for the profiler because it looks the most like software development. But performance tuning is **not** like development, and a slightly different mindset is needed. – kittylyst Mar 06 '12 at 18:21
1

Using regular expressions to pull values from HTML is always a mistake. HTML syntax is a lot more complex that it may first appear and it's very easy for a page to catch out even a very complex regular expression.

Also Pattern Matching for parsing HTML is always a tedious task..because in regex long strings are divided into groups and sub-groups and then each group and sub-group is matched for Pattern..May be thats why your performance is slow..

Use an HTML Parser instead. See also What are the pros and cons of the leading Java HTML parsers?

Community
  • 1
  • 1
Ramandeep Singh
  • 5,063
  • 3
  • 28
  • 34