2

I currently have a OSB project with a set of 21 modules that take roughly 4 minutes to build on my local 2 core/12GB ram laptop running Windows using no threading, just a simple build install. It takes 10-20 seconds per module.

When building this exact same project on my CI server running on Ubuntu, with 8 cores/16GB RAM build time is closer to 110 minutes, using around 4 minutes per module.

Some details on the Linux build:

  • Most of these 4 minutes per module is spent sitting idle on 0% CPU utilization.
  • MAVEN_OPTS are "-Xmx512m -Xms512m"
  • Same build time on Java 7 and 8
  • When running with the -X flag it spends most of it's time at "-- end configuration --"

I have tried increasing the file descriptor limit, thinking this was the problem. This did not do anything to the build time.

Jon Heller
  • 34,999
  • 6
  • 74
  • 132
karl
  • 1,766
  • 2
  • 13
  • 24
  • How do you build your OSB projects? Offline OEPE, or com.bea.alsb.tools.configjar.ConfigJar? – Trent Bartlem Oct 13 '15 at 03:16
  • @TrentBartlem I am using the oracle-service-bus plugin for maven. The POM points to a local installation of the Middleware SOA Suite. – karl Oct 13 '15 at 08:27

2 Answers2

4

After profiling maven with VisualVM both on Windows and Linux I found that on Linux it spent abnormal amounts of time generating a random seed.

enter image description here

So by changing to (the slighty less secure) /dev/./urandom build time went from 110 minutes down to 1minute 47seconds.

An example of how to do this is by passing in the setting as a flag:

-Djava.security.egd=file:/dev/./urandom

If you would like to set this permanently, this can be done in the file jdk1.7.0_75/jre/lib/security/java.security by changing:

securerandom.source=file:/dev/urandom to
securerandom.source=file:/dev/./urandom.

This might bring some security implications of which you should do some research first if you need to do this.

karl
  • 1,766
  • 2
  • 13
  • 24
3

There are a lot of variables here. I can't provide an answer, but in general, I try to pare the problem down as small as possible. You're saying it's around ~21 projects. Is it equally slow with 1? I know you said you get it with 4min/module, but that's not the same as a project with 1 module in it. The sheer scope of the file descriptors (ulimit) can be really troublesome, even if you're only looking at one module at a time during the build.

Second, ensure your own laptop's environment variables are similar. Windows to Linux is not exactly the easiest to compare to, but you should be able to determine if JAVA_OPTS, MAVEN_OPTS, the various -X/-D flags are the same, whether -Xms/-Xmx are set the same, etc. etc.

Further, have you reviewed any of the Google-able results I found?

The difficult part with your problem is we're not looking at even remotely close environments. We don't know

  1. environment variables
  2. settings.xml (and /etc/.../.settings.xml)
  3. Is the CI server software running the build or are you just running the same mvn clean install on both your local machine and the remote?
  4. etc. etc.

And I can't say that this site would even be the best place to have someone troubleshoot. If you're building an OSB set of projects, you might have better luck filing an SR with Oracle Support and asking them to help you out, per your support plan. At least in the SR, there's a bit more back-and-forth in the communication. Here, you're expected to provide all the possible information, and then people spitball the answer to you. Without any data on your question, we have nowhere to go, and wild assumptions/guesses to make.

Community
  • 1
  • 1
Nick Klauer
  • 5,893
  • 5
  • 42
  • 68
  • Thanks for the answer. Yes it's as slow per module when running with only one. I've carefully compared all environmental variables and maven configuration, and they are all as similar as I can get them between platforms. While testing I'm SSH-ing into the CI box and just manually running "mvn clean install", so that is similar as well. I've Googled anything I can think of, but no luck so far. But I think you are right, StackOverflow is not the correct platform for this issue, and I will contact Oracle. Thanks for your time. – karl Oct 13 '15 at 07:56
  • 2
    You're welcome. One thing to add, if you find the answer, feel free to provide it as an answer to your own question. Anything less and you're leaving us all hanging on edge... ;) – Nick Klauer Oct 13 '15 at 11:17
  • 1
    I have found the solution and thought you might be interested in the answer I just posted. :) – karl Oct 14 '15 at 07:33
  • 2
    That's actually pretty interesting. I know that SOA / OSB servers will take longer to boot depending on your choice of `/dev/random` vs `/dev/urandom`, but that's certainly something worth looking in to. Not to add too much more to this, but I wonder if it's because you're running Ubuntu versus an Oracle Linux 6/7 server, and whether that would be a non-issue on those servers. I would think that if it was slower everywhere, Oracle would really want to fix that to make their "blessed" operating system performant out-of-the-box – Nick Klauer Oct 14 '15 at 15:58
  • From what I have gathered, `/dev/urandom` uses `/dev/random` in Java. By using `.` you "trick" Java into using urandom (as explained [here](http://www.freelists.org/post/oracle-l/Difference-between-devurandom-and-devurandom-Was-swingbench-connection-issue,1)) and thus improves performance. My colleague also mentions that he has seen this been mentioned in miscellaneous Oracle documentation. The reason is slow however is no mystery, `/dev/random` has been notoriously slow on low activity machines, especially VMs. The issue with using `/dev/random` is that it blocks until the entropy pool fills. – karl Oct 15 '15 at 06:47