Glassfish/Payara: why might deployment of a large web app to Glassfish-4.1.1 (or Payara Server 4.1.1.163) take 5 times longer than to Glassfish-4.1

Question

Mac OS X: Yosemite 10.10.5

Until recently I have been using:

NetBeans8.1beta
Glassfish4.1
Mojarra 2.2.7

I have recently investigated using:

NetBeans8.1
Glassfish4.1.1
Mojarra 2.2.12.

With a very large JSF web app (with ObjectDB as JPA persistence provider) I am experiencing massively longer deployment times to Glassfish4.1.1 (over 10 minutes) than to Glassfish4.1 (around 2 minutes). That's about 5 times slower.

[EDIT: 2016-09-13 Tried with Payara 163 Full (Payara Server 4.1.1.163) and it took over 20 minutes, even slower than on Glassfish4.1.1, with nearly identical setup]

I can't possibly reproduce here or provide examples of the large web app.

Q1: Has anybody experienced any similar difference between Glassfish-4.1 and Glassfish-4.1.1 [EDIT: OR Payara Server 4.1.1.163], and where I might I look for the point of difference ?

(I am already trying to investigate it by using the NetBeans Profiler, but it is not turning out to be so easy yet to find the cause.)

[EDIT: 2016-09-15 timing tests]

I have performed careful side-by-side comparison using screencast recording and comparing with timings in the log.

In Payara41 (4.1.1.163) and Glassfish4.1.1 it gets "stuck" as shown below for about 10 minutes, and I have not found any log setting that gives a single line of output here where the problem occurs, and running the server in DEBUG mode also does not seem to give any additional information.

[2016-09-14T23:02:53.450+1000] [Payara 4.1] [INFO] [NCLS-LOGGING-00009] [javax.enterprise.logging] [tid: _ThreadID=19 _ThreadName=RunLevelControllerThread-1473858173343] [timeMillis: 1473858173450] [levelValue: 800] [[
  Running Payara Version: Payara Server  4.1.1.163 #badassfish (build 215)]]

... runs ok through all steps and then through dozens of these EJB JNDI name declarations:

[2016-09-14T23:03:48.823+1000] [Payara 4.1] [INFO] [AS-EJB-00054] [javax.enterprise.ejb.container] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858228823] [levelValue: 800] [[
  Portable JNDI names for EJB BeanName: [java:global/app-name/BeanName!com.example.BeanName, java:global/app-name/BeanName]]]

Gives this WELD version message:

[2016-09-14T23:03:48.875+1000] [Payara 4.1] [INFO] [] [org.jboss.weld.Version] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858228875] [levelValue: 800] [[
  WELD-000900: 2.3.5 (Final)]]

... then takes about 8 seconds before this message ..

[2016-09-14T23:03:56.666+1000] [Payara 4.1] [INFO] [] [org.jboss.weld.Event] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858236666] [levelValue: 800] [[
WELD-000411: Observer method [BackedAnnotatedMethod] private org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider.processAnnotatedType(@Observes ProcessAnnotatedType) receives events for all annotated types. Consider restricting events using @WithAnnotations or a generic type with bounds.]]

... Then take many minutes (typically about 10 minutes) before resuming these otherwise harmless warnings ..

[2016-09-14T23:11:33.387+1000] [Payara 4.1] [INFO] [] [org.jboss.weld.Bootstrap] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858693387] [levelValue: 800] [[
WELD-001125: Illegal bean type java.util.Comparator<javax.persistence.metamodel.EntityType<?>> ignored on [EnhancedAnnotatedTypeImpl] public static  class com.example.ElementManager$EntityTypeSorter]]

[2016-09-14T23:11:59.736+1000] [Payara 4.1] [WARNING] [] [org.glassfish.jersey.internal.Errors] [tid: _ThreadID=74 _ThreadName=admin-thread-pool(2)] [timeMillis: 1473858719736] [levelValue: 900] [[
The following warnings have been detected: WARNING: Parameter 1 of type T from public void com.example.NumberProperty.setDefaultValue(T) is not resolvable to a concrete type.

]]

Similar warning messages happen in the Glassfish4.1 log, but there is no long pause/holdup.

Q2: What else can I do to find out why it is pausing/stalling there for ages ?

Try setting the logging of the server to debug first and check/compare timings — Kukeltje, Feb 29 '16 at 16:00
I don't see anything JSF related in the 'logging/timing' info in your question. — Kukeltje, Sep 15 '16 at 09:22
@Kukeltje At the point where recent timing diagnosis (now included in question) reveals it gets "stuck" or "stalls" JSF is indeed probably not relevant, I will remove the JSF tag. — Webel IT Australia - upvoter, Sep 17 '16 at 09:15

score 1 · Answer 1 · answered Nov 02 '16 at 12:01

Thanks to detailed help from the Payara team I have an answer.

The max heap settings were different between my old Glassfish4.1 install (-Xmx2048m) and the new Glassfish4.1.1 and Payara41 install (-Xmx512m). On building my web app project's database model on startup, it was hitting the 512m heap limit and causing excessive garbage collection (as revealed by JVisualVM), which massively slowed the deployment.

I already had these as run-time VMargs for the NetBeans web app project:

-Xms2048m -Xmx2048m

But on installing NetBeans8.1 (with Glassfish4.1.1), and then Payara41, I had neglected to also match this on the new servers to make room for it. With the increased max heap and fixed minimum the deployment time is now about 2 mins on both Glassfish4.1.1 and Payara41.

Glassfish/Payara: why might deployment of a large web app to Glassfish-4.1.1 (or Payara Server 4.1.1.163) take 5 times longer than to Glassfish-4.1

1 Answers1

Linked