After several days' struggling, I am pretty sure here is the right answer. Hope this can help others in case they run into the same issue.
To make a mp4 be able to play with html5 video in a web browser, first you should refer to this post : HTML5 - How to stream large .mp4 files?
In my case, when I "Ensure that the moov (metadata) is before the mdat (audio/video data)", the desktop and android browsers can work, but safari won't. Following the clues "Ensure that your web server is configured to serve byte range requests." given in the same post mentioned above, we finally found the root cause comes from a bug of wildfly (https://developer.jboss.org/thread/249279?start=0&tstart=0). Wildfly (at least 8.0) doesn't handle the byte-range request correctly that leads the safari can not play mp4 well. Our final solution is to change the app server from wildfly to apache httpd and work as expected.
That's that.