3

I have two servers running on Glassfish 2.1 both have the same web app.

Two times this error occurred: Some jsp pages stop displaying only showing a blank page, and the following errors are printed in the logs...

PWC1231: Servlet.service() for servlet jsp threw exception java.io.FileNotFoundException: /path/to/jsp/file/jsp_file.jsp.java (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:179) at java.io.FileOutputStream.(FileOutputStream.java:70) at org.apache.jasper.compiler.AntJavaCompiler.getJavaWriter(AntJavaCompiler.java:213) at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:173) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:409) at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:344) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:366) at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056) at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:261) at org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388) at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:316) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415) at javax.servlet.http.HttpServlet.service(HttpServlet.java:738) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at com.my.app.filtro.FiltroCallcenter.doFilter(FiltroCallcenter.java:90) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

Followed by this:

PWC6344: Unable to create output writer for file /path/to/jsp/file/jsp_file.jsp.java|#]

Sometimes only the PWC6344 error is printed and sometimes both, the PWC1231 error is always followed by PWC6344 (which somewhat makes sense, because that exception is thrown when an IOException occurs).

Well, both times this errors happened, the only thing I did was stop and start the instance, and everything was alright again, also this error has only occurred in one of the servers.

Whats is happening ?... Or how can I diagnose what is causing this so I can fix the problem instead of stopping and re starting for eternity...

UPDATES:

I was looking into the possibility of this been a file descriptor problem, as suggested by sbridges but!, the maximum number of file handlers is 811975 with 4520 opened files in one server and 359532 with only 6894 in the other.

So, I guess its safe to say this is not the problem!

Does someone have another theory?

jsedano
  • 4,088
  • 2
  • 20
  • 31
  • What OS are you running? Maybe the file/folder permissions or ownership settings make it fail... – Powerslave Apr 17 '13 at 14:30
  • One is Red Hat and the other one is CentOS, but how can the permission or ownership can change randomly?, this instances are being restarted on daily basis, and is not everyday that this errors appear, and they go away with a restart. – jsedano Apr 17 '13 at 14:56
  • '...jsp.java (Permission denied) at java.io.FileOutputStream.open(Native Method)': Jasper JSP compiler tries to compile your JSP file, makes OS native call to open file, which returns error Permission denied. 'guess its safe to say this is not the problem' - unfortunately no; unprivileged users have a lesser number of handles available, e.g. 1024 set in /etc/security/limits.conf. See http://prefetch.net/blog/index.php/2009/07/31/increasing-the-number-of-available-file-descriptors-on-centos-and-fedora-linux-servers/ and https://bugzilla.redhat.com/show_bug.cgi?id=702670 – Glen Best Apr 18 '13 at 01:42

3 Answers3

1

It looks like the permissions are set incorrectly and you can't write the compiled jsp page to disk,

/path/to/jsp/file/jsp_file.jsp.java (Permission denied)

Are the permissions correct on that directory/file?

sbridges
  • 24,960
  • 4
  • 64
  • 71
  • Yes, remember this error comes and goes, the instances are re started every day in the morning, and sometimes some jsp pages of just one of the two servers starts throwing the errors, and we just stop and start the instance and everything works fine again – jsedano Apr 16 '13 at 14:34
  • nothing is changing the permissions on those dirs? FileNotFoundException can also be caused by running out of open file handles, if you are using linux, you can use lsof to see what file handles are open – sbridges Apr 16 '13 at 22:31
  • How can I replicate this?? I mean "running out of open file handles" to test this on development environment? Also, the same server has other instance with other application where the errors in the question, never happens, could the file handlers be running out in just one instance of the server? Thank you!, And yes, I'm running on Linux – jsedano Apr 16 '13 at 22:41
  • you will probably need to debug on the production server, try doing lsof | wc -l periodically on the server which is having problems, saving the ouput to a file, and see if the number of open files keeps going up. If it is, see what files are being leaked. – sbridges Apr 16 '13 at 23:08
  • Productions servers are kept by another department, I'm gonna ask them to run the following commands on both servers: cat /proc/sys/fs/file-max and lsof | wc -l and post back the results, I'm currently researching file descriptor leaking, Its a starting point! thanks!! http://docs.oracle.com/cd/E19879-01/820-4343/abejj/index.html I'm gonna post back the results, probably update my question – jsedano Apr 16 '13 at 23:28
  • After all it was the file permissions!! turns out in one server another area had one app that would monitor ours, but they needed file permissions, so every time this another area would change the permissions the error would appear!!, and then when we call asking to re start the server, they would do it, but as root! so the permissions were not an issue... – jsedano Apr 18 '13 at 02:55
1

Rename jsp_file.jsp.java to jsp_file.jsp.

Bishan
  • 15,211
  • 52
  • 164
  • 258
  • The file is named jsp_file.jsp, but the error is displaying the name with .java at the end, I don't know why either – jsedano Apr 16 '13 at 14:31
  • First go to your project folder and delete all files in **build** and **dist** directories. Then rename `jsp_file.jsp.java` to `jsp_file.jsp`. now **Clean and Build** your project. Then undeploy old version in `glassfish` server and deploy newly created `war` file to `glassfish` server. – Bishan Apr 16 '13 at 14:41
  • Every time we create a war for production we do that exact same procedure. – jsedano Apr 16 '13 at 15:28
  • Ignore this line of analysis. Don't rename the file. The web container compiles JSP files into servlets, adding the .java extension. At runtime, there are no JSP files - only compiled servlet classes. – Glen Best Apr 18 '13 at 01:15
1

PWC6344: Unable to create output writer for file /path/to/jsp/file/jsp_file.jsp.java|#]

This can also happen if the underlying operating system has run out of file handlers/descriptors which are required in order to open a file for reading or writing. I'm not closely familiar with CentOS, but Google hints that it has a "relatively low" limit of 1024 given the lot of problems related to it in the search results. Among the results you'll see a lot of questions/answers as to how to increase it, such as the following blog:

Increase the number of file descriptors on Centos and Fedora Linux

Raising the number of file descriptors for a regular user on CentOS/Fedora/Redhat is surprisingly difficult to learn how to do. There are lots of incomplete walk throughs on the web, some with typos and other problems.

Here are the steps that worked for me to raise the open file descriptor limit from 1024 (the default) to 65535:

  1. As root, edit /etc/sysctl.conf and add the line:

     fs.file-max = 512000

  2. At the bash prompt, run:

     $ sysctl -p

    That will cause the settings to take effect. You can also cat 512000 > /proc/sys/fs/file-max but that may reset on reboot.

  3. Edit /etc/security/limits.conf and add the following:

     * - nofile 65535

    See the inline comments for more details on what that does and how to make it more restrictive if you prefer.

  4. As root, run

     $ ulimit -n 65535

    and make sure you have no errors. To double check, run ulimit -n and make sure the response is 65535.

  5. Ensure that PAM authentication is turned on for SSH, or else when you try to connect as a regular user, you won’t see the new limits. Edit /etc/ssh/sshd_config and make sure you have:

     UsePAM yes

    Restart SSH /sbin/service sshd restart if you made any changes.

  6. Login as a regular user with a new SSH session & shell and run:

     $ ulimit -n 65535

    Run ulimit -n again to check and good luck!

Community
  • 1
  • 1
BalusC
  • 1,082,665
  • 372
  • 3,610
  • 3,555
  • Thank you, I asked to the department in charge of the production servers to run this commands cat /proc/sys/fs/file-max and lsof | wc -l, I am waiting for the results. – jsedano Apr 17 '13 at 15:56