Is it possible to manually cache an entire web page?

Question

I need to cache a web page and then for future requests, check the cache (using url as key) and if found, return the web page from the cache instead of making a request.

I'm using Smiley's ProxyServlet and the method where the servlet writes to the OutputStream seems perfect for caching. I've added just two lines of code:

/**
 * Copy response body data (the entity) from the proxy to the servlet client.
 * TODO: CACHE entity here for retrieval in filter
 */
protected void copyResponseEntity( HttpResponse proxyResponse, HttpServletResponse servletResponse,
        HttpRequest proxyRequest, HttpServletRequest servletRequest ) throws IOException
{
    HttpEntity entity = proxyResponse.getEntity();
    if ( entity != null )
    {
        String key =  getCurrentUrlFromRequest( servletRequest );  // 1
        basicCache.getCache().put( key, proxyResponse.getEntity() ); // 2
        OutputStream servletOutputStream = servletResponse.getOutputStream();
        entity.writeTo( servletOutputStream );
    }
}

and it kinda works, it does store the HttpEntity in the cache. But when I go back to a browser and request the same url once again, when the code gets back in my Filter, I obtain the HttpEntity using the url as key, and I write it to the response, but I get a "Stream closed" error:

java.io.IOException: Stream closed
    at java.base/java.util.zip.GZIPInputStream.ensureOpen(GZIPInputStream.java:63) ~[na:na]
    at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:114) ~[na:na]
    at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107) ~[na:na]
    at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:64) ~[httpclient-4.5.9.jar:4.5.9]
    at org.apache.http.client.entity.DecompressingEntity.writeTo(DecompressingEntity.java:93) ~[httpclient-4.5.9.jar:4.5.9]
    at com.myapp.test.foo.filters.TestFilter.doFilter(TestFilter.java:37) ~[classes/:na]

Here's the filter:

@Component
@WebFilter( urlPatterns = "/proxytest", description = "a filter for test servlet", initParams = {
        @WebInitParam( name = "msg", value = "==> " ) }, filterName = "test filter" )
public class TestFilter implements Filter
{

    private FilterConfig filterConfig;

    @Autowired BasicCache basicCache;


    @Override
    public void doFilter( ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain )
            throws IOException, ServletException
    {
        String url = getCurrentUrlFromRequest( servletRequest ); // 1
        HttpEntity page = (HttpEntity) basicCache.getCache().get( url ); //2
        if ( null != page )  // 3
        {
            OutputStream servletOutputStream = servletResponse.getOutputStream();  // 4
            page.writeTo( servletOutputStream );  // 5 stream closed :(
        }
        else
        {
            filterChain.doFilter( servletRequest, servletResponse );
        }

    }


    public String getCurrentUrlFromRequest( ServletRequest request )
    {
        if ( !( request instanceof HttpServletRequest ) ) return null;

        return getCurrentUrlFromRequest( (HttpServletRequest) request );
    }

    public String getCurrentUrlFromRequest( HttpServletRequest request )
    {
        StringBuffer requestURL = request.getRequestURL();
        String queryString = request.getQueryString();

        if ( queryString == null ) return requestURL.toString();

        return requestURL.append( '?' ).append( queryString ).toString();
    }

    @Override
    public void destroy()
    {
    }

    @Override
    public void init( FilterConfig filterConfig ) throws ServletException
    {
        this.filterConfig = filterConfig;
    }

}

oh, and the BasicCache class just in case:

@Component
public class BasicCache
{

    private UserManagedCache<String, HttpEntity> userManagedCache;

    public BasicCache()
    { 
        userManagedCache = UserManagedCacheBuilder.newUserManagedCacheBuilder( String.class, HttpEntity.class )
                .build( true );
    }

    public UserManagedCache getCache()
    {
        return userManagedCache;
    }

    public void destroy()
    {
        if ( null != userManagedCache )
        {
            userManagedCache.close();
        }
    }
}

I am stuck with this very localized / manual / whatever you want to call it kind of caching -- I can't use the obvious "just hook up ehcache / redis / whatever and let it do it's thing". So while I know those fine caches can cache entire webpages, I don't know if they allow me to work in this admittedly unusual way.

So I'm hoping SO can show me how to get this done. I first tried just wiring in a ConcurrentHashMap for my basic cache but that didn't work either, so I thought I'd see if I could tap into whatever magic the big caching guns have, but so far I can't.

Thanks for any help!

`HttpEntity` is just a handle for an io stream that is currently established with your source, you can only read once from it. To cache the result you need to copy the stream contents into your own data structures. E.g. a byte array. — cruftex, Aug 19 '19 at 08:15

score 1 · Answer 1 · answered Aug 19 '19 at 03:58

1

In the TestFilter class, can you put a breakpoint on this line and debug?

HttpEntity page = (HttpEntity) basicCache.getCache().get( url );

The underlying stream is likely not in a state for you to actually do:

page.writeTo( servletOutputStream );

You can perhaps rewrite according to the reference example here. Essentially, declare a PrintWriter that you control the state, take the content from the cache, write into the response, and close the writer.

answered Aug 19 '19 at 03:58

mOchi

166
1
6

here's the state of the HttpEntity variable called page on that line of code after the second call from the browser (the first call it's obviously null since it hasn't yet been cached): https://i.imgur.com/KgMloa9.png I hope you can glean something from it, thanks for looking! – user2659207 Aug 19 '19 at 15:00

score 1 · Answer 2 · answered Aug 19 '19 at 05:09

1

As already had been said here Should one call .close() on HttpServletResponse.getOutputStream()/.getWriter()?

You'd better to implement a wrapper for your servlet , details in the article below.

https://www.oracle.com/technetwork/java/filters-137243.html#72674

answered Aug 19 '19 at 05:09

wtsiamruk

339
3
16

thanks for your response. After reading your links, I understand I can wrap the servlet response, but I'm missing a final step. If I cache the HttpServletResponseWrapper in my proxy servlet, then on the next request from the browser, when my filter gets the HttpServletResponseWrapper from the cache, how can I push that out the door and break the filter chain? – user2659207 Aug 19 '19 at 16:32

score 0 · Answer 3 · answered Aug 19 '19 at 18:45

I was wondering why all these great answers agree on wrapping the HttpServletResponse and I couldn't get it to work -- I feel like an idiot. It wasn't working because there's nothing in the response to wrap.

The page content is all in the HttpEntity from way up in the beginning: HttpEntity entity = proxyResponse.getEntity();

After I realized chasing the servlet responses / requests wasn't the answer, I got lucky and found: org.apache.http.entity.BufferedHttpEntity

which does what all the helpful SO posters above have been telling me to do (wraps an HttpEntity so you can repeatably get the content), but does it on the correct object.

So the very first method up top just gets a slight tweak and it still proxies just fine:

    if ( entity != null )
    {
        String key =  getCurrentUrlFromRequest( servletRequest );
        OutputStream servletOutputStream = servletResponse.getOutputStream();
        BufferedHttpEntity wrapper = new BufferedHttpEntity( entity );
        basicCache.getCache().put( key, wrapper );
        wrapper.writeTo( servletOutputStream );
    }

change the BasicCache to expect a String and a BufferedHttpEntity, and then for subsequent requests, in the filter grab the BufferedHttpEntity out of the cache and the line that does all the work is the same as the last line above:

    if ( null != page )
    {           
        OutputStream servletOutputStream = servletResponse.getOutputStream();
        page.writeTo( servletOutputStream );  // bingo
    }
    else
    {
        filterChain.doFilter( servletRequest, servletResponse );
    }

Thanks to everyone for your help!

glad that we helped you, you can choose most relevant answer to your question of course :) — wtsiamruk, Aug 20 '19 at 09:24

Is it possible to manually cache an entire web page?

3 Answers3