There are two separate questions asked in the post, I will answer them accordingly:
1) I would recommend using a Template Engine which integrates with JSPs.
Apache Tiles fits in this category. Using this framework, you can define template which contains common sections for header, footer, body and menu layouts. Furthermore, it supports derived templates via inheritance.
2) Caching solution : Since you want to load values from database to make the menu dynamic, caching and pre-populating the menu will minimize page load times. This should be done on server side.
A separate thread/process should be made responsible for pre-populating and refreshing cache periodically.
From clean code perspective, JSP or Tiles template should not invoke database calls. Instead the page should interact with a DAO/component to separate concerns and ensure MVC pattern is not broken.
Helpful links:
Tiles Example
Thymeleaf - an alternative to Tiles