I'll assume that you actually mean the following as the new doctype:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Functionally, your JSP/HTML will not break. The modern webbrowser is forgiving enough and it'll close open tags where needed and it'll accept unspecified attributes. In the meanwhile you should however really consider to fix them anyway so that it w3-validates against the new doctype as much as possible.
However, major problems may arise in look'n'feel as specified by CSS. Your old doctype pushes the the browser in quirksmode which reveals among others the pretty serious box model bug in MSIE. With this bug, MSIE accounts the element's padding
and border
into element's width
. So if you change this doctype, you'll only see changes in MSIE.
If your website was designed on normal webbrowsers according the web standards (i.e., it's been designed for Firefox, Chrome, Safari, etc), then you don't need to worry. It'll only going to look like in MSIE as intented, which is just an advantage.
But if your website was designed for MSIE and hereby thus ignoring the webstandards, then you'll have to make a lot of changes in CSS (mainly width/padding/border/margin) so that it look the same in MSIE as intented. You'll also gain the additional advantage that it will now look properly on normal browsers.
As to the doctype choice, well this is going to be subjective, but if you need my opinion, then read the following answers: