This is normal behavior and ordinarily I'd just live with it. As long as you've set the expiration or max-age settings properly then the 304 messages will only happen upon refresh, as you say. And as long as your users aren't hitting refresh regularly, then the 304 will not typically occur as people browse the site.
However, there are some situations in which you can prevent the 304 message even when the user refreshed the page.
- Setting the background-image of an element through Javascript.
- Using an iframe to pull content that has a max-age (note that they can still refresh the iframe by right-clicking inside it and clicking refresh).
Here's a background-image example if you want to try it out...
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<div id="example-image" style="height: 250px;"></div>
<script type="text/javascript">
$("#example-image").css("background", "white url(/images/myImage.jpg) no-repeat");
</script>
UPDATE
I was thinking AJAX GETs would work but it appears I was wrong so I've removed that from the list. In any case, the solutions I've listed above do work but they are clunky and unpleasant which is why I normally don't bother with this.
UPDATE #2
Steve Souders has some interesting experiments regarding this issue...
http://www.stevesouders.com/blog/2013/02/26/reloading-post-onload-resources/