I dont have 50 reputation to ask you more details in a comment, so i will answer directly hoping this is what you're asking about.
I think you mean that your problem is that the content of the ModalPanel is not rerendered dynamically. but for this you can wrap your table (or the components you want to update) in an <a4j:outputPanel>
with ajaxRendered="true"
Setting ajaxRendered="true"
will cause the <a4j:outputPanel>
to be updated with each Ajax response for the page, even when not listed explicitly by the requesting component. This can in turn be overridden by specific attributes on any requesting components.
http://docs.jboss.org/richfaces/latest_4_0_X/Component_Reference/en-US/html/chap-Component_Reference-Containers.html#sect-Component_Reference-Containers-a4joutputPanel
In my code I have something like :
<a4j:commandLink id="timelineBtn" action="#{timelineBean.doGenerateLog}"
oncomplete="Richfaces.showModalPanel('timelinePnl');return false;"
value="#{labels['timeline.button.lbl']}"/>
that opens the modalPanel :
<rich:modalPanel style="width:800;height:600;" id="timelinePnl">
<a4j:outputPanel id="dataPanel" ajaxRendered="true">
<!-- your components to be updated go here -->
</a4j:outputPanel>
</rich:modalPanel>
So my content is updated with the results of my timelineBean.doGenerateLog
method each time i open the modalPanel .