I have found that the built in pagination feature of the Primefaces data table is one of the best features and did a good amount of load testing on it, bringing in recordsets with over 30,000 Hibernate entities and found the performance to be lackluster. This of course means that you will have 30,000 entities in session so I have the following in my web.xml
to help by storing session on the server side.
<context-param>
<description>State saving method: 'client' or 'server' (=default). See JSF Specification 2.5.2</description>
<param-name>javax.faces.STATE_SAVING_METHOD</param-name>
<param-value>server</param-value>
</context-param>
This will reduce the size of the ViewState allowing request/response size to be greatly reduced however server side memory can suffer enormously by doing this.
Another potential option in some JSF implementations to help mitigate the size of ViewStat or session memory usage is compression. The following link describes a number of SUN RI and MyFaces JSF configuration parameters that can be set, some of which give the option of compression of the session state. http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Frweb_jsfengine.html
As far as learning more about how the Primefaces DataTable pagination feature works, why not go straight to the source? Primefaces is after all an open source project, so just look at the code and see what you can learn: http://code.google.com/p/primefaces/source/browse/#svn%2Fprimefaces