It seems you want a validation of file size on the client side i.e. using Alloy UI javascript, so I am sorry to bring to your notice that this is not possible in pre-HTML5 browsers due to some security reasons but this is possible with browsers which support the File-API in HTML5. Here is a good-demo site for the file-API.
So for now I would suggest that you do this server-side as a fallback mechanism & there are different plugins available to help you do this using Ajax or just go with plain server-side validation.
You can also try using flash with alloy-ui to check file-size as liferay does in its Documents & Media portlet. The drawback with this approach is that it requires flash to be installed in your browser.
Here is a SO answer which may help you give more information about client-side validation alongwith a demo.
Hope this would give you some lead.