Ok, I just want to give a quick overview of the way I ended up solving this.
First thing to notice is that the file is uploaded before any validators are run. It is placed in a directory named xspupload in the Domino data library - and the file gets an "obscure" name with no file suffix.
However, I can grap this file (I have the file upload control mapped to an object of type com.ibm.xsp.component.UIFileuploadEx.UploadedFile in a managed bean). When a value is set for this object I can get the uploaded file by calling getUploadedFile() of the control, which will return an object of type: com.ibm.xsp.http.UploadedFile. This file object has a getClientFileName() and getServerFile() that can be used to find out what file the user uploaded and temp. rename it to another name. If you do not rename it then it will automatically be removed when the http request is done. So you have to remember to rename (or remove) the file once you are done (I found some inspiration in this: http://www.bleedyellow.com/blogs/m.leusink/entry/processing_files_uploaded_to_an_xpage?lang=en).
I then remeber the temp. filename of the file in a simple hidden input field bound to a string value in my bean. This way the page knows it has been uploaded and I can show the name to the user (by hiding the normal file upload control and provide my own icon that will show any "pending name" and "click" the native control when clicked on).
This way I can get a hold on the file and keep it - and validate it - together with the other fields. Even in a repeat control where I do it.
But, alas, no simple "out of the box" solution. You will have to code it yourself (or use a client centric approach as suggested by Sven).
Thanks for your input.