After checking around for problems similar to yours and looking at some of my silverlight projects I would suggest the following:
As you stated you are using a custom .proj file, you should check the custom .prog file and make sure this exists in a <PropertyGroup/>
<SilverlightApplicationList>{GUID}|..\Silverlight_Project_Folder\Silverlight_Project_Name.csproj|ClientBin|False</SilverlightApplicationList>
This value tells MSBuild to copy the XAP in the ClientBin directory.
Take note of the silverlight projects GUID. It can be obtained from the Silverlight app .csproj file:
<ProjectGuid>{00000000-0000-0000-0000-000000000000}</ProjectGuid>
There was also a mention of including the CopySilverlightApplications; task on the build server's Microsoft.WebApplications.targets
Hope this helps
References:
New Silverlight app doesn't generate XAP
How to get the Silverlight XAP copied to the clientbin on build
Silverlight xap file not being copied to ClientBin on Build Server
MSBuild SilverlightApplicationList