I use Facebook Connect within a popup and it works pretty well.
1) "Fake" the Facebook-Login-Button
Take a Screenshot of the Button, wrap it's img-tag with an a-tag with attributes
href="<?= $facebook->getLoginUrl(...); ?>"
onclick="facebookPopup(this.href); return false"
2) Create Javascript Popup Function
function facebookPopup (url) {
popup = window.open(url, "facebook_popup",
"width=620,height=400,status=no,scrollbars=no,resizable=no");
popup.focus();
}
3) Create FB Connect "Landingpage"
Create a php-File that will handle the user's data when he finished the connect dialog, for example save his username and facebook-uid if he's new, or just log him in if his ID was already known.
Then close the Popup and Refresh your main page by Javascript:
function CloseAndRefresh()
{
window.opener.location.href = window.opener.location.href;
window.close();
}
You may trigger that function by <body onload='CloseAndRefresh()'>
Remember to specify your landingpage's URL in the Login-Url (redirect_uri).
Also, if you specify 'display'=>'popup' in the login URL, it will show a condensed version of the "Permission Request"-Dialog.