If you control your server, meaning that is hosted by you, or you have it with a VPS, then you can test your FFFMPEG and MEMCODER that you need to convert those files to the apropiate format, PHPFOX needs to convert and process your video files. I have messed with it on the command shell prompt before I inserted those parameters on the setup admin panel. After a while I got it working to suit PHPFOX V3. Make sure you enable FLVTOOLS also. Remember, if it does not work on shell prompt on your server, there is no way is going to work on the setup of PHPFOX. The settings I came up with finally are listed below. Another issue, the player used in that version PHPFOX V3, is FLOWPLAYER, it only plays FLV files (flash enabled browsers) and is limited to browsers that uses Flash. Forget about Androids or iPhones, unless you use PUFFIN or any other flash enabled browser. I hope my settings work on your server. It depends really on your server setup and those 3 utilities below. If you have any problems, let me know. Its a tricky setup but its possible to change the player to a HTML5 format so you dont have to mess with Flash FLV files and formats any more.
FLVTOOLS: -U {destination}
FFMPEG : -i {source} -ar 44100 -vcodec libx264 {destination}
MENCODER: {source} -o {destination} -of lavf -oac mp3lame -lameopts abr:br=56 -ovc lavc -lavcopts vcodec=flv:vbitrate=800:mbd=2:mv0:trell:v4mv:last_pred=3 -vf scale={width}:{height}
Another aproach and could solve your problem without any major server setup, and if you are lucky, you may be able to include that format (or any other) in the list of available formats for convertion, on my version I had to modify to accept celular formats such as '3gpp' => 'video/mp4'... good luck!!
The file you need to modify is located here:
//yourdomain/module/video/include/service/video.class.php
class Video_Service_Video extends Phpfox_Service
{
private $_aExt = array(
'mpg' => 'video/mpeg',
'mpeg' => 'video/mpeg',
'wmv' => 'video/x-ms-wmv', // Here you add after wmv like this: (it may work)
'wma' => 'video/x-ms-wmv',
'avi' => 'video/avi',
'mov' => 'video/quicktime',
'flv' => 'video/x-flv',
'mp4' => 'video/mp4',
'3gp' => 'video/3gpp'
);