Try the following:
1- Have listener detecting when a phone call is happening.
http://bitgriff.blogspot.com/2013/01/detecting-incoming-and-outgoing-phone.html
2-Then according the the caller, you can set the ringer mode from silent to vibrate or a ringtone.
AudioManager audioManager = (AudioManager) getSystemService(AUDIO_SERVICE);
audioManager.setRingerMode(AudioManager.RINGER_MODE_NORMAL);
Check AudioManager documentation for other RingerMode
Link: http://developer.android.com/reference/android/media/AudioManager.html
**Changing Ringtone, not mode:
Basically what you have to do is the following:
1- Get the contact lookup Uri
2- Get the Uri of the ringtone that you want to set
3- Use ContentValues in order to update the ringtone
Check this question in order to achieve a custom ringtone for a contact: Set as Contact Ringtone? Android
This is another good reference:
Setting contact custom ringtone, how?