The answer actually belongs to Physics(Collision dynamics) rather than Computer Science.
There are mainly two types of collisions - elastic and inelastic. To simulate a real world scenario, you should implement the inelastic rules. However, they are more complex, and several variables are involved other than the particle mass and velocity. Therefore, you better start with elastic collisions.
Solution(Elastic) : Here two rules always hold - conservation of momentum and conservation of kinetic energy. Say u1,u2 are initial velocities of the balls having mass m1,m2 and their final velocities are V1,V2 :
m1u1 + m2u2 = m1V1 + m2V2
m1u12 + m2u22 = m1V12 + m2V22
Solve the above equation from the known values m1, u1, m2, u2 and you will get V1 and V2.