So, basically you want to get intersection points of two circles:
- The big one (BluePoint, radius = R)
- A small one (GreenPoint, radius = D)

(Please excuse my amazing drawing skills :P)
I've at first tried to solve it myself, and fruitlessly wasted several sheets of paper.
Then I started googling and found an algorithm in other question.
Here is my Java implementation
double[][] getCircleIntersection(
double x0, double y0, double r0,
double x1, double y1, double r1) {
// dist of centers
double d = sqrt(sq(x0 - x1) + sq(y0 - y1));
if (d > r0 + r1) return null; // no intersection
if (d < abs(r0 - r1)) return null; // contained inside
double a = (sq(r0) - sq(r1) + sq(d)) / (2 * d);
double h = sqrt(sq(r0) - sq(a));
// point P2
double x2 = x0 + a * (x1 - x0) / d;
double y2 = y0 + a * (y1 - y0) / d;
// solution A
double x3_A = x2 + h * (y1 - y0) / d;
double y3_A = y2 - h * (x1 - x0) / d;
// solution B
double x3_B = x2 - h * (y1 - y0) / d;
double y3_B = y2 + h * (x1 - x0) / d;
return new double[][] {
{ x3_A, y3_A },
{ x3_B, y3_B }
};
}
// helper functions
double sq(double val) {
return Math.pow(val, 2);
}
double sqrt(double val) {
return Math.sqrt(val);
}
double abs(double val) {
return Math.abs(val);
}
This is how you would use it for the question situation:
double centerX = 0;
double centerY = 0;
double radius = 5;
double pointX = 10;
double pointY = 0;
double newPointDist = 5;
double[][] points = getCircleIntersection(centerX, centerY, radius, pointX, pointY, newPointDist);
System.out.println("A = [" + points[0][0] + " , " + points[0][3] + "]");
System.out.println("B = [" + points[1][0] + " , " + points[1][4] + "]");