Heavily edited due to question being corrected
Consider the function f(x,y) which gives the number of floors you can test with a limit of x throws and y deaths. If the first throw results in a death, you have x-1 throws and y-1 deaths remaining, so you can check f(x-1, y-1) floors. If the first throw doesn't result in a death, you have x-1 throws and y deaths remaining, so you can check f(x-1,y) floors above the one you threw from. Therefore we have the recurrence f(x,y) = f(x-1, y-1) + f(x-1, y) + 1. The base conditions are that f(x,0) = 0 (because if you make even one throw you risk a death, so you can make no throws and can't even check the first floor), and f(1,x) = 1 where x>0 (you have to make the only throw from the first floor, because if you throw from a higher floor and get a death you don't have a result).
Now, consider the function g(x, y) = SUM 1<=r<=y of xCr. (That's a binomial coefficient, x choose r. I don't think the TeX notation is supported on this site). The assertion is that f(x, y) = g(x, y). To check the base cases, consider that g(x, 0) is a sum over 0 elements, so that matches; and g(1, y) where y>0 gives 1C1 = 1. So it just remains to check that g satisfies the recurrence.
g(x-1, y-1) + g(x-1, y) + 1 = SUM 1<=r<=y-1 x-1Cr + SUM 1<=r<=y x-1Cr + 1
g(x-1, y-1) + g(x-1, y) + 1 = SUM 2<=r<=y x-1Cr-1 + SUM 1<=r<=y x-1Cr + 1
g(x-1, y-1) + g(x-1, y) + 1 = SUM 2<=r<=y x-1Cr-1 + SUM 1<=r<=y x-1Cr + x-1C0
g(x-1, y-1) + g(x-1, y) + 1 = SUM 1<=r<=y x-1Cr-1 + SUM 1<=r<=y x-1Cr
g(x-1, y-1) + g(x-1, y) + 1 = SUM 1<=r<=y (x-1Cr-1 + x-1Cr)
g(x-1, y-1) + g(x-1, y) + 1 = SUM 1<=r<=y (xCr)
g(x-1, y-1) + g(x-1, y) + 1 = g(x, y)
QED
In fact this can be derived directly by considering it as a combinatorial problem. If we make full use of every bit of information gained, then each possible sequence of survival or death corresponds to a different maximal floor. E.g. for three throws, one permitted death, the possibilities are death; survival-death; survival-survival-death; or survival-survival-survival. However, we have to discount the case that there are no deaths, because in that case we haven't determined the floor. So if we have x throws and y permitted deaths, we can have an actual number of deaths r from 1 to y, and for each of those we have xCr possible sequences. (If r = y then any "survivals" after the yth death are actually "didn't throw"s).
jdmetz's solution for F consists of evaluating g(D, B). It can't be done in better than O(min(D, B)) time because there's no closed hypergeometric form for that sum (which fact can be proven using Gosper's algorithm).
Addendum
In fact, all three parts of the original problem can be done in linear time (assuming multiplication and arithmetic to be constant-time operations, which we have so far - not true, but we'll leave that aside). The O(n lg n) operation in jdmetz's solution is finding the smallest D such that f(D, B) >= F.
If we combine our original recurrence f(D, B) = f(D-1, B-1) + f(D-1, B) + 1 with the term difference from the form of g, f(D, B) = f(D, B-1) + DCB we get f(D, B) = 2 * f(D-1, B) + 1 - D-1CB. We can then use aCb = a-1Cb * a / (a - b) to loop over D from 1.
private static int d_opt(final long f, final int b)
{
int d = 0, dCb = 0;
long f_db = 0;
while (f_db < f)
{
dCb = (d == b) ? 1 : d * dCb / (d-b);
f_db = 2 * f_db + 1 - dCb;
d++;
}
return d;
}
