As you said, one binary search has a complexity of O(log n). You could imagine log(n) as the (maximum) number of steps that your algorithm need. For each of these steps your algorithm performs a nested binary search. That means, you could imagine this as log(n) times log(m) steps.
So you are right with O(log(n) x log(m)).
If for example n is bigger than m, then you have O((log(n))^2).
But normally we don't distinguish between different inputs. We consider them together as "the one" input of size n. Then your complexity is just O((log(n))^2).
Not distinguishing different inputs at least makes sense for polynomial algorithms. To see this consider an example: You have two inputs of sizes n and m with m <= n. Your total input size is therefore n+m. Let's see what happens for different complexities.
- O(n+m) <= 2*O(n). Normally when talking about complexity we are not interested in constant factors. so we can say, we have O(n).
- O(log(n+m)) <= O(log(2n)). Now we remember the logarithm rules from school. They say log(2n) = log(2) + log(n). Normally when talking about complexity we are not interested in constants. So we can say, we have O(log(n))
- O((n+m)^2) = O(n^2+2nm+m^2) <= O(n^2+2nn+n^2) = O(4n^2). Again we are not interested in constant factors and just say, we have O(n^2).