Ultimately, you cannot know until you have made the actual compression.
On the other hand, (Shannon's) data entropy is usually a good estimation of the compressed file bitrate. For optimal accuracy, you would need to compute the discrete cosine transform (DCT) of the data and apply the quantization corresponding to the quality Q you want ot use. However, you can also try to apply the quantization in the original domain and then calculate the entropy. You can also try to evaluate a portion of the image and extrapolate the compressed bitrate to the whole image.
Finally, if the type of images you are compressing are more or less homogeneous --i.e., you obtain similar compression ratios for most images-- you can build a look-up table to get your estimations. If your images are not homogeneous, you could also try to classify the image (maybe based on the entropy, too?) and use a different look-up table for each class.