I assume that you are interested in isolating the digits of the image and that you want to specify the ROI manually (given what you wrote).
You can use better coordinates for the ROI and crop that into a new cv::Mat
with it to get something like the output below:

Executing a threshold on this image only makes sense if you want to isolate the digits to do some recognition later. A good technique for doing that is offered by cv::inRange()
which performs a threshold operation on all channels (RGB image == 3 channels).
Note: cv::Mat
stores pixels in the BGR order, this is important to remember when you specify the values for the threshold.
As a simple test, you can perform a threshold from B:70 G:90 R:100 to B:140 G:140 R:140 to get the following output:

Not bad! I changed your code a little to get these results:
#include <cv.h>
#include <highgui.h>
#include <iostream>
int main()
{
cv::Mat image = cv::imread("input.jpg");
if (!image.data)
{
std::cout << "!!! imread failed to load image" << std::endl;
return -1;
}
cv::Rect roi;
roi.x = 165;
roi.y = 50;
roi.width = 440;
roi.height = 80;
/* Crop the original image to the defined ROI */
cv::Mat crop = image(roi);
cv::imwrite("colors_roi.png", crop);
/* Threshold the ROI based on a BGR color range to isolate yellow-ish colors */
cv::Mat dest;
cv::inRange(crop, cv::Scalar(70, 90, 100), cv::Scalar(140, 140, 140), dest);
cv::imwrite("colors_threshold.png", dest);
cv::imshow("Example", dest);
cv::waitKey();
return 0;
}