As others have said, the problem is that you have to pass malloc
and integer because it doesn't make sense to allocate e.g. 1.2
bytes.
Even with this message, you code would normally compile and run: it is a warning, not an error. But you must have turned on treating warnings as errors (the clue is that the error message explicitly tells you this!), which is why it's not compiling at all. If you didn't have that turned on then the amount allocated would be rounded down from the number you asked, because that's how floating point numbers are converted to integers e.g.
char* x = malloc(3.7);
Will produce equivalent code to:
char* x = malloc(3);
If you are OK with that behaviour then you can avoid the warning - which in your case will allow the code to compile - by adding a cast:
char* buffer = (char*)malloc((size_t)(height * widthStep * 1.2));
Almost certainly you want to round up rather than down though, and probably don't want to use floating point numbers in your computation at all:
char* buffer = (char*)malloc((height * widthStep * 6 + 4) / 5);
More to the point, you probably don't want to be using malloc in C++ at all:
std::vector<char> buffer((height * widthStep * 6 + 4) / 5);