This solution below will iterate through each cell with a value in column B of sheet1 and check it against every value in Column A of sheet89 (although you named this ColumnO, according to your getValues function, it will grab values from Column A).
If it finds a match, it will turn green the cell in column B of sheet1. In your example code you use the i loop variable (which iterates through rows on sheet89) to get the cell on sheet1 to turn green. It's not clear which cells you want to turn green. I assumed it was the cells on sheet1 so I changed the the code to
sheet.getRange(h+1, 2).setBackgroundColor('green');
Also, the getRange function for a single cell only requires 2 arguments, so I removed the numRows and numColumns arguments for the line which colors the cell green.
I'm not sure why bValues and oValues exclude the last row, but I removed the -1 in each of these as it will cause the code to fail if for any reason it is run on a blank worksheet. The getLastRow() returns the last row with a value, not the next blank row in the sheet. If you want to capture the whole sheet, then you shouldn't use -1.
function formatting() {
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet1');
var columnB = sheet.getRange(1, 2, sheet.getLastRow(), 1);
var bValues = columnB.getValues();
var sheet89 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet89');
var columnO = sheet89.getRange(1, 1, sheet.getLastRow(), 1);
var oValues = columnO.getValues();
for (var h = 0; h < bValues.length; h++) {
for (var i = 0; i < oValues.length; i++) {
if (oValues[i][0] == bValues[h][0]) {
sheet.getRange(h + 1, 2).setBackgroundColor('green');
}
}
}
}