How to find the empty squares in a chess board image?

Question

I am currently writing an image recognition script that makes a 2D array out of an image of a chess board for my chess project. However, I found it quite difficult to find which squares are empty:

So far, I have used the Canny edge detection on my image after applying the gaussian blur, which yielded the following results:

The code I used was:

sigma = 0.33
v = np.median(img) 
img = cv2.GaussianBlur(img, (7, 7), 2) # we use gaussian blur on the image to make it clear.
lower = int(max(0, (1.0 - sigma) * v)) # we find the lower threshold.
upper = int(min(255, (1.0 + sigma) * v)) # we find the higher threshold.
img_edge = cv2.Canny(img, 50, 50) # we use the canny function to edge canny the image.
cv2.imshow('question', img_edge) # we show the image.
cv2.waitKey(0)

(You may notice I did not use the threshold I got, that's because I found it inaccurate. If anyone has any tips I'd love them!)

Now, after doing these steps, I have tried many other things such as finding contours, Hough transform, etc. Yet I can't seem to figure out how to move on from that and actually find out whether a square is empty.

Any help is appreciated!

Answer 1

Assuming you have some kind of square shaped input image covering the whole chess board (as the example suggests), you can resize the image by rounding width and height to the next smaller multiple of 8. So, you can derive 64 equally sized tiles from your image. For each tile, count the number of unique colors. Set up some threshold to distinguish two classes (empty vs. non-empty square), maybe by using Otsu's method.

That'd be my code (half of that is simply visualization stuff):

import cv2
import matplotlib.pyplot as plt
import numpy as np
from skimage.filters import threshold_otsu


# Round to next smaller multiple of 8
# https://www.geeksforgeeks.org/round-to-next-smaller-multiple-of-8/
def round_down_to_next_multiple_of_8(a):
    return a & (-8)


# Read image, and shrink to quadratic shape with width and height of
# next smaller multiple of 8
img = cv2.imread('m0fAx.png')
wh = np.min(round_down_to_next_multiple_of_8(np.array(img.shape[:2])))
img = cv2.resize(img, (wh, wh))

# Prepare some visualization output
out = img.copy()
plt.figure(1, figsize=(18, 6))
plt.subplot(1, 3, 1), plt.imshow(img)

# Blur image
img = cv2.blur(img, (5, 5))

# Iterate tiles, and count unique colors inside
# https://stackoverflow.com/a/56606457/11089932
wh_t = wh // 8
count_unique_colors = np.zeros((8, 8))
for x in np.arange(8):
    for y in np.arange(8):
        tile = img[y*wh_t:(y+1)*wh_t, x*wh_t:(x+1)*wh_t]
        tile = tile[3:-3, 3:-3]
        count_unique_colors[y, x] = np.unique(tile.reshape(-1, tile.shape[-1]), axis=0).shape[0]

# Mask empty squares using cutoff from Otsu's method
val = threshold_otsu(count_unique_colors)
mask = count_unique_colors < val

# Some more visualization output
for x in np.arange(8):
    for y in np.arange(8):
        if mask[y, x]:
            cv2.rectangle(out, (x*wh_t+3, y*wh_t+3),
                          ((x+1)*wh_t-3, (y+1)*wh_t-3), (0, 255, 0), 2)
plt.subplot(1, 3, 2), plt.imshow(count_unique_colors, cmap='gray')
plt.subplot(1, 3, 3), plt.imshow(out)
plt.tight_layout(), plt.show()

And, that'd be the output:

As you can see, it's not perfect. One issue is the camera position, specifically that angle, but you already mentioned in the comments, that you can correct that. The other issue, as also already discussed in the comments, is the fact, that some pieces are placed between two squares. It's up to you, how to handle that. (I'd simply place the pieces correctly.)

----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.19041-SP0
Python:        3.9.1
PyCharm:       2021.1.1
Matplotlib:    3.4.2
NumPy:         1.20.3
OpenCV:        4.5.2
scikit-image:  0.18.1
----------------------------------------

Answer 2

Not like the given original image, but if you have a chessboard with pieces of colours which are not of the same colour as the chessboard (as discussed in the comments), then you can do something like this:

import cv2
import numpy

img = cv2.imread("Chesss.PNG")             # read image using cv2

for x in range(0,img.shape[0] - 8, img.shape[0]//8):
    for y in range(0,img.shape[1] - 8, img.shape[1]//8):      
        square = img[x:x+img.shape[0]//8, y:y+img.shape[1]//8, :]          # creating 8*8 squares of image
        avg_colour_per_row = numpy.average(square, axis=0)
        avg_colour = numpy.array(list(map(int, numpy.average(avg_colour_per_row, axis=0))))//8         # finding average colour of the square
        
        if list(avg_colour) == list(numpy.array([0, 0, 0])) or list(avg_colour) == list(numpy.array([31, 31, 31])):         # if average colour  of the squareis black or white, then print the coordinates of the square
            print(x//(img.shape[0]//8), y//(img.shape[1]//8))

My example image (I do not have a chessboard right now, so I used a rendered image):

Output:

0 0
0 1
0 2
0 3
0 4
0 5
0 6
0 7
1 1
1 3
1 4
1 6
2 0
2 1
2 2
2 3
2 4
2 5
2 6
2 7
3 0
3 1
3 2
3 3
3 5
3 6
3 7
4 0
4 1
4 2
4 3
4 4
4 6
4 7
5 0
5 1
5 2
5 3
5 4
5 5
5 6
5 7
6 0
6 3
6 4
6 5
6 6
7 0
7 1
7 2
7 3
7 4
7 5
7 6
7 7

Note that I have divided the average colour vales by 8. This is because we will perceive [0, 0, 0] and [1, 1, 1] (and similarly) as black only.

Answer 3

You can find chessboard and even find it's pose like here. Then you'll able to estimate ellipse shape of piece base. Find ellipses, using, for instance, this project.

Filter out trash ellipses using pose knowledge, and you'll get pieces positions. Then you can find free cells.