how are edge square attacks handled

Question

I've been trying to understand magic bitboards for attack generation in chess engines. I think I roughly understand them, but one thing everyone fails to explain properly is how to handle blockers with edge squares

For example, this stackoverflow answer https://stackoverflow.com/a/30862064/18749115

The blocker mask is all of the squares that can be occupied and block your piece from moving further. The edge squares don't need to be a part of that, because your piece can't move further past that square anyway

The move board is the resulting available moves for your piece, for a given blocker board. This includes possible captures for your piece. Note that it also includes capturing your own pieces (but you can just AND it with a NOT of your own piece locations to remove those).

Here he explains edge squares don't need to be included, but the example he gave doesn't have an edge square, so he just easily and the bitboard.

I understand that not including edge squares drastically reduce the size of the table. But I'm confused how we would go about generating attacks when an enemy piece happens to be in an edge square

Answer 1

The point is that it doesn't matter whether there's a piece on the edge square, the sliding piece will have the same attacks in either case. If an edge square is occupied (by either friend or foe)? The piece can attack it. If it isn't occupied? Well, the square is still under attack and the attack pattern is the same.

The bottom line is, the sliding piece will always be able to attack the edge square no matter it's occupancy status.

Consider the following board:

. . . P . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . R . P . .
. . . . . . . .
. . . . . . . .
. . . . . . . . 

There's 3 pieces on the board. A rook in the middle and two other pieces (doesn't matter what pieces). What is the attack pattern for the rook?

. . . 1 . . . .
. . . 1 . . . .
. . . 1 . . . .
. . . 1 . . . .
1 1 1 R 1 1 . .
. . . 1 . . . .
. . . 1 . . . .
. . . 1 . . . . 

Now let's consider another board configuration:

. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . R . P . .
. . . . . . . .
. . . . . . . .
. . . . . . . . 

This time there is no piece on D8. What is the attack pattern?

. . . 1 . . . .
. . . 1 . . . .
. . . 1 . . . .
. . . 1 . . . .
1 1 1 R 1 1 . .
. . . 1 . . . .
. . . 1 . . . .
. . . 1 . . . . 

It is exactly the same.

The presence of a piece on D8 did not change the attack pattern - we would get the same one if it was't there. So when we are probing our precalculated attacks table, we can safely mask off the edge squares from our occupancy bitboard that we use for the key calculation, because we know that we will get the same attack pattern whether they are occupied or not. This allows us to make our table with precalculated attacks a bit smaller and save some space.

If the sliding piece is located in the inner portion of the board, you can just mask off the A and H files, and 1st and 8nd rank entirely. The only issue is when the sliding piece is on the edge of the board

. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
R . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . . 

Now you have to be careful to consider the occupacy of the A file as well. You can only mask off 1st and 8th rank and H file.

So the mask calculation is:

Bitboard rank_edge_mask = (rank_mask[RANK_1] | rank_mask[RANK_8]) & ~rank_mask[attacking_piece_rank];
Bitboard file_edge_mask = (file_mask[FILE_A] | file_mask[FILE_H]) & ~file_mask[attacking_piece_file];
Bitboard inner_mask = ~(rank_edge_mask | file_edge_mask);