The reason why the execution of this function grinds to a crawl is that it utilizes multiple recursion e.g. the function calls itself twice upon each recursive call. That means that there are repeated computations taking place during the execution of this recursive function, and that the time complexity of the computation increases exponentially as the size of the inputs increase.
The effects of this are more noticeable with larger input values like 20.
Let's look at a call to grid(5, 5).
This expands as follows.
grid(5, 5)
grid(4, 5) + grid(5, 4)
(grid(3, 5) + grid(4, 4)) + (grid(4, 4) + grid(5, 3))
((grid(2, 5) + grid(3, 4)) + (grid(4, 3) + grid(3, 4))) +
((grid(3, 4) + grid(4, 3)) + (grid(4, 3) + grid(5, 2)))
...and so on
As you can see things get out of hand quickly even with small values of x and y, grid(3, 4) and grid(4, 3) are calculated multiple times. As stated previously, a solution that utilizes zipWith will be much more efficient.