I prove that at least for one matrix you have a worst case of n comparisons, for that purpose I create a specific matrix A.
Consider the diagonal line from A[1][n] to A[n][1], this includes all values
A[i][j] for i+j=n+1
Set all elements to the left of that diagonal to 0:
A[i][j]=0 for i+j<n+1
and all remaining elements, including the diagonal itself, to 2:
A[i][j]=2 for i+j>=n+1
As you can easily check this is a valid matrix according to the required conditions. Now you can set any value on the diagonal to 1:
A[z][n+1-z] = 1
The result then looks like this:
0 0 0 2
0 0 2 2
0 1 2 2
2 2 2 2
This remains a valid matrix. Now search for X=1. In order to check if any value on the diagonal is 1 you have to look at each one, because they are independent. There are n values on the diagonal, you have to check each one in order to find the 1, so you need to do n comparisons.