Need or not to allocate whole domain grid

Question

I have coded a small application which solves a partial equation with MPI library.

For each MPI rank, I allocate the whole grid domain to solve. The code is structured like this :

1. Allocate global array for whole domain

2. Define neighbours with MPI_Cart_shift

3. Calculate coordinates (xs,ys) (starting) and (xe,ye) (ending) of subdomains (on the grid) for the current MPI rank called "me"

4. Update the boundaries for current MPI rank

5. Enter into the main loop :

// Main loop 
while(!convergence)
{
    step = step + 1;
    t = t + dt ;

    /* Perform one step of the explicit scheme */
    Explicit(x0, x, dt, &resLoc, hx, hy, me, xs, ys, xe, ye, k0);

    /* Update the partial solution along the interface */
    updateBound(x0, NeighBor, comm2d, column_type, me, xs, ys, xe, ye, xcell);

    /* Reduce all cartesian subgrids for convergence */
    MPI_Allreduce(&resLoc, &result, 1, MPI_DOUBLE, MPI_SUM, comm);

    result= sqrt(result);

    if ((result<epsilon) || (step>maxStep)) break;
}

Code remains into the main loop until convergence is reached. Into Explicit, I compute the next values for current iteration. Then, I update the deges of each sub-domains and I sen/receive them to neighbours.

Finally, I compute the by doing a reduction the error as a respect of convergence value of current iteration.

If error is smaller than defined convergence (result < epsilon), then I break from this main loop.

6. Afterwards, I gather all sub-domains in a final array xfinal by doing :

MPI_Gather(xtemp,xcell*ycell, MPI_DOUBLE, xfinal, xcell*ycell, MPI_DOUBLE, 0, comm);

Then, I print the solution (xfinal array) into output file.

My question is that don't know how to avoid to allocate a whole domain array for each MPI rank : indeed, if size of 2D domain is very high, one may not have enough RAM memory on each single node to allocate the two 2D arrays that I use (x[i][j] and x0[i][j]).

I don't know if it is possible to allocate just one sub-domain per MPI process : but in this case, how to communicate between them ? I need a whole domain to send and receive the lines/columns between processes, don't I ?

If someone could help me and tell me if it is possible to just allocate a sub-domain for each MPI rank ?

Thanks in advance.