[mpich-devel] Win questions

[Bob Cernohous]

I haven't worked on rma before but was working on a problem and ran into 
this **comment in MPI_Win_post:

"Starts an RMA exposure epoch for the local window associated with win. 
**Only the processes belonging to group should access the window with RMA 
calls on win during this epoch.   Each process in group must issue a 
matching call to MPI_Win_start. MPI_Win_post does not block."

Would overlapping epochs be violating the ** line?  I decided I probably 
need to support this but I wondered if it's bending or breaking the 
'rules'?

The problem (code at the bottom of this email) is using a cartesian 
communicator and alternating "left/right' accumulates with 'up/down' 
accumulates on a single win.  So:

- Ranks 0,1,2,3 are doing a left/right accumulate.
- Ranks 4,5,6,7 are doing a left/right accumulate.
- ...

and then sometimes...

- Ranks 0,1,2,3 complete and enter the 'up/down' accumulate epoch
-- Rank 0 does MPI_Win_post to ranks 4,12
-- Rank 1 doesn MPI_Win_post to ranks 5,13
... 

So is Rank 0 posting to Rank 4 while 4 is still in the epoch with 5/6/7 a 
violation of "Only the processes belonging to group should access the 
window with RMA calls on win during this epoch"?   From Rank 4's point of 
view, rank 0 isn't in the group for the current win/epoch.

Putting a barrier (or something) in between or using two different win's 
fixes it.  I like using two win's since it separates the epochs and 
clearly doesn't use the wrong group/rank on the win.

    /* RMA transfers in left-right direction */
    MPI_Win_post(grp_lr, 0, win);
    MPI_Win_start(grp_lr, 0, win);
    MPI_Accumulate(&i, 1, MPI_INT, ranks_lr[LEFT] , 0, 1, MPI_INT, 
MPI_SUM, win);
    MPI_Accumulate(&i, 1, MPI_INT, ranks_lr[RIGHT], 0, 1, MPI_INT, 
MPI_SUM, win);
    MPI_Win_complete(win);
    MPI_Win_wait(win);

    /* RMA transfers in up-down direction */
    MPI_Win_post(grp_ud, 0, win);
    MPI_Win_start(grp_ud, 0, win);
    MPI_Accumulate(&i, 1, MPI_INT, ranks_ud[UP]  , 0, 1, MPI_INT, MPI_SUM, 
win);
    MPI_Accumulate(&i, 1, MPI_INT, ranks_ud[DOWN], 0, 1, MPI_INT, MPI_SUM, 
win);
    MPI_Win_complete(win);
    MPI_Win_wait(win);

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/devel/attachments/20130607/a7f48f19/attachment.html>