[mpich-devel] progress in flushall

[Balaji, Pavan]

Jeff,

On Apr 30, 2014, at 4:29 PM, Jeff Hammond <jeff.science at gmail.com> wrote:

> On Wed, Apr 30, 2014 at 3:58 AM, Balaji, Pavan <balaji at anl.gov> wrote:
>> 
>> On Apr 29, 2014, at 10:20 PM, Jeff Hammond <jeff.science at gmail.com> wrote:
>>> Yeah, I know that's what MPICH gives me.  My point is that is
>>> undesirable.  All sync operations should make progress, no matter
>>> what.
>> 
>> They are not required to make progress from the spec perspective.  It’s a bad assumption to make that Win_flush will always make progress.  I think it’s OK for it to be mostly a no-op when there are no operations posted.  If you need progress, why don’t you use test or something similar?
> 
> I want flushall to make progress because there might be incoming RMA
> that won't be processed otherwise.  While today MPICH makes progress
> on all Ch3 ops in Barrier, if that ever changes, ARMCI_Barrier is
> going to deadlock because it does win_flush_all then mpi_barrier.

I don’t think that’ll change.  Barrier will need to make progress on RMA operations (unless the RMA operations are asynchronous) in order to remain compliant with the standard.  The following program has to always work correctly:

P0:
	MPI_Put
	MPI_Win_flush(_all)
	MPI_Barrier

P1:
	MPI_Barrier

The only case that’ll change is if we decide to somehow do async progress for all RMA operations (e.g., using a separate thread). But in that case, this entire problem wouldn’t exist anyway.

> I'm already adding Iprobe for progress inside of ARMCI-MPI, but it
> seems silly to have to test for p2p to get progress in RMA and for RMA
> sync ops to not guarantee progress.  I'd much prefer the design where
> p2p and rma progress are decoupled and my RMA sync ops are burdened
> with progress on p2p and vice versa.

I’m not sure how you can reliably do that, in a standard compliant way.

  — Pavan