[mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS

[Jeff Hammond]

>> >> For example, on an 8-core node, I was hoping to get async progress on
>> >> 7 processes by pinning 7 comm threads to the 8th core.
>> >
>> > Did this work at all?
>>
>> What is your definition of work?
>
> Does it make better async progress than the default?

There's absolutely no reason why it should.  I am not trying to
maximize progress; I am trying to maximize computational performance.
Pinning the 7 comm threads to one core is going to be terrible for
them, but I am assuming that I don't need that much progress, whereas
I do need to the computation to run at top speed.  DGEMM on 7 cores
and 1 core of MPI should be much better than DGEMM on 8 cores where
each core is time-shared with a comm thread.

>> As far as I know, there is no API in MPICH for controlling thread
>> affinity.  The right way to improve this could would be to move it
>> inside of Nemesis and then add support for hwloc for comm threads.  I
>> assume you can move the parent processes around such that your 7
>> comm-intensive procs are closer to the NIC though.  You should look at
>> hwloc though.
>
> I'm thinking about how to interact with other threads _because_ I'm writing
> hwloc-based affinity management now. The behavior that I think people will
> want is to use MPI to set process affinity and then tell me to use some
> affinity policy to _restrict_ the process affinity to each thread. I prefer
> using MPI to set process affinity because I think it's more messy for
> software at a higher level than MPI to "figure out" what other processes are
> running on the same node and agree among themselves how to divvy up
> resources. If there was an important configuration that could not be
> supported this way, however, then I suppose I could do it.

You should run Intel MPI, MKL, OpenMP and TBB and only compile your
code with Intel compilers.  That's your best chance to have everything
work together.  I really don't see how Hydra is supposed to know what
GOMP is doing and try to deal with it.  Maybe I am reading too much
into your statement, but it sounds a lot like "give me a low calorie
stout that tastes fantastic." :-)

>> My diff w.r.t. the SVN trunk (I did all of this before the SVN->Git
>> conversion) is below.  It is clearly a hack and I don't care.  It only
>> works on Linux or other systems that support CPU_SET.  It does not
>> work on my Mac, for example.
>>
>> I have not done very much experimenting with this code other than to
>> verify that it works (as in "does not crash and gives the same result
>> for cpi").  Eventually, I am going to see how it works with ARMCI-MPI.
>
> Cool, I'll be curious.

What testing I have done of MPICH_NEMESIS_ASYNC_PROGRESS=1 on Cray
XC30 indicates that NWChem is better off without comm threads since it
communicates and computes at fine enough granularity such that the
lack of progress is less than the overhead of internal locking
(because ASYNC_PROGRESS implies MPI_THREAD_MULTIPLE) and competition
for execution resources (even though XC30 has Intel SNB with HT
enabled).

Jeff

-- 
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jhammond at alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond