[mpich-discuss] CPU usage versus Nodes, Threads

[Qiguo Jing]

Hey Huiwei,

Thanks for your quick response.

I think I am using blocking MPI calls.  MPI_Send() and MPI_Recv() are blocking calls, right?  For each time, the master thread has to collect data from all slaves, and do some calculation/output (only master thread does these extra calculation/output), so I guess non-blocking calls may be not suitable?  Maybe I will try non-blocking calls see if the results are different from blocking calls.

All threads will call MPI, so I think it is MPI_THREAD_MULTIPLE?

Jing

-----Original Message-----
From: Lu, Huiwei [mailto:huiweilu at mcs.anl.gov] 
Sent: Wednesday, October 22, 2014 6:40 PM
To: discuss at mpich.org
Subject: Re: [mpich-discuss] CPU usage versus Nodes, Threads

The cause of low utilization of CPU use can be blocking communications, load imbalance or thread contentions in your application.

What MPI calls are you using? Are they blocking or nonblocking? Are the load balanced on each node?

What thread level are your using? MPI_THREAD_MULTIPLE or MPI_THREAD_FUNNELED?

-
Huiwei

> On Oct 22, 2014, at 5:02 PM, Qiguo Jing <qjing at trinityconsultants.com> wrote:
> 
> Hi All,
>  
> We have a parallel program running on a cluster.  We recently found a case, which decreases the CPU usage and increase the run-time when increases Nodes.   Below is the results table.
>  
> The particular run requires a lot of data communication between nodes.
>  
> Any thoughts about this phenomena?  Or is there any way we can improve the CPU usage when using higher number of nodes?
>  
> Average CPU Usage (%)
> Number of Nodes
> Number of Threads/Node
> 100
> 1
> 8
> 92
> 2
> 8
> 50
> 3
> 8
> 40
> 4
> 8
> 35
> 5
> 8
> 30
> 6
> 8
> 25
> 7
> 8
> 20
> 8
> 8
> 20
> 8
> 4
>  
>  
> Thanks!
> 
> ______________________________________________________________________
> ___
> 
> The information transmitted is intended only for the person or entity 
> to which it is addressed and may contain confidential and/or 
> privileged material. Any review, retransmission, dissemination or 
> other use of, or taking of any action in reliance upon, this 
> information by persons or entities other than the intended recipient 
> is prohibited. If you received this in error, please contact the 
> sender and delete the material from any computer.
> ______________________________________________________________________
> ___ _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss

-- 
_________________________________________________________________________

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
_________________________________________________________________________
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss