<html xmlns="http://www.w3.org/1999/xhtml"><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><title></title>
</head>
<body>
<div name="messageBodySection">Hi,<br>
<br>
<font color="#333333">You are using -bind-to socket and -map-by socket with mpirun when running with MVAPICH2. These will not result in correct process mapping with MVAPICH2+OpenMP. You needs to use the options mentioned in the userguide.<br>
(<a href="http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3b-userguide.html#x1-900006.20"></a></font><a><u>http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3b-userguide.html#x1-900006.20</u></a><u><font color="#333333"></font></u><font>)<br></font></div>
<div name="messageSignatureSection"><br>
regards<br>
Jahanzeb</div>
<div name="messageReplySection"><br>
On Aug 28, 2017, 9:39 AM -0400, Pasha Pashaei <pasha.313@hotmail.com>, wrote:<br>
<blockquote type="cite">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p></p>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Dear friends</div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">I am going to run a hybrid MPI+OPENMP code.</div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">As you can see in the below I played with the number of threads in various cases while running my main code with (Openmpi,Mpich,Mpich2).</div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">As you can see in OpenMPi and Mpich it seems that openmp did not work at all as Total time did not change considerably. But in Mpich2 Total computational time increased with increasing the number of threads. It could be because of using virtual threads instead of physical threads(or you said that over-subscribing).</div>
<br>
<p></p>
<p></p>
<div>Hybrid code result (MPI + OpenMP) :</div>
<div>Your suggestions: </div>
<div>mvapich</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 1 --bind-to hwthread:1 ./pjet.gfortran > output.txt</div>
<div>Total time = 7.290E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 ./pjet.gfortran > output.txt</div>
<div>Total time = 4.940E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 -map-by hwthread:8 ./pjet.gfortran > output.txt</div>
<div>Total time = 4.960E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 16 --bind-to hwthread:16 ./pjet.gfortran > output.txt</div>
<div>Total time = 4.502E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to core:16 -map-by core:16 ./pjet.gfortran > output.txt</div>
<div>Total time = 4.628E+02</div>
<div><br></div>
<div>Pervios commands </div>
<div>OpenMPI 1.8.1</div>
<div>mpirun -np 4 -x OMP_NUM_THREADS=1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.475E+02</div>
<div>mpirun -np 4 -x OMP_NUM_THREADS=8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.525E+02</div>
<div>mpirun -np 4 -x OMP_NUM_THREADS=16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.611E+02</div>
<div><br></div>
<div>mvapich </div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.441E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.535E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.552E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.591E+02</div>
<div><br></div>
<div><span style="font-size: 12pt;">mvapich2 </span><br></div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 4.935E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 5.562E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 6.392E+02</div>
<div>mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt</div>
<div>Total time = 8.170E+02</div>
<div><br></div>
<div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;"><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Then I used a simple "hybrid.f90" code and its result which I used to check whether the computer can recognize correct value of cores and threads or not. It showed that the correct values in all three in different cases.</span><br></div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">here is its result: </div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">
<p class="MsoPlainText"><span style="font-family:"Courier New""> Starting omp_dotprod_hybrid. Using 4 Cores...<br>
Core 3 using 16 threads<br>
Core 0 using 16 threads<br>
Core 2 using 16 threads<br>
Core 1 using 16 threads<br>
Core 1 thread 0 partial sum = 0.0000000000000000 <br>
Core 3 thread 0 partial sum = 0.0000000000000000 <br>
Core 1 thread 4 partial sum = 0.0000000000000000 <br>
Core 3 thread 7 partial sum = 200.00000000000000 <br>
Core 1 thread 8 partial sum = 200.00000000000000 <br>
Core 3 thread 9 partial sum = 200.00000000000000 <br>
Core 1 thread 11 partial sum = 200.00000000000000 <br>
Core 3 thread 3 partial sum = 200.00000000000000 <br>
Core 1 thread 2 partial sum = 0.0000000000000000 <br>
Core 3 thread 5 partial sum = 0.0000000000000000 <br>
Core 1 thread 3 partial sum = 200.00000000000000 <br>
Core 3 thread 2 partial sum = 0.0000000000000000 <br>
Core 1 thread 13 partial sum = 200.00000000000000 <br>
Core 3 thread 12 partial sum = 200.00000000000000 <br>
Core 1 thread 1 partial sum = 200.00000000000000 <br>
Core 3 thread 1 partial sum = 200.00000000000000 <br>
Core 3 thread 8 partial sum = 0.0000000000000000 <br>
Core 1 thread 7 partial sum = 0.0000000000000000 <br>
Core 3 thread 11 partial sum = 200.00000000000000 <br>
Core 1 thread 15 partial sum = 0.0000000000000000 <br>
Core 3 thread 15 partial sum = 0.0000000000000000 <br>
Core 1 thread 10 partial sum = 200.00000000000000 <br>
Core 1 thread 9 partial sum = 200.00000000000000 <br>
Core 3 thread 13 partial sum = 0.0000000000000000 <br>
Core 1 thread 5 partial sum = 200.00000000000000 <br>
Core 3 thread 6 partial sum = 0.0000000000000000 <br>
Core 3 thread 4 partial sum = 0.0000000000000000 <br>
Core 1 thread 6 partial sum = 0.0000000000000000 <br>
Core 3 thread 14 partial sum = 200.00000000000000 <br>
Core 1 thread 12 partial sum = 0.0000000000000000 <br>
Core 3 thread 10 partial sum = 200.00000000000000 <br>
Core 0 thread 0 partial sum = 0.0000000000000000 <br>
Core 0 thread 14 partial sum = 200.00000000000000 <br>
Core 0 thread 8 partial sum = 200.00000000000000 <br>
Core 0 thread 7 partial sum = 0.0000000000000000 <br>
Core 0 thread 15 partial sum = 200.00000000000000 <br>
Core 0 thread 5 partial sum = 200.00000000000000 <br>
Core 0 thread 9 partial sum = 200.00000000000000 <br>
Core 0 thread 11 partial sum = 0.0000000000000000 <br>
Core 0 thread 10 partial sum = 200.00000000000000 <br>
Core 0 thread 6 partial sum = 200.00000000000000 <br>
Core 0 thread 3 partial sum = 200.00000000000000 <br>
Core 0 thread 4 partial sum = 0.0000000000000000 <br>
Core 0 thread 2 partial sum = 0.0000000000000000 <br>
Core 0 thread 13 partial sum = 0.0000000000000000 <br>
Core 0 thread 12 partial sum = 0.0000000000000000 <br>
Core 0 thread 1 partial sum = 0.0000000000000000 <br>
Core 0 partial sum = 1600.0000000000000 <br>
Core 2 thread 3 partial sum = 0.0000000000000000 <br>
Core 2 thread 15 partial sum = 0.0000000000000000 <br>
Core 2 thread 0 partial sum = 0.0000000000000000 <br>
Core 2 thread 2 partial sum = 200.00000000000000 <br>
Core 2 thread 4 partial sum = 0.0000000000000000 <br>
Core 2 thread 5 partial sum = 0.0000000000000000 <br>
Core 2 thread 9 partial sum = 200.00000000000000 <br>
Core 2 thread 7 partial sum = 0.0000000000000000 <br>
Core 2 thread 14 partial sum = 200.00000000000000 <br>
Core 2 thread 8 partial sum = 200.00000000000000 <br>
Core 2 thread 12 partial sum = 200.00000000000000 <br>
Core 2 thread 10 partial sum = 200.00000000000000 <br>
Core 2 thread 6 partial sum = 200.00000000000000 <br>
Core 2 thread 1 partial sum = 0.0000000000000000 <br>
Core 2 thread 13 partial sum = 0.0000000000000000 <br>
Core 2 thread 11 partial sum = 200.00000000000000 <br>
Core 2 partial sum = 1600.0000000000000 <br>
Core 3 partial sum = 1600.0000000000000 <br>
Core 1 thread 14 partial sum = 0.0000000000000000 <br>
Core 1 partial sum = 1600.0000000000000 <br>
Done. Hybrid version: global sum = 6400.0000000000000 <br>
</span></p>
<br></div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;"><br></div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Please tell me If I should check something. I still getting nowhere.</div>
<div style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Best regards</div>
</div>
<p></p>
<div id="Signature">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255); font-family: Calibri, Arial, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p><font color="#FD7832">Pasha Pashaei</font><br>
<br></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>