[mpich-discuss] scheduling to real hw cores, not using hyperthreading (mpich-3.2)
Heinz-Ado Arnolds
arnolds at MPA-Garching.MPG.DE
Wed Apr 12 10:02:57 CDT 2017
Dear MPIch users and developers,
first of all many thanks for all the great work you have done for MPIch!
I'd like to have 4 MPI jobs scheduled by SGE starting 1 OpenMP job each with 10 threads, running on 2 nodes, each having 2 sockets, with 10 cores & 10 hwthreads. Only 10 cores (no hwthreads) should be used on each socket.
4 MPI: 1 OpenMP with 10 thread (i.e. 4x10 threads)
2 nodes, 2 sockets each, 10 cores & 10 hwthreads each
lscpu -a -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3
0 0 0 0 0:0:0:0
1 1 1 1 1:1:1:1
2 0 0 2 2:2:2:0
3 1 1 3 3:3:3:1
4 0 0 4 4:4:4:0
5 1 1 5 5:5:5:1
6 0 0 6 6:6:6:0
7 1 1 7 7:7:7:1
8 0 0 8 8:8:8:0
9 1 1 9 9:9:9:1
10 0 0 10 10:10:10:0
11 1 1 11 11:11:11:1
12 0 0 12 12:12:12:0
13 1 1 13 13:13:13:1
14 0 0 14 14:14:14:0
15 1 1 15 15:15:15:1
16 0 0 16 16:16:16:0
17 1 1 17 17:17:17:1
18 0 0 18 18:18:18:0
19 1 1 19 19:19:19:1
20 0 0 0 0:0:0:0
21 1 1 1 1:1:1:1
22 0 0 2 2:2:2:0
23 1 1 3 3:3:3:1
24 0 0 4 4:4:4:0
25 1 1 5 5:5:5:1
26 0 0 6 6:6:6:0
27 1 1 7 7:7:7:1
28 0 0 8 8:8:8:0
29 1 1 9 9:9:9:1
30 0 0 10 10:10:10:0
31 1 1 11 11:11:11:1
32 0 0 12 12:12:12:0
33 1 1 13 13:13:13:1
34 0 0 14 14:14:14:0
35 1 1 15 15:15:15:1
36 0 0 16 16:16:16:0
37 1 1 17 17:17:17:1
38 0 0 18 18:18:18:0
39 1 1 19 19:19:19:1
When I try to submit the job by using
HYDRA_TOPO_DEBUG=1 mpirun -np 4 -bind-to socket:1 -map-by socket ./myid
the distribution is done to 2 sockets on 2 nodes correctly, but all 10 cores + 10 hwthreads are used on each node:
process 0 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
process 1 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
process 2 binding: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
process 3 binding: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Additionally it seems, that the CPU masks & Cpus_allowed_list are not set for the called processes:
MPI Instance 0001 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39
MPI Instance 0002 of 0004 is on pascal-1-04, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39
MPI Instance 0003 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39
MPI Instance 0004 of 0004 is on pascal-3-06, 0x000000ff,0xffffffff, Cpus_allowed_list: 0-39
Another way to achive my task would be to use "-bind-to user"
HYDRA_TOPO_DEBUG=1 mpirun -np $nmpi -ppn $nmpipn -bind-to user:0+2+4+6+8+10+12+14+16+18,1+3+5+7+9+11+13+15+17+19 ./myid
This works great up to specifying cores 9 cores on each socket ("0+2+4+6+8+10+12+14+16,1+3+5+7+9+11+13+15+17"). As soon as I add ...+18,...+19, the job crashes with these messages:
*** Error in `/usr/bin/hydra_pmi_proxy': free(): invalid next size (fast): 0x0000000000f9f0c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x704fb)[0x2b3f62f5c4fb]
/lib64/libc.so.6(+0x76976)[0x2b3f62f62976]
/lib64/libc.so.6(+0x7716e)[0x2b3f62f6316e]
/usr/bin/hydra_pmi_proxy[0x41aa27]
/usr/bin/hydra_pmi_proxy[0x418aea]
/usr/bin/hydra_pmi_proxy[0x407c0c]
/usr/bin/hydra_pmi_proxy[0x420bb0]
/usr/bin/hydra_pmi_proxy[0x403e49]
/lib64/libc.so.6(__libc_start_main+0xf1)[0x2b3f62f0c4e1]
/usr/bin/hydra_pmi_proxy[0x404cca]
======= Memory map: ========
00400000-0045c000 r-xp 00000000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy
0065c000-0065d000 r--p 0005c000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy
0065d000-0065f000 rw-p 0005d000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy
0065f000-006ff000 rw-p 00000000 00:00 0
00f98000-00fda000 rw-p 00000000 00:00 0 [heap]
2b3f619c5000-2b3f619e8000 r-xp 00000000 00:02 1229 /lib64/ld-2.25.so
2b3f619e8000-2b3f619eb000 rw-p 00000000 00:00 0
2b3f61a09000-2b3f61a0d000 rw-p 00000000 00:00 0
2b3f61be7000-2b3f61be8000 r--p 00022000 00:02 1229 /lib64/ld-2.25.so
2b3f61be8000-2b3f61be9000 rw-p 00023000 00:02 1229 /lib64/ld-2.25.so
2b3f61be9000-2b3f61bea000 rw-p 00000000 00:00 0
2b3f61bea000-2b3f61c03000 r-xp 00000000 00:02 1417 /lib64/libpthread-2.25.so
2b3f61c03000-2b3f61e02000 ---p 00019000 00:02 1417 /lib64/libpthread-2.25.so
2b3f61e02000-2b3f61e03000 r--p 00018000 00:02 1417 /lib64/libpthread-2.25.so
2b3f61e03000-2b3f61e04000 rw-p 00019000 00:02 1417 /lib64/libpthread-2.25.so
2b3f61e04000-2b3f61e08000 rw-p 00000000 00:00 0
2b3f61e08000-2b3f61e2c000 r-xp 00000000 00:02 1227 /lib64/libudev.so.1.6.3
2b3f61e2c000-2b3f6202b000 ---p 00024000 00:02 1227 /lib64/libudev.so.1.6.3
2b3f6202b000-2b3f6202c000 r--p 00023000 00:02 1227 /lib64/libudev.so.1.6.3
2b3f6202c000-2b3f6202d000 rw-p 00024000 00:02 1227 /lib64/libudev.so.1.6.3
2b3f6202d000-2b3f62035000 r-xp 00000000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b3f62035000-2b3f62234000 ---p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b3f62234000-2b3f62235000 r--p 00007000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b3f62235000-2b3f62236000 rw-p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b3f62236000-2b3f62390000 r-xp 00000000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b3f62390000-2b3f62590000 ---p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b3f62590000-2b3f62598000 r--p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b3f62598000-2b3f6259a000 rw-p 00162000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b3f6259a000-2b3f6259b000 rw-p 00000000 00:00 0
2b3f6259b000-2b3f6259e000 r-xp 00000000 00:02 1433 /lib64/libdl-2.25.so
2b3f6259e000-2b3f6279d000 ---p 00003000 00:02 1433 /lib64/libdl-2.25.so
2b3f6279d000-2b3f6279e000 r--p 00002000 00:02 1433 /lib64/libdl-2.25.so
2b3f6279e000-2b3f6279f000 rw-p 00003000 00:02 1433 /lib64/libdl-2.25.so
2b3f6279f000-2b3f627b4000 r-xp 00000000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b3f627b4000-2b3f629b3000 ---p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b3f629b3000-2b3f629b4000 r--p 00014000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b3f629b4000-2b3f629b5000 rw-p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b3f629b5000-2b3f629d9000 r-xp 00000000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b3f629d9000-2b3f62bd9000 ---p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b3f62bd9000-2b3f62bda000 r--p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b3f62bda000-2b3f62bdb000 rw-p 00025000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b3f62bdb000-2b3f62ceb000 r-xp 00000000 00:02 1401 /lib64/libm-2.25.so
2b3f62ceb000-2b3f62eea000 ---p 00110000 00:02 1401 /lib64/libm-2.25.so
2b3f62eea000-2b3f62eeb000 r--p 0010f000 00:02 1401 /lib64/libm-2.25.so
2b3f62eeb000-2b3f62eec000 rw-p 00110000 00:02 1401 /lib64/libm-2.25.so
2b3f62eec000-2b3f63081000 r-xp 00000000 00:02 1233 /lib64/libc-2.25.so
2b3f63081000-2b3f63280000 ---p 00195000 00:02 1233 /lib64/libc-2.25.so
2b3f63280000-2b3f63284000 r--p 00194000 00:02 1233 /lib64/libc-2.25.so
2b3f63284000-2b3f63286000 rw-p 00198000 00:02 1233 /lib64/libc-2.25.so
2b3f63286000-2b3f6328a000 rw-p 00000000 00:00 0
2b3f6328a000-2b3f632a0000 r-xp 00000000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b3f632a0000-2b3f6349f000 ---p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b3f6349f000-2b3f634a0000 r--p 00015000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b3f634a0000-2b3f634a1000 rw-p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b3f64000000-2b3f64021000 rw-p 00000000 00:00 0
2b3f64021000-2b3f68000000 ---p 00000000 00:00 0
7ffc37219000-7ffc3723a000 rw-p 00000000 00:00 0 [stack]
7ffc37355000-7ffc37357000 r--p 00000000 00:00 0 [vvar]
7ffc37357000-7ffc37359000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
*** Error in `/usr/bin/hydra_pmi_proxy': free(): invalid next size (fast): 0x0000000001c734c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x704fb)[0x2b4dc54914fb]
/lib64/libc.so.6(+0x76976)[0x2b4dc5497976]
/lib64/libc.so.6(+0x7716e)[0x2b4dc549816e]
/usr/bin/hydra_pmi_proxy[0x41aa27]
/usr/bin/hydra_pmi_proxy[0x418aea]
/usr/bin/hydra_pmi_proxy[0x407c0c]
/usr/bin/hydra_pmi_proxy[0x420bb0]
/usr/bin/hydra_pmi_proxy[0x403e49]
/lib64/libc.so.6(__libc_start_main+0xf1)[0x2b4dc54414e1]
/usr/bin/hydra_pmi_proxy[0x404cca]
======= Memory map: ========
00400000-0045c000 r-xp 00000000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy
0065c000-0065d000 r--p 0005c000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy
0065d000-0065f000 rw-p 0005d000 00:10 2142949266 /afs/mpa/@sys/system/MPA-8.03/usr/bin/hydra_pmi_proxy
0065f000-006ff000 rw-p 00000000 00:00 0
01c6b000-01cad000 rw-p 00000000 00:00 0 [heap]
2b4dc3efa000-2b4dc3f1d000 r-xp 00000000 00:02 4297 /lib64/ld-2.25.so
2b4dc3f1d000-2b4dc3f20000 rw-p 00000000 00:00 0
2b4dc3f3e000-2b4dc3f42000 rw-p 00000000 00:00 0
2b4dc411c000-2b4dc411d000 r--p 00022000 00:02 4297 /lib64/ld-2.25.so
2b4dc411d000-2b4dc411e000 rw-p 00023000 00:02 4297 /lib64/ld-2.25.so
2b4dc411e000-2b4dc411f000 rw-p 00000000 00:00 0
2b4dc411f000-2b4dc4138000 r-xp 00000000 00:02 5136 /lib64/libpthread-2.25.so
2b4dc4138000-2b4dc4337000 ---p 00019000 00:02 5136 /lib64/libpthread-2.25.so
2b4dc4337000-2b4dc4338000 r--p 00018000 00:02 5136 /lib64/libpthread-2.25.so
2b4dc4338000-2b4dc4339000 rw-p 00019000 00:02 5136 /lib64/libpthread-2.25.so
2b4dc4339000-2b4dc433d000 rw-p 00000000 00:00 0
2b4dc433d000-2b4dc4361000 r-xp 00000000 00:02 4295 /lib64/libudev.so.1.6.3
2b4dc4361000-2b4dc4560000 ---p 00024000 00:02 4295 /lib64/libudev.so.1.6.3
2b4dc4560000-2b4dc4561000 r--p 00023000 00:02 4295 /lib64/libudev.so.1.6.3
2b4dc4561000-2b4dc4562000 rw-p 00024000 00:02 4295 /lib64/libudev.so.1.6.3
2b4dc4562000-2b4dc456a000 r-xp 00000000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b4dc456a000-2b4dc4769000 ---p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b4dc4769000-2b4dc476a000 r--p 00007000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b4dc476a000-2b4dc476b000 rw-p 00008000 00:10 2142711818 /afs/mpa/@sys/system/MPA-8.03/usr/X11/lib64/libpciaccess.so.0.11.1
2b4dc476b000-2b4dc48c5000 r-xp 00000000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b4dc48c5000-2b4dc4ac5000 ---p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b4dc4ac5000-2b4dc4acd000 r--p 0015a000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b4dc4acd000-2b4dc4acf000 rw-p 00162000 00:10 2142836572 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libxml2.so.2.9.4
2b4dc4acf000-2b4dc4ad0000 rw-p 00000000 00:00 0
2b4dc4ad0000-2b4dc4ad3000 r-xp 00000000 00:02 5152 /lib64/libdl-2.25.so
2b4dc4ad3000-2b4dc4cd2000 ---p 00003000 00:02 5152 /lib64/libdl-2.25.so
2b4dc4cd2000-2b4dc4cd3000 r--p 00002000 00:02 5152 /lib64/libdl-2.25.so
2b4dc4cd3000-2b4dc4cd4000 rw-p 00003000 00:02 5152 /lib64/libdl-2.25.so
2b4dc4cd4000-2b4dc4ce9000 r-xp 00000000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b4dc4ce9000-2b4dc4ee8000 ---p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b4dc4ee8000-2b4dc4ee9000 r--p 00014000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b4dc4ee9000-2b4dc4eea000 rw-p 00015000 00:10 2142836590 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libz.so.1.2.6
2b4dc4eea000-2b4dc4f0e000 r-xp 00000000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b4dc4f0e000-2b4dc510e000 ---p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b4dc510e000-2b4dc510f000 r--p 00024000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b4dc510f000-2b4dc5110000 rw-p 00025000 00:10 2142835680 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/liblzma.so.5.2.3
2b4dc5110000-2b4dc5220000 r-xp 00000000 00:02 4469 /lib64/libm-2.25.so
2b4dc5220000-2b4dc541f000 ---p 00110000 00:02 4469 /lib64/libm-2.25.so
2b4dc541f000-2b4dc5420000 r--p 0010f000 00:02 4469 /lib64/libm-2.25.so
2b4dc5420000-2b4dc5421000 rw-p 00110000 00:02 4469 /lib64/libm-2.25.so
2b4dc5421000-2b4dc55b6000 r-xp 00000000 00:02 4301 /lib64/libc-2.25.so
2b4dc55b6000-2b4dc57b5000 ---p 00195000 00:02 4301 /lib64/libc-2.25.so
2b4dc57b5000-2b4dc57b9000 r--p 00194000 00:02 4301 /lib64/libc-2.25.so
2b4dc57b9000-2b4dc57bb000 rw-p 00198000 00:02 4301 /lib64/libc-2.25.so
2b4dc57bb000-2b4dc57bf000 rw-p 00000000 00:00 0
2b4dc57bf000-2b4dc57c4000 r-xp 00000000 00:02 4321 /lib64/libnss_dns-2.25.so
2b4dc57c4000-2b4dc59c3000 ---p 00005000 00:02 4321 /lib64/libnss_dns-2.25.so
2b4dc59c3000-2b4dc59c4000 r--p 00004000 00:02 4321 /lib64/libnss_dns-2.25.so
2b4dc59c4000-2b4dc59c5000 rw-p 00005000 00:02 4321 /lib64/libnss_dns-2.25.so
2b4dc59c5000-2b4dc59d7000 r-xp 00000000 00:02 4457 /lib64/libresolv-2.25.so
2b4dc59d7000-2b4dc5bd7000 ---p 00012000 00:02 4457 /lib64/libresolv-2.25.so
2b4dc5bd7000-2b4dc5bd8000 r--p 00012000 00:02 4457 /lib64/libresolv-2.25.so
2b4dc5bd8000-2b4dc5bd9000 rw-p 00013000 00:02 4457 /lib64/libresolv-2.25.so
2b4dc5bd9000-2b4dc5bdb000 rw-p 00000000 00:00 0
2b4dc5bdb000-2b4dc5bf1000 r-xp 00000000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b4dc5bf1000-2b4dc5df0000 ---p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b4dc5df0000-2b4dc5df1000 r--p 00015000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b4dc5df1000-2b4dc5df2000 rw-p 00016000 00:10 2142835216 /afs/mpa/@sys/system/MPA-8.03/usr/lib64/libgcc_s.so.1
2b4dc8000000-2b4dc8021000 rw-p 00000000 00:00 0
2b4dc8021000-2b4dcc000000 ---p 00000000 00:00 0
7fff6a554000-7fff6a575000 rw-p 00000000 00:00 0 [stack]
7fff6a58e000-7fff6a590000 r--p 00000000 00:00 0 [vvar]
7fff6a590000-7fff6a592000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
[mpiexec at pascal-3-06] control_cb (/tmp/S/mpich-3.2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed
[mpiexec at pascal-3-06] HYDT_dmxu_poll_wait_for_event (/tmp/S/mpich-3.2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status
[mpiexec at pascal-3-06] HYD_pmci_wait_for_completion (/tmp/S/mpich-3.2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event
[mpiexec at pascal-3-06] main (/tmp/S/mpich-3.2/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion
Do you have any hint which option & parameters of mpirun I have to choose to achieve my task?
Kind regards,
Ado Arnolds
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4992 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20170412/1e93ae6e/attachment.p7s>
-------------- next part --------------
_______________________________________________
discuss mailing list discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss
More information about the discuss
mailing list