<div dir="ltr"><div class="markdown-here-wrapper" style=""><p style="margin:0px 0px 1.2em!important">Hi MPICH-developers, </p>
<p style="margin:0px 0px 1.2em!important">I’ve built MPICH-3.3.2 from source on my univ. cluster to ensure that jobs can be launched with slurm’s srun (via —mpi=pmi2, slurm’s native pmi version). I built mpich from source using spack using ch4:ucx. </p>
<p style="margin:0px 0px 1.2em!important">I see the following errors in my outfile (from osu-micro-benchmarks runs): </p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline;white-space:pre;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important">....
slurmstepd: error: mpi/pmi2: no value for key in req
....
[1574612953.443924] [qnode5038:30333:0] mpool.c:38 UCX WARN object 0x1b88d00 was not returned to mpool ucp_rkeys
...
`
</code></pre><p style="margin:0px 0px 1.2em!important">as shown in a typical outfile here.</p>
<p style="margin:0px 0px 1.2em!important">I don’t see such errors with using openmpi-4.0.2 which was also built with UCX as the primary transport layer protocol where I compiled a newer version of UCX from source.</p>
<p style="margin:0px 0px 1.2em!important">I’m attaching the build log from mpich install for reference. </p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px"><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline;white-space:pre;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important">Windows Terminal[sas4990@quser10 ~]$ spack find -ldv mpich%gcc@8.3.0
==> 1 installed package
-- linux-rhel7-ivybridge / gcc@8.3.0 ----------------------------
btlyk64 mpich@3.3.2 device=ch4 +hydra netmod=ucx +pci pmi=pmi2 +romio+slurm~verbs+wrapperrpath
eb74crw libpciaccess@0.13.5
t6h6tgg libxml2@2.9.9~python
46i2h6v libiconv@1.16
dnbfmo2 xz@5.2.4
xaq5v23 zlib@1.2.11+optimize+pic+shared
77kx7yy slurm@19-05-3-2~gtk~hdf5~hwloc~mariadb~pmix+readline
[sas4990@quser10 ~]$
</code></pre><p style="margin:0px 0px 1.2em!important">Thanks in advance for your help! </p>
<div title="MDH:PGRpdj48ZGl2PkhpIE1QSUNILWRldmVsb3BlcnMsIDxicj48YnI+PC9kaXY+SSd2ZSBidWlsdCBN
UElDSC0zLjMuMiBmcm9tIHNvdXJjZSBvbiBteSB1bml2LiBjbHVzdGVyIHRvIGVuc3VyZSB0aGF0
IGpvYnMgY2FuIGJlIGxhdW5jaGVkIHdpdGggc2x1cm0ncyBzcnVuICh2aWEgLS1tcGk9cG1pMiwg
c2x1cm0ncyBuYXRpdmUgcG1pIHZlcnNpb24pLiBJIGJ1aWx0IG1waWNoIGZyb20gc291cmNlIHVz
aW5nIHNwYWNrIHVzaW5nIGNoNDp1Y3guIDxicj48YnI+PC9kaXY+SSBzZWUgdGhlIGZvbGxvd2lu
ZyBlcnJvcnMgaW4gbXkgb3V0ZmlsZSAoZnJvbSBvc3UtbWljcm8tYmVuY2htYXJrcyBydW5zKTog
PGJyPjxkaXY+YGBgPC9kaXY+PGRpdj4uLi4uPGJyPjwvZGl2PjxkaXY+CnNsdXJtc3RlcGQ6IGVy
cm9yOiBtcGkvcG1pMjogbm8gdmFsdWUgZm9yIGtleSAgaW4gcmVxIDxicj48L2Rpdj48ZGl2Pi4u
Li48L2Rpdj48ZGl2PgpbMTU3NDYxMjk1My40NDM5MjRdIFs8c3BhbiB6ZXVtNGMxPSJQUl8xMF8w
IiBkYXRhLWRkbndhYj0iUFJfMTBfMCIgYXJpYS1pbnZhbGlkPSJzcGVsbGluZyIgY2xhc3M9IkxJ
IG5nIj5xbm9kZTUwMzg8L3NwYW4+OjMwMzMzOjBdICAgICAgICAgIG1wb29sLmM6MzggICBVQ1gg
IFdBUk4gIG9iamVjdCAweDFiODhkMDAgd2FzIG5vdCByZXR1cm5lZCB0byA8c3BhbiB6ZXVtNGMx
PSJQUl8xMV8wIiBkYXRhLWRkbndhYj0iUFJfMTFfMCIgYXJpYS1pbnZhbGlkPSJzcGVsbGluZyIg
Y2xhc3M9IkxJIG5nIj5tcG9vbDwvc3Bhbj4gdWNwX3JrZXlzIDxicj48L2Rpdj48ZGl2Pi4uLjwv
ZGl2PjxkaXY+YGBgYDwvZGl2PjxkaXY+YXMgc2hvd24gaW4gYSB0eXBpY2FsIG91dGZpbGUgPGEg
aHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL3Mtc2FqaWQtYWxpL251LXF1ZXN0LW1waS9ibG9iL21h
c3Rlci9uZXdfbXBpX2JlbmNoL2VrM29vcnovYndfMTUiPmhlcmU8L2E+LjwvZGl2PjxkaXY+PGJy
PjwvZGl2PjxkaXY+SSBkb24ndCBzZWUgc3VjaCBlcnJvcnMgd2l0aCB1c2luZyBvcGVubXBpLTQu
MC4yIHdoaWNoIHdhcyBhbHNvIGJ1aWx0IHdpdGggVUNYIGFzIHRoZSBwcmltYXJ5IHRyYW5zcG9y
dCBsYXllciBwcm90b2NvbCB3aGVyZSBJIGNvbXBpbGVkIGEgbmV3ZXIgdmVyc2lvbiBvZiBVQ1gg
ZnJvbSBzb3VyY2UuPGJyPjxicj48L2Rpdj48ZGl2PkknbSBhdHRhY2hpbmcgdGhlIGJ1aWxkIGxv
ZyBmcm9tIG1waWNoIGluc3RhbGwgZm9yIHJlZmVyZW5jZS4gPGJyPjwvZGl2PjxkaXY+YGBgPC9k
aXY+PGRpdj48c3Bhbj5XaW5kb3dzIFRlcm1pbmFsPC9zcGFuPjxkaXYgc3R5bGU9ImRpc3BsYXk6
IGlubGluZS1ibG9jazsgd2hpdGUtc3BhY2U6IHByZTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEy
LCAxMiwgMTIpOyBmb250LWZhbWlseTogJnF1b3Q7Q29uc29sYXMmcXVvdDssIG1vbm9zcGFjZTsg
Zm9udC1zaXplOiAxMnB0OyBwYWRkaW5nOiA0cHg7Ij48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigy
MDQsIDIwNCwgMjA0KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+W3NhczQ5
OTBAcXVzZXIxMCB+XSQgc3BhY2sgZmluZCAtbGR2IG1waWNoJWdjY0A4LjMuMDxicj4gPC9zcGFu
PjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDU5LCAxMjAsIDI1NSk7IGJhY2tncm91bmQtY29sb3I6
IHJnYigxMiwgMTIsIDEyKTsiPj09Jmd0Ozwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigy
MDQsIDIwNCwgMjA0KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+IDEgaW5z
dGFsbGVkIHBhY2thZ2U8YnI+LS0gPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDEzNiwg
MjMsIDE1Mik7IGJhY2tncm91bmQtY29sb3I6IHJnYigxMiwgMTIsIDEyKTsiPmxpbnV4LXJoZWw3
LWl2eWJyaWRnZTwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigyMDQsIDIwNCwgMjA0KTsg
YmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+IC8gPC9zcGFuPjxzcGFuIHN0eWxl
PSJjb2xvcjogcmdiKDE5LCAxNjEsIDE0KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwg
MTIpOyI+Z2NjQDguMy4wPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDIwNCwgMjA0LCAy
MDQpOyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMTIsIDEyLCAxMik7Ij4gLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLTxicj48L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoMTE4LCAxMTgs
IDExOCk7IGJhY2tncm91bmQtY29sb3I6IHJnYigxMiwgMTIsIDEyKTsiPmJ0bHlrNjQ8L3NwYW4+
PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoMjA0LCAyMDQsIDIwNCk7IGJhY2tncm91bmQtY29sb3I6
IHJnYigxMiwgMTIsIDEyKTsiPiBtcGljaDwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYig1
OCwgMTUwLCAyMjEpOyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMTIsIDEyLCAxMik7Ij5AMy4zLjI8
L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoNTksIDEyMCwgMjU1KTsgYmFja2dyb3VuZC1j
b2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+IGRldmljZT1jaDQgK2h5ZHJhIG5ldG1vZD11Y3ggK3Bj
aSBwbWk9cG1pMiArcm9taW8rc2x1cm1+dmVyYnMrd3JhcHBlcnJwYXRoPGJyPjwvc3Bhbj48c3Bh
biBzdHlsZT0iY29sb3I6IHJnYigxMTgsIDExOCwgMTE4KTsgYmFja2dyb3VuZC1jb2xvcjogcmdi
KDEyLCAxMiwgMTIpOyI+ZWI3NGNydzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigyMDQs
IDIwNCwgMjA0KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+ICAgICBsaWJw
Y2lhY2Nlc3M8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoNTgsIDE1MCwgMjIxKTsgYmFj
a2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+QDAuMTMuNTxicj48L3NwYW4+PHNwYW4g
c3R5bGU9ImNvbG9yOiByZ2IoMTE4LCAxMTgsIDExOCk7IGJhY2tncm91bmQtY29sb3I6IHJnYigx
MiwgMTIsIDEyKTsiPnQ2aDZ0Z2c8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoMjA0LCAy
MDQsIDIwNCk7IGJhY2tncm91bmQtY29sb3I6IHJnYigxMiwgMTIsIDEyKTsiPiAgICAgbGlieG1s
Mjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYig1OCwgMTUwLCAyMjEpOyBiYWNrZ3JvdW5k
LWNvbG9yOiByZ2IoMTIsIDEyLCAxMik7Ij5AMi45Ljk8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9y
OiByZ2IoNTksIDEyMCwgMjU1KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+
fnB5dGhvbjxicj48L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoMTE4LCAxMTgsIDExOCk7
IGJhY2tncm91bmQtY29sb3I6IHJnYigxMiwgMTIsIDEyKTsiPjQ2aTJoNnY8L3NwYW4+PHNwYW4g
c3R5bGU9ImNvbG9yOiByZ2IoMjA0LCAyMDQsIDIwNCk7IGJhY2tncm91bmQtY29sb3I6IHJnYigx
MiwgMTIsIDEyKTsiPiAgICAgICAgIGxpYmljb252PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjog
cmdiKDU4LCAxNTAsIDIyMSk7IGJhY2tncm91bmQtY29sb3I6IHJnYigxMiwgMTIsIDEyKTsiPkAx
LjE2PGJyPjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigxMTgsIDExOCwgMTE4KTsgYmFj
a2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+ZG5iZm1vMjwvc3Bhbj48c3BhbiBzdHls
ZT0iY29sb3I6IHJnYigyMDQsIDIwNCwgMjA0KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAx
MiwgMTIpOyI+ICAgICAgICAgeHo8L3NwYW4+PHNwYW4gc3R5bGU9ImNvbG9yOiByZ2IoNTgsIDE1
MCwgMjIxKTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+QDUuMi40PGJyPjwv
c3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigxMTgsIDExOCwgMTE4KTsgYmFja2dyb3VuZC1j
b2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+eGFxNXYyMzwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6
IHJnYigyMDQsIDIwNCwgMjA0KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+
ICAgICAgICAgemxpYjwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYig1OCwgMTUwLCAyMjEp
OyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMTIsIDEyLCAxMik7Ij5AMS4yLjExPC9zcGFuPjxzcGFu
IHN0eWxlPSJjb2xvcjogcmdiKDU5LCAxMjAsIDI1NSk7IGJhY2tncm91bmQtY29sb3I6IHJnYigx
MiwgMTIsIDEyKTsiPitvcHRpbWl6ZStwaWMrc2hhcmVkPGJyPjwvc3Bhbj48c3BhbiBzdHlsZT0i
Y29sb3I6IHJnYigxMTgsIDExOCwgMTE4KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwg
MTIpOyI+NzdreDd5eTwvc3Bhbj48c3BhbiBzdHlsZT0iY29sb3I6IHJnYigyMDQsIDIwNCwgMjA0
KTsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDEyLCAxMiwgMTIpOyI+ICAgICBzbHVybTwvc3Bhbj48
c3BhbiBzdHlsZT0iY29sb3I6IHJnYig1OCwgMTUwLCAyMjEpOyBiYWNrZ3JvdW5kLWNvbG9yOiBy
Z2IoMTIsIDEyLCAxMik7Ij5AMTktMDUtMy0yPC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjogcmdi
KDU5LCAxMjAsIDI1NSk7IGJhY2tncm91bmQtY29sb3I6IHJnYigxMiwgMTIsIDEyKTsiPn5ndGt+
aGRmNX5od2xvY35tYXJpYWRifnBtaXgrcmVhZGxpbmU8YnI+PC9zcGFuPjxzcGFuIHN0eWxlPSJj
b2xvcjogcmdiKDAsIDAsIDApOyBiYWNrZ3JvdW5kLWNvbG9yOiByZ2IoMCwgMCwgMCk7Ij48YnI+
PC9zcGFuPjxzcGFuIHN0eWxlPSJjb2xvcjogcmdiKDIwNCwgMjA0LCAyMDQpOyBiYWNrZ3JvdW5k
LWNvbG9yOiByZ2IoMTIsIDEyLCAxMik7Ij5bc2FzNDk5MEBxdXNlcjEwIH5dJDwvc3Bhbj48L2Rp
dj48L2Rpdj48ZGl2PmBgYDxicj48L2Rpdj48ZGl2Pjxicj48YnI+PC9kaXY+PGRpdj48ZGl2Pjxk
aXY+PGRpdj5UaGFua3MgaW4gYWR2YW5jZSBmb3ImbmJzcDt5b3VyIGhlbHAhIDxicj48L2Rpdj48
ZGl2Pjxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj4=" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0"></div></div><div><div><div><div class="markdown-here-signature">-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.8px">Sajid Ali | PhD Candidate<br></div><div style="font-size:12.8px">Applied Physics<br></div><div style="font-size:12.8px">Northwestern University</div><div style="font-size:12.8px"><a href="http://s-sajid-ali.github.io" target="_blank">s-sajid-ali.github.io</a></div></div></div></div></div></div></div></div></div></div></div></div>