<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Gus, Reuti, Rajeev and others,<br>
I do have mpif77 and mpif90 in mpi/bin, so I conclude that it has
compiled ok.<br>
<br>
I have checked reuti's suggestions, indicating that the environment
is ok. I had added 'PATH=/my/mpi/bin:$PATH' to each of the .bashrc
files (for the user I am running on all three)<br>
<br>
I am using the same username on all three computers - is that ok?<br>
<br>
I have installed MPI on to a NFS drive, all accessing the same;
'which' confirms. Are there any libraries that needs to be installed
elsewhere? do need to run make install as root?<br>
<br>
I have added the 'LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpi/lib',
though made no difference. (should those "{" and "}" be included? I
entered it exactly like I have typed)<br>
<br>
I have mpicc the cpi program as suggested by Gus and tried it, as
well as the originally provided cpi (part of the downloaded mpi)
which I called cpi_orig, see the screen shot below. It looks like
there is an issue with libgfortran.so.3... Perhaps I have not got
the various compilers installed correctly? I would appreciate if
someone could give me a suggestion on how to properly remove and
re-install these (and select the correct ones).<br>
<br>
By the way, in the runs of cpi below, I reduced the number of cores
to be used on the first pc listed in the machinefile, from 8 to 4,
in the case there was an issue with the 9th one. As can be seen, it
is the first node (or core) on a remote computer where the problems
start to be listed which is node 5 in this case...<br>
<br>
If I run cpi on only one computer, it works. And yes, hostname works
even if all computers are included in the machinefile.<br>
<br>
Clear as mud, hey?<br>
<br>
Thanks for your help.<br>
<br>
Ron<br>
<br>
<br>
<img src="cid:part1.02010603.08000701@pgcgroup.com.au" alt=""><br>
<br>
<br>
<div class="moz-cite-prefix">On 2/03/2014 05:20, Gustavo Correa
wrote:<br>
</div>
<blockquote
cite="mid:FE51E930-0922-4F7F-B24B-2B09C8255597@ldeo.columbia.edu"
type="cite">
<pre wrap="">Hi Ron
Please, see my comments inline.
On Mar 1, 2014, at 6:29 AM, Reuti wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
Am 01.03.2014 um 12:18 schrieb Ron Palmer:
</pre>
<blockquote type="cite">
<pre wrap="">Gus and others,
many thanks for your comments and suggestions. I am unsure whether I need fortran included as I am not in control of the software to be run on this linux cluster. I have asked but not yet got a reply...
However, I did a yum search gfortran and installed the gcc-gfortran.x86_64 (I hope that was the one you referred to). I followed up with the ./configure ... as per your suggestion. That did not work. I then tried with only ./configure --prefix=/my/bin/folder. That worked so I also did make and make install.
</pre>
</blockquote>
</blockquote>
<pre wrap="">
Maybe there was a misspell in the configure command line with
I am not 100% sure about the required gfortran yum RPMs.
It may also need the corresponding "devel" package, if you want to check,
but I am not really sure. Maybe you already installed everything you need for
a functional gfortran.
As far as I can tell from your c2.txt file, configure seems to have recognized gfortran.
I can't tell for sure if it actually built the MPICH F77 and F90, although chances are that it did.
However, you may check if you have mpif77 and mpif90 in the MPICH bin directory,
which would be a good sign.
I would not leave the F77 and F90 interfaces out of the game,
because sooner or later somebody will come with a Fortran MPI program
to run on your machine.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">I have a machinefile with three linux computers, all sharing the same mpi binaries. Running hostname on all 20 cores works, so I guess I should conclude that mpi works, right?
</pre>
</blockquote>
<pre wrap="">
Are they using the same binaries /my/bin/folder for a non-interactive startup too, i.e.:
$ ssh sargeant which hydra_pmi_proxy
returns the same as on the master (maybe ~/.bashrc needs to set the $PATH)?
-- Reuti
</pre>
</blockquote>
<pre wrap="">
Make sure you check Reuti's suggestions.
Did you install MPICH in an NFS shared (across all machines) directory,
or did you install on each machine in a local directory?
As Reuti pointed out, the environment setup is important.
If MPICH is not installed in a directory that Linux searches by default, such as /usr or /usr/local,
you need to add the MPICH bin directory to the PATH, and probably the lib directory to
LD_LIBRARY_PATH (if built with shared libraries).
The simplest way is in the .bashrc/.tcshrc file in the user home directory (which I presume
is shared via NFS across the computers, otherwise you need to edit these files on each
computer)
Say:
- bash:
export PATH=${PATH}:/my/mpich/bin
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/my/mpich/lib
- tcsh:
setenv PATH ${PATH}:/my/mpich/bin
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/my/mpich/lib
Also, you can go a long way troubleshooting these problems if you read the MPICH
documentation (Installer's Guide, User's Guide, README file)
<a class="moz-txt-link-freetext" href="http://www.mpich.org/documentation/guides/">http://www.mpich.org/documentation/guides/</a>
If you really could run mpiexec hostname across all computers, I would guess you
don't have any security or network related issues (i.e. passwordless ssh works
across the nodes, there is no firewall blocking, the nodes names are properly
resolved most likely via /etc/hosts, and so on).
To make sure MPICH works with a simple MPI program,
compile the cpi.c program in the examples directory (comes with the source code tarball),
and run it across all nodes.
Something like this:
mpicc -o cpi cpi.c
mpiexec -machinefile all_machines -np 24 ./cpi
If it runs, then MPICH is alright, and any issues you are having with the other program,
are not an MPICH issue, but a bug or misconfiguration in the other program.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">I then tried the software I am supposed to run on the cluster. It works on all nodes on the master computer, but not if other computers are involved. I have captured the output from such failed attempt to run that software on all available nodes on all available computers in the cluster, see below. I have attached c2.txt and m2.txt (and the output of the failed run) in the attached gzipped file (the failed run in run2.txt and in the image below may not be from the same failed run...).
</pre>
</blockquote>
</blockquote>
<pre wrap="">
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">So, is it an issue for the software developers or is it an issue with mpich?
</pre>
</blockquote>
</blockquote>
<pre wrap="">
The screenshot you sent MPI ranks 9, 11,12, 14 don't printout their ranks.
This may or may not indicate a programming error, however combined with the fact that
the program fails in MPI_Barrier, which is a collective call, suggests it may be a bug in the program.
One possibility is that the MPI program is wrong, and somehow ranks 9,11,12,14
did not call MPI_Barrier (collective calls, as the name says, are collective,
must be called by all ranks in the "communicator").
Another line of search would be that in both the screenshot and the run2.txt file
complain of "Communication error with rank 8" in the MPI_Barrier.
In each case rank 8 is in a different computer (gainsborough and constable).
It may be that rank 8 never called MPI_Barrier somehow.
For instance, MPI collective calls inside "if" conditionals can cause traps like this,
as the "if" condition may turn out to be true on some ranks and false on others.
Judging from the small amount of output, the program seems to have failed
very early, which may help search for a possible bug (early in the code).
However, these are just wild guesses.
Is this an in-house program, public domain, or commercial software?
Anyway, this may be an issue to bring to that code developers.
I hope this helps,
Gus Correa
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
</blockquote>
</blockquote>
<br>
</body>
</html>