[mpich-discuss] Using external load-balancer with mpich

[Halim Amer]

I don't know what is this load balancer you are using, but Hydra (the 
process manager) can interact with several standard job schedulers, such 
as SLURM and PBS. You can refer to 
https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager 
for more information about how to use Hydra.

Alternatively, if you want to keep your own scheduler, the only solution 
that comes to mind is for the scheduler to communicate indirectly with 
Hydra through a hosts file. The scheduler publishes the list of hosts in 
a file, say hosts.txt, and this file gets passed to hydra with the -f 
option: mpiexec -f hosts.txt. It is up to you to synchronize the 
scheduler and Hydra properly.

Halim
www.mcs.anl.gov/~aamer

On 10/27/16 11:59 PM, lostfreeman at gmail.com wrote:
> I have a system, that can start and stop machines depending on fleet’s
> current load. Essentially, it must be the one to decide, which machine
> will receive a new task, and optionally start acquire a new one from
> some pool for it.
>
>
>
> For example, I can have two hosts of six currently executing something
> (might be not related to MPICH), and my load balancer machine is aware
> of it. Other 4 are in low-power state to conserve costs. So when I call
> mpiexec, I want to tell it to send all the tasks to the load balancer,
> requesting 2 hosts, so that load balancer then could start two new hosts
> to handle that request. If the initial hosts would not be busy with some
> other task, load balancer would send incoming jobs to them, and kept all
> other four deallocated.
>
>
>
> - Victor
>
>
>
> *From: *Halim Amer <mailto:aamer at anl.gov>
> *Sent: *Thursday, October 27, 2016 9:18 PM
> *To: *discuss at mpich.org <mailto:discuss at mpich.org>
> *Subject: *Re: [mpich-discuss] Using external load-balancer with mpich
>
>
>
> I don't understand what you are trying to do. Can you give an example?
>
>
>
> Halim
>
> www.mcs.anl.gov/~aamer
>
>
>
> On 10/26/16 5:21 PM, lost wrote:
>
>> Can I use an external load balancer with mpiexec by providing a single
>
>> hostname in hosts file with, optionally, some large number for host
>
>> rank, and putting load balancer listening on that hostname and
>
>> forwarding connections to the actual hosts?
>
>>
>
>> I am trying to achieve autoscaling (load balancer tracks liveness of
>
>> hosts and spins up new ones on demand).
>
>>
>
>>
>
>> _______________________________________________
>
>> discuss mailing list     discuss at mpich.org
>
>> To manage subscription options or unsubscribe:
>
>> https://lists.mpich.org/mailman/listinfo/discuss
>
>>
>
> _______________________________________________
>
> discuss mailing list     discuss at mpich.org
>
> To manage subscription options or unsubscribe:
>
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
>
_______________________________________________
discuss mailing list     discuss at mpich.org
To manage subscription options or unsubscribe:
https://lists.mpich.org/mailman/listinfo/discuss