[mpich-discuss] How To Limit Memory for each MPI Process

[Lu, Huiwei]

Hi, Steven,

You mentioned the mapping between a container and a MPI process is one-to-one, right? If you limit the memory usage of a container, then you limit the memory usage of a *process*. Why is that not enough? The “-Xmx” can only be applied to process 0? There should be a way to apply “-Xmx” to all processes.

>> All the processes are forked by Hydra Process Manager Proxies, which is not controlled by the container. 

I am also guessing what you really want is to limit is the memory usage on a *node* that have multiple processes. If so, you can control two things: 1. the memory usage of a container which control the memory usage of each process; 2. the number of processes on a node. Then you should be able to control the the max memory usage of a node.

Best,
—
Huiwei

On Sep 12, 2014, at 4:24 AM, Antonio J. Peña <apenya at mcs.anl.gov> wrote:

> 
> Hi Steven,
> 
> Sorry, we don't have that functionality built-in in MPICH. You'll have to find an external alternative like those you mentioned.
> 
> Best,
>   Antonio
> 
> 
> On 09/12/2014 01:09 AM, Bowen Yu wrote:
>> Hi,
>> 
>> It's easy to control the memory usage of Java containers. Specify java parameters like -Xmx when launch the containers and JVM will assure the maximum memory usage. However, limitation on containers is not enough. When the Application Master invokes mpiexec, Hydra will spawn proxy to all the nodes needed by this application via ssh. All the processes are forked by Hydra Process Manager Proxies, which is not controlled by the container. 
>> 
>> I don't know if MPI has mechanism to limit memory for each MPI processes. What I really mean is like I specify MAX_MEMORY_USAGE_PER_PROCESS in environment variable, and I am assured that none of the MPI processes will exceed that amount. 
>> 
>> 
>> ------------------ Original ------------------
>> From:  "Lu, Huiwei";<huiweilu at mcs.anl.gov>;
>> Date:  Thu, Sep 11, 2014 08:49 PM
>> To:  "discuss at mpich.org"<discuss at mpich.org>;
>> Subject:  Re: [mpich-discuss] How To Limit Memory for each MPI Process
>> 
>> Hi Steven,
>> 
>> It’s good to know that you have enabled MPICH to work with Hadoop YARN.
>> 
>> So you have a Java container inside a MPI process and want to limit the memory usage of a container. A MPI process is as the same as a system process. And you can limit its resource usage use either ulimit or cgroup. However, if the Java container is not aware of the limit and continue to allocate the memory, it will still exceed the memory usage. I don’t know if I understand the problem correctly. But maybe it’s better to limit the memory usage of a Java container. Is there a way to limit the memory usage of a Java container?
>> 
>> Thanks.
>> —
>> Huiwei Lu
>> Postdoc Appointee
>> Mathematics and Computer Science Division, Argonne National Laboratory
>> http://www.mcs.anl.gov/~huiweilu/
>> 
>> On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002 at qq.com> wrote:
>> 
>> > Hi,
>> > 
>> > I'm developing a application that enables MPICH executables running at Hadoop YARN cluster, and most functionality has been finished: https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses MPICH-3.1.2 to run MPI executable.
>> > 
>> > YARN allocate resources in container, and in one container there are specific amount of memory and CPU virtual cores. MPICH-YARN assumes one MPI process is one-to-one correspondence with one container, so the MPI process' memory should be limited. But I have no idea how. How to do so that when I run mpiexec, each process is running with a limited resource, such as memory, and CPU utilization; and if one of the process' memory exceeds, the MPI whole program fails?
>> > 
>> > I know two ways to implement resource limitation in Linux, one is to use system call in programs or ulimit command in shell; the other is to use cgroup kernel module.
>> > 
>> > Thanks!
>> > Steven
>> > _______________________________________________
>> > discuss mailing list     discuss at mpich.org
>> > To manage subscription options or unsubscribe:
>> > https://lists.mpich.org/mailman/listinfo/discuss
>> 
>> _______________________________________________
>> discuss mailing list     discuss at mpich.org
>> To manage subscription options or unsubscribe:
>> https://lists.mpich.org/mailman/listinfo/discuss
>> 
>> 
>> _______________________________________________
>> discuss mailing list     
>> discuss at mpich.org
>> 
>> To manage subscription options or unsubscribe:
>> 
>> https://lists.mpich.org/mailman/listinfo/discuss
> 
> 
> -- 
> Antonio J. Peña
> Postdoctoral Appointee
> Mathematics and Computer Science Division
> Argonne National Laboratory
> 9700 South Cass Avenue, Bldg. 240, Of. 3148
> Argonne, IL 60439-4847
> 
> apenya at mcs.anl.gov
> www.mcs.anl.gov/~apenya
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss