[mpich-discuss] How To Limit Memory for each MPI Process

["Antonio J. Peña"]

Hi Steven,

Sorry, we don't have that functionality built-in in MPICH. You'll have 
to find an external alternative like those you mentioned.

Best,
   Antonio


On 09/12/2014 01:09 AM, Bowen Yu wrote:
> Hi,
>
> It's easy to control the memory usage of Java containers. Specify java 
> parameters like -Xmx when launch the containers and JVM will assure 
> the maximum memory usage. However, limitation on containers is not 
> enough. When the Application Master invokes mpiexec, Hydra will spawn 
> proxy to all the nodes needed by this application via ssh. All the 
> processes are forked by Hydra Process Manager Proxies, which is not 
> controlled by the container.
>
> I don't know if MPI has mechanism to limit memory for each MPI 
> processes. What I really mean is like I specify 
> MAX_MEMORY_USAGE_PER_PROCESS in environment variable, and I am assured 
> that none of the MPI processes will exceed that amount.
>
>
> ------------------ Original ------------------
> *From: * "Lu, Huiwei";<huiweilu at mcs.anl.gov>;
> *Date: * Thu, Sep 11, 2014 08:49 PM
> *To: * "discuss at mpich.org"<discuss at mpich.org>;
> *Subject: * Re: [mpich-discuss] How To Limit Memory for each MPI Process
>
> Hi Steven,
>
> It’s good to know that you have enabled MPICH to work with Hadoop YARN.
>
> So you have a Java container inside a MPI process and want to limit 
> the memory usage of a container. A MPI process is as the same as a 
> system process. And you can limit its resource usage use either ulimit 
> or cgroup. However, if the Java container is not aware of the limit 
> and continue to allocate the memory, it will still exceed the memory 
> usage. I don’t know if I understand the problem correctly. But maybe 
> it’s better to limit the memory usage of a Java container. Is there a 
> way to limit the memory usage of a Java container?
>
> Thanks.
> —
> Huiwei Lu
> Postdoc Appointee
> Mathematics and Computer Science Division, Argonne National Laboratory
> http://www.mcs.anl.gov/~huiweilu/
>
> On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002 at qq.com> wrote:
>
> > Hi,
> >
> > I'm developing a application that enables MPICH executables running 
> at Hadoop YARN cluster, and most functionality has been finished: 
> https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses 
> MPICH-3.1.2 to run MPI executable.
> >
> > YARN allocate resources in container, and in one container there are 
> specific amount of memory and CPU virtual cores. MPICH-YARN assumes 
> one MPI process is one-to-one correspondence with one container, so 
> the MPI process' memory should be limited. But I have no idea how. How 
> to do so that when I run mpiexec, each process is running with a 
> limited resource, such as memory, and CPU utilization; and if one of 
> the process' memory exceeds, the MPI whole program fails?
> >
> > I know two ways to implement resource limitation in Linux, one is to 
> use system call in programs or ulimit command in shell; the other is 
> to use cgroup kernel module.
> >
> > Thanks!
> > Steven
> > _______________________________________________
> > discuss mailing list     discuss at mpich.org
> > To manage subscription options or unsubscribe:
> > https://lists.mpich.org/mailman/listinfo/discuss
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss
>
>
> _______________________________________________
> discuss mailing list     discuss at mpich.org
> To manage subscription options or unsubscribe:
> https://lists.mpich.org/mailman/listinfo/discuss


-- 
Antonio J. Peña
Postdoctoral Appointee
Mathematics and Computer Science Division
Argonne National Laboratory
9700 South Cass Avenue, Bldg. 240, Of. 3148
Argonne, IL 60439-4847
apenya at mcs.anl.gov
www.mcs.anl.gov/~apenya

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpich.org/pipermail/discuss/attachments/20140912/64292d06/attachment.html>