[mpich-discuss] How To Limit Memory for each MPI Process
jxb002 at qq.com
Fri Sep 12 01:09:57 CDT 2014
It's easy to control the memory usage of Java containers. Specify java parameters like -Xmx when launch the containers and JVM will assure the maximum memory usage. However, limitation on containers is not enough. When the Application Master invokes mpiexec, Hydra will spawn proxy to all the nodes needed by this application via ssh. All the processes are forked by Hydra Process Manager Proxies, which is not controlled by the container.
I don't know if MPI has mechanism to limit memory for each MPI processes. What I really mean is like I specify MAX_MEMORY_USAGE_PER_PROCESS in environment variable, and I am assured that none of the MPI processes will exceed that amount.
------------------ Original ------------------
From: "Lu, Huiwei";<huiweilu at mcs.anl.gov>;
Date: Thu, Sep 11, 2014 08:49 PM
To: "discuss at mpich.org"<discuss at mpich.org>;
Subject: Re: [mpich-discuss] How To Limit Memory for each MPI Process
It’s good to know that you have enabled MPICH to work with Hadoop YARN.
So you have a Java container inside a MPI process and want to limit the memory usage of a container. A MPI process is as the same as a system process. And you can limit its resource usage use either ulimit or cgroup. However, if the Java container is not aware of the limit and continue to allocate the memory, it will still exceed the memory usage. I don’t know if I understand the problem correctly. But maybe it’s better to limit the memory usage of a Java container. Is there a way to limit the memory usage of a Java container?
Mathematics and Computer Science Division, Argonne National Laboratory
On Sep 11, 2014, at 6:55 AM, Bowen Yu <jxb002 at qq.com> wrote:
> I'm developing a application that enables MPICH executables running at Hadoop YARN cluster, and most functionality has been finished: https://github.com/alibaba/mpich2-yarn. This MPICH-YARN uses MPICH-3.1.2 to run MPI executable.
> YARN allocate resources in container, and in one container there are specific amount of memory and CPU virtual cores. MPICH-YARN assumes one MPI process is one-to-one correspondence with one container, so the MPI process' memory should be limited. But I have no idea how. How to do so that when I run mpiexec, each process is running with a limited resource, such as memory, and CPU utilization; and if one of the process' memory exceeds, the MPI whole program fails?
> I know two ways to implement resource limitation in Linux, one is to use system call in programs or ulimit command in shell; the other is to use cgroup kernel module.
> discuss mailing list discuss at mpich.org
> To manage subscription options or unsubscribe:
discuss mailing list discuss at mpich.org
To manage subscription options or unsubscribe:
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the discuss