[mpich-devel] patch for ticket #1526 (MPI_Alloc_mem registered memory)

[Zhao, Xin]

Hi Steffen,

Thank you very much for your contribution. We pushed your patch into MPICH (03d4c77b2148769436b8ce0b682b9b7a662baeef) with some modification. You can directly check it from our repo: git.mpich.org/mpich.git.

Thanks,
Xin
________________________________________
From: Steffen Christgau [christgau at cs.uni-potsdam.de]
Sent: Tuesday, January 20, 2015 7:57 AM
To: devel at mpich.org
Subject: [mpich-devel] patch for ticket #1526 (MPI_Alloc_mem registered memory)

Hi,

I'm currently going to improve the RMA operations (and only those) of an
existing CH3 channel for the underlying hardware in a local development
branch of MPICH. I appreciate the feature to override the window
creation functions in the channel implementation, and the possibility
to override the RMA functions via the function table associated to each
window. These together simplify development a lot.

As one can override the OSC/RMA functions, I really wonder why it is not
possible to do the same with the implementation of MPI_Alloc_mem at a
CH3 channel level. I found that there was an attempt to introduce this
in ticket #1526. Although the original ticket points at a slightly
different spot, the discussion is about the issue I'm currently facing.
Anyways, it seems to be dormant for a long time and I was not able to
find traces of the OSC branch mentioned in the comments (likely that is
has been merged into master or it is lost in history) where a solution
for this issue was presented according to the comment. That said, I'd
like to propose a patch to allow a channel implementation to provide a
custom implementation of MPI_Alloc_mem resp. MPI_Free_mem.

The idea is borrowed from MPIDI_CH3_Channel_close() which is declared if
the MPIDI_CH3_HAS_CHANNEL_CLOSE macro is defined or falls back to the
default preprocessor macro "implementation". Same thing here: if
according macros are defined, the channel provides an implementation. If
not, the defaults are used, i.e. the current MPIU-calls. The CH3 device
implementation had to be adjusted to account these changes. As the
solution falls back to macro substitution by default, I don't expect any
performance impacts, broken code or test failures for existing channel
devices. The RMA test cases succeeded both for the my channel device as
they did before.

The patch (see attachment as well) is based on the v3.1.3 tag. Maybe it
helps other implementers (if there's any ;-)) to integrate the
allocation of special memory into their channel.

diff --git a/src/mpid/ch3/include/mpidimpl.h
b/src/mpid/ch3/include/mpidimpl.h
index 6a9db2d..1b0ae17 100644
--- a/src/mpid/ch3/include/mpidimpl.h
+++ b/src/mpid/ch3/include/mpidimpl.h
@@ -1222,6 +1222,19 @@ int MPIDI_Win_sync(MPID_Win *win);
 void *MPIDI_Alloc_mem(size_t size, MPID_Info *info_ptr);
 int MPIDI_Free_mem(void *ptr);

+#ifdef MPIDI_CH3_HAS_ALLOC_MEM
+void* MPIDI_CH3_Alloc_mem(size_t size, MPID_Info *info_ptr);
+/* fallback to MPIU_Malloc if channel does not have its own RMA memory
allocator */
+#else
+#define MPIDI_CH3_Alloc_mem(size, info_ptr)    MPIU_Malloc(size)
+#endif
+
+#ifdef MPIDI_CH3_HAS_FREE_MEM
+int MPIDI_Free_mem(void *ptr);
+#else
+#define MPIDI_CH3_Free_mem(ptr)    MPIU_Free(ptr);
+#endif
+
 /* Pvars */
 void MPIDI_CH3_RMA_Init_Pvars(void);

diff --git a/src/mpid/ch3/src/ch3u_rma_ops.c
b/src/mpid/ch3/src/ch3u_rma_ops.c
index f30c464..7a3e22b 100644
--- a/src/mpid/ch3/src/ch3u_rma_ops.c
+++ b/src/mpid/ch3/src/ch3u_rma_ops.c
@@ -472,7 +472,7 @@ void *MPIDI_Alloc_mem( size_t size, MPID_Info
*info_ptr )

     MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_ALLOC_MEM);

-    ap = MPIU_Malloc(size);
+    ap = MPIDI_CH3_Alloc_mem(size, info_ptr);

     MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_ALLOC_MEM);
     return ap;
@@ -490,7 +490,7 @@ int MPIDI_Free_mem( void *ptr )

     MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_FREE_MEM);

-    MPIU_Free(ptr);
+    MPIDI_CH3_Free_mem(ptr);

     MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_FREE_MEM);
     return mpi_errno;


Regards, Steffen