[mpich-devel] MPICH memory pools

[Archer, Charles J]

Hi.

MPICH has some pretty nice functionality for memory pools implemented, but as far as I can tell, it’s a bit limited for internal device use because each pool you implement needs to consume an entry in the handle space.

Looking at the available “kinds” of memory pools already implemented:

typedef enum MPID_Object_kind {
  MPID_COMM       = 0x1,
  MPID_GROUP      = 0x2,
  MPID_DATATYPE   = 0x3,
  MPID_FILE       = 0x4, /* only used obliquely inside MPID_Errhandler objs */
  MPID_ERRHANDLER = 0x5,
  MPID_OP         = 0x6,
  MPID_INFO       = 0x7,
  MPID_WIN        = 0x8,
  MPID_KEYVAL     = 0x9,
  MPID_ATTR       = 0xa,
  MPID_REQUEST    = 0xb,
  MPID_PROCGROUP  = 0xc,               /* These are internal device objects */
  MPID_VCONN      = 0xd,
  MPID_GREQ_CLASS = 0xf
  } MPID_Object_kind;

It looks like only 0xe is available for implementing a new type of memory pool, limiting me to one additional pool.
Furthermore, the internal device objects don’t need publishable handles, right?
It looks like the handle contains 2 bits for (internal, valid, invalid, direct), and 4 bits to contain the object kind.

Are there any memory pool routines that I’m missing somewhere that aren’t restricted to the limits of what we can publish in a handle?
Since my object pools are internal, I don’t need to encode anything into a handle.

Furthermore, if we had a set of non-handle pool routines, internal pools like procgroup, vconn, wouldn’t consume entries in the handle space that could be used for future versions of MPI.

Looking for some guidance here, I don’t want to publish any internal device gorp into the object_kind space…but I want to use memory pools.
I’ve used the mpich pools on internal objects with a garbage kind value (unintentionally set to the wrong enum value), and it appears I get a new pool and everything is working, but just because it works, doesn’t mean it’s correct.

What should I do?  Bracing for "implement your own memory pools, lazy”.