[mpich-devel] Request for comments: MPICH device-specific timers

[Balaji, Pavan]

Folks,

I'm considering removing the device timers from MPICH and fully relying on MPL timers.  This means, we'll no longer have MPID_Wtime and friends, but simply use MPL_Wtime (which internally would use the OS provided timers).

Why did we have device-specific timers in the first place?

The intent of the device-specific timers was to allow for platforms that provide node-synchronized timers to provide their own timers for MPI_Wtime and MPI_Wtick.  The last set of platforms that I know of that gave such synchronized timers were the Blue Gene machines.

What has changed now?

AFAICT from reading online, Blue Gene/Q eventually integrated these timers to update the TSC register, so the OS-provided timers such as clock_gettime (and hence MPL) would give the same time.  Plus, going forward, it seems more likely that vendors would integrate such synchronized timers into the OS timers anyway.  Thus the value of the device-provided timers doesn't seem to exist any longer.

Why can't we leave the current code in MPICH as-is?

The problem with allowing for device timers is that they are initialized with the device (e.g., in MPID_Init or in MPID_Wtime_init).  More importantly, this initialization of timers is now collective over all processes for platforms that have device-specific timers.  This creates a problem in the order of initialization of the various components because the timers need to be initialized early before things like logging can be initialized.  By moving to MPL timers, such initialization can be done completely at the MPI layer and completely locally.  A local-only initialization would be a significant improvement in the maintainability of the code.  Also, are we are looking to modify the initialization to integrate additional functionality (such as threading improvements), the initialization is becoming a big spaghetti mess, which this would help with.

AFAIK, none of the MPICH derivatives rely on this feature, but I'd love to hear thoughts from other developers if my understanding is incorrect.  If they are OK with this change, I'd also appreciate a note saying so.

Regards,

  -- Pavan