rank = 2 hostname = woogie Iter = 6260 / 100000, Current rate: 4.78 Mbit/sec (for 20.5secs), Average rate: 4.78 Mbit/sec (for 20.5secs) Iter = 12376 / 100000, Current rate: 0.41 Mbit/sec (for 232.9secs), Average rate: 0.76 Mbit/sec (for 253.4secs) Iter = 12902 / 100000, Current rate: 0.20 Mbit/sec (for 41.0secs), Average rate: 0.68 Mbit/sec (for 294.3secs) Iter = 13159 / 100000, Current rate: 0.19 Mbit/sec (for 21.1secs), Average rate: 0.65 Mbit/sec (for 315.4secs) Iter = 13404 / 100000, Current rate: 0.18 Mbit/sec (for 21.5secs), Average rate: 0.62 Mbit/sec (for 336.9secs) Iter = 13633 / 100000, Current rate: 0.16 Mbit/sec (for 21.8secs), Average rate: 0.59 Mbit/sec (for 358.7secs) Iter = 13874 / 100000, Current rate: 0.17 Mbit/sec (for 22.2secs), Average rate: 0.57 Mbit/sec (for 380.9secs) Iter = 14124 / 100000, Current rate: 0.17 Mbit/sec (for 22.4secs), Average rate: 0.55 Mbit/sec (for 403.3secs) Iter = 14374 / 100000, Current rate: 0.17 Mbit/sec (for 22.9secs), Average rate: 0.53 Mbit/sec (for 426.2secs) Iter = 14623 / 100000, Current rate: 0.17 Mbit/sec (for 23.2secs), Average rate: 0.51 Mbit/sec (for 449.4secs) Iter = 14873 / 100000, Current rate: 0.17 Mbit/sec (for 23.6secs), Average rate: 0.49 Mbit/sec (for 473.0secs) Iter = 15123 / 100000, Current rate: 0.16 Mbit/sec (for 23.9secs), Average rate: 0.48 Mbit/sec (for 496.9secs) Iter = 15372 / 100000, Current rate: 0.16 Mbit/sec (for 24.3secs), Average rate: 0.46 Mbit/sec (for 521.1secs) Iter = 15622 / 100000, Current rate: 0.16 Mbit/sec (for 24.5secs), Average rate: 0.45 Mbit/sec (for 545.6secs) Iter = 15872 / 100000, Current rate: 0.16 Mbit/sec (for 24.9secs), Average rate: 0.43 Mbit/sec (for 570.5secs) Iter = 16121 / 100000, Current rate: 0.15 Mbit/sec (for 25.2secs), Average rate: 0.42 Mbit/sec (for 595.7secs) Iter = 16371 / 100000, Current rate: 0.15 Mbit/sec (for 25.6secs), Average rate: 0.41 Mbit/sec (for 621.3secs) Iter = 16621 / 100000, Current rate: 0.15 Mbit/sec (for 25.8secs), Average rate: 0.40 Mbit/sec (for 647.1secs) Iter = 16870 / 100000, Current rate: 0.15 Mbit/sec (for 26.1secs), Average rate: 0.39 Mbit/sec (for 673.2secs) Iter = 17120 / 100000, Current rate: 0.15 Mbit/sec (for 26.5secs), Average rate: 0.38 Mbit/sec (for 699.7secs) Iter = 17309 / 100000, Current rate: 0.15 Mbit/sec (for 20.1secs), Average rate: 0.38 Mbit/sec (for 719.8secs) Iter = 17498 / 100000, Current rate: 0.15 Mbit/sec (for 20.3secs), Average rate: 0.37 Mbit/sec (for 740.1secs) Iter = 17685 / 100000, Current rate: 0.14 Mbit/sec (for 20.4secs), Average rate: 0.36 Mbit/sec (for 760.4secs) Iter = 17869 / 100000, Current rate: 0.14 Mbit/sec (for 20.6secs), Average rate: 0.36 Mbit/sec (for 781.1secs) Iter = 18056 / 100000, Current rate: 0.14 Mbit/sec (for 20.7secs), Average rate: 0.35 Mbit/sec (for 801.8secs) Iter = 18243 / 100000, Current rate: 0.14 Mbit/sec (for 21.0secs), Average rate: 0.35 Mbit/sec (for 822.7secs) Iter = 18431 / 100000, Current rate: 0.14 Mbit/sec (for 21.2secs), Average rate: 0.34 Mbit/sec (for 843.9secs) Iter = 18618 / 100000, Current rate: 0.14 Mbit/sec (for 21.2secs), Average rate: 0.34 Mbit/sec (for 865.1secs) Iter = 18805 / 100000, Current rate: 0.14 Mbit/sec (for 21.5secs), Average rate: 0.33 Mbit/sec (for 886.6secs) Iter = 18992 / 100000, Current rate: 0.13 Mbit/sec (for 21.7secs), Average rate: 0.33 Mbit/sec (for 908.3secs) Iter = 19179 / 100000, Current rate: 0.13 Mbit/sec (for 21.7secs), Average rate: 0.32 Mbit/sec (for 930.0secs) Iter = 19367 / 100000, Current rate: 0.13 Mbit/sec (for 22.1secs), Average rate: 0.32 Mbit/sec (for 952.1secs) Iter = 19554 / 100000, Current rate: 0.13 Mbit/sec (for 22.0secs), Average rate: 0.31 Mbit/sec (for 974.1secs) Iter = 19741 / 100000, Current rate: 0.13 Mbit/sec (for 22.4secs), Average rate: 0.31 Mbit/sec (for 996.5secs) ========================================== Sender fail: (rank = 2): ------------------------------------------ ErrorCode: 0x3009AE0F ErrorString: Other MPI error, error stack: MPI_Send(173)..............: MPI_Send(buf=0x24768d0, count=2048, MPI_CHAR, dest=0, tag=50, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1827): Communication error with rank 0: Connection reset by peer ========================================== ========================================== Sender fail: (rank = 2): ------------------------------------------ ErrorCode: 0x3021AE0F ErrorString: Other MPI error, error stack: MPI_Send(173).............: MPI_Send(buf=0x24768d0, count=2048, MPI_CHAR, dest=0, tag=50, MPI_COMM_WORLD) failed MPIDI_EagerContigSend(186): failure occurred while attempting to send an eager message MPIDI_CH3_iStartMsgv(44)..: Communication error with rank 0 ========================================== ========================================== Sender fail: (rank = 2): ------------------------------------------ ErrorCode: 0x3099AE0F ErrorString: Other MPI error, error stack: MPI_Send(173).............: MPI_Send(buf=0x24768d0, count=2048, MPI_CHAR, dest=0, tag=50, MPI_COMM_WORLD) failed MPIDI_EagerContigSend(186): failure occurred while attempting to send an eager message MPIDI_CH3_iStartMsgv(44)..: Communication error with rank 0 ========================================== Last Sent number = 2100000 End of slave subroutine