rank = 19 hostname = woogie Iter = 4233 / 100000, Current rate: 3.22 Mbit/sec (for 20.5secs), Average rate: 3.22 Mbit/sec (for 20.5secs) Iter = 8355 / 100000, Current rate: 0.83 Mbit/sec (for 78.1secs), Average rate: 1.32 Mbit/sec (for 98.6secs) Iter = 9412 / 100000, Current rate: 0.33 Mbit/sec (for 50.2secs), Average rate: 0.99 Mbit/sec (for 148.8secs) Iter = 9850 / 100000, Current rate: 0.30 Mbit/sec (for 22.6secs), Average rate: 0.90 Mbit/sec (for 171.3secs) Iter = 10349 / 100000, Current rate: 0.36 Mbit/sec (for 21.5secs), Average rate: 0.84 Mbit/sec (for 192.9secs) Iter = 10813 / 100000, Current rate: 0.36 Mbit/sec (for 20.0secs), Average rate: 0.79 Mbit/sec (for 212.9secs) Iter = 11277 / 100000, Current rate: 0.35 Mbit/sec (for 21.0secs), Average rate: 0.75 Mbit/sec (for 233.9secs) Iter = 11720 / 100000, Current rate: 0.32 Mbit/sec (for 21.9secs), Average rate: 0.72 Mbit/sec (for 255.8secs) Iter = 12125 / 100000, Current rate: 0.28 Mbit/sec (for 22.9secs), Average rate: 0.68 Mbit/sec (for 278.7secs) Iter = 12480 / 100000, Current rate: 0.27 Mbit/sec (for 20.3secs), Average rate: 0.65 Mbit/sec (for 299.0secs) Iter = 12846 / 100000, Current rate: 0.27 Mbit/sec (for 21.0secs), Average rate: 0.63 Mbit/sec (for 319.9secs) Iter = 13220 / 100000, Current rate: 0.27 Mbit/sec (for 21.6secs), Average rate: 0.60 Mbit/sec (for 341.5secs) Iter = 13595 / 100000, Current rate: 0.26 Mbit/sec (for 22.2secs), Average rate: 0.58 Mbit/sec (for 363.8secs) Iter = 13969 / 100000, Current rate: 0.26 Mbit/sec (for 22.9secs), Average rate: 0.56 Mbit/sec (for 386.6secs) Iter = 14344 / 100000, Current rate: 0.25 Mbit/sec (for 23.5secs), Average rate: 0.55 Mbit/sec (for 410.1secs) Iter = 14664 / 100000, Current rate: 0.25 Mbit/sec (for 20.0secs), Average rate: 0.53 Mbit/sec (for 430.1secs) Iter = 14984 / 100000, Current rate: 0.24 Mbit/sec (for 20.5secs), Average rate: 0.52 Mbit/sec (for 450.6secs) Iter = 15297 / 100000, Current rate: 0.23 Mbit/sec (for 20.9secs), Average rate: 0.51 Mbit/sec (for 471.5secs) Iter = 15597 / 100000, Current rate: 0.22 Mbit/sec (for 21.3secs), Average rate: 0.49 Mbit/sec (for 492.8secs) Iter = 15904 / 100000, Current rate: 0.22 Mbit/sec (for 21.7secs), Average rate: 0.48 Mbit/sec (for 514.6secs) Iter = 16216 / 100000, Current rate: 0.22 Mbit/sec (for 22.2secs), Average rate: 0.47 Mbit/sec (for 536.7secs) Iter = 16528 / 100000, Current rate: 0.22 Mbit/sec (for 22.6secs), Average rate: 0.46 Mbit/sec (for 559.3secs) Iter = 16840 / 100000, Current rate: 0.21 Mbit/sec (for 23.0secs), Average rate: 0.45 Mbit/sec (for 582.4secs) Iter = 17152 / 100000, Current rate: 0.21 Mbit/sec (for 23.4secs), Average rate: 0.44 Mbit/sec (for 605.8secs) Iter = 17464 / 100000, Current rate: 0.20 Mbit/sec (for 23.8secs), Average rate: 0.43 Mbit/sec (for 629.6secs) Iter = 17777 / 100000, Current rate: 0.20 Mbit/sec (for 24.2secs), Average rate: 0.42 Mbit/sec (for 653.8secs) Iter = 18089 / 100000, Current rate: 0.20 Mbit/sec (for 24.9secs), Average rate: 0.42 Mbit/sec (for 678.7secs) Iter = 18340 / 100000, Current rate: 0.18 Mbit/sec (for 21.6secs), Average rate: 0.41 Mbit/sec (for 700.3secs) Iter = 18588 / 100000, Current rate: 0.17 Mbit/sec (for 23.3secs), Average rate: 0.40 Mbit/sec (for 723.6secs) Iter = 18838 / 100000, Current rate: 0.16 Mbit/sec (for 24.4secs), Average rate: 0.39 Mbit/sec (for 748.0secs) ========================================== MPI_Send Recognized error rank - 19 iter = 18900, fail: ------------------------------------------ ErrorCode: 0x3809AE0F ErrorString: Other MPI error, error stack: MPI_Send(173)..............: MPI_Send(buf=0x11258d0, count=2048, MPI_CHAR, dest=0, tag=100, MPI_COMM_WORLD) failed MPID_nem_tcp_connpoll(1827): Communication error with rank 0: Connection reset by peer ==========================================