Elapsed Time Not Correct

[mich007]

Conundrum: I need to be able to restart a rendering frame if it gets caught in a perpetual loop (hung up), meaning, instead of completing it just renders for ever.

If I migrate frames that are hung up, frame time continues to increase, which makes it difficult to impossible to keep track of how long a frame has been running since migrated.

If I kill and retry frames the elapsed time is being reset to zero for the entire job and subjob, great for keeping track of when I last reset frame but...

I still need to know how long job and subjobs have been running, the total elapsed time.

Example

Subjob(s) Status:
        0: complete - render0304 (172.16.9.104) 00:19:B9:F7:08:6E (0 s)
        1: complete - render0528 (172.16.11.28) 00:26:B9:F8:FB:14 (0 s)
        2: complete - render0262 (172.16.9.62) 00:1D:09:F2:BF:45 (0 s)
        3: complete - render0193 (172.16.8.193) 00:1D:09:F2:9C:22 (0 s)
        4: complete - render0210 (172.16.9.10) 00:1D:09:F2:B5:90 (0 s)
        5: complete - render0247 (172.16.9.47) 00:1D:09:F2:9A:FB (0 s)
        6: complete - render0087 (172.16.8.87) 00:1D:09:F2:3C:E1 (0 s)
        7: complete - render0030 (172.16.8.30) 00:1D:09:F2:46:3C (0 s)
        8: complete - render0042 (172.16.8.42) 00:1D:09:F2:30:DE (0 s)
        9: complete - render0661 (172.16.11.161) 00:26:B9:F9:42:BC (0 s)

Work Status:
        [1] 101: complete - 1238439.0 render0518 (172.16.11.18) 00:26:B9:F8:E8:F8 (6.9 m)
        [2] 102: complete - 1238439.1 render0217 (172.16.9.17) 00:1D:09:F2:9F:92 (11.9 m)
        [3] 103: complete - 1238439.2 render0299 (172.16.9.99) 00:1D:09:F2:7C:38 (11.3 m)
        [4] 104: complete - 1238439.3 render0142 (172.16.8.142) 00:1D:09:F2:AE:06 (11.3 m)
        [5] 105: complete - 1238439.4 render0039 (172.16.8.39) 00:1D:09:F2:53:0C (14.0 m)
        [6] 106: complete - 1238439.5 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (8.7 m)
        [7] 107: complete - 1238439.6 render0330 (172.16.9.130) 00:19:B9:F7:1D:D6 (11.7 m)
        [8] 108: complete - 1238439.7 render0611 (172.16.11.111) 00:26:B9:F9:34:B4 (8.7 m)
        [9] 109: complete - 1238439.8 render0214 (172.16.9.14) 00:1D:09:F2:95:06 (11.8 m)
        [10] 110: complete - 1238439.9 render0175 (172.16.8.175) 00:15:C5:FF:29:3B (11.7 m)
        [11] 111: complete - 1238439.0 render0518 (172.16.11.18) 00:26:B9:F8:E8:F8 (6.8 m)
        [12] 112: complete - 1238439.5 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (6.9 m)
        [13] 113: complete - 1238439.7 render0611 (172.16.11.111) 00:26:B9:F9:34:B4 (6.8 m)
        [14] 114: complete - 1238439.1 render0217 (172.16.9.17) 00:1D:09:F2:9F:92 (10.9 m)
        [15] 115: complete - 1238439.3 render0142 (172.16.8.142) 00:1D:09:F2:AE:06 (10.8 m)
        [16] 116: complete - 1238439.0 render0518 (172.16.11.18) 00:26:B9:F8:E8:F8 (6.8 m)
        [17] 117: complete - 1238439.8 render0214 (172.16.9.14) 00:1D:09:F2:95:06 (10.7 m)
        [18] 118: complete - 1238439.4 render0039 (172.16.8.39) 00:1D:09:F2:53:0C (12.8 m)
        [19] 119: complete - 1238439.5 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (6.6 m)
        [20] 120: complete - 1238439.7 render0611 (172.16.11.111) 00:26:B9:F9:34:B4 (6.8 m)
        [21] 121: complete - 1238439.0 render0518 (172.16.11.18) 00:26:B9:F8:E8:F8 (6.9 m)
        [22] 122: complete - 1238439.3 render0142 (172.16.8.142) 00:1D:09:F2:AE:06 (10.9 m)
        [23] 123: complete - 1238439.1 render0217 (172.16.9.17) 00:1D:09:F2:9F:92 (10.7 m)
        [24] 124: complete - 1238439.5 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (6.8 m)
        [25] 125: complete - 1238439.7 render0611 (172.16.11.111) 00:26:B9:F9:34:B4 (6.9 m)
        [26] 126: complete - 1238439.0 render0518 (172.16.11.18) 00:26:B9:F8:E8:F8 (6.7 m)
        [27] 127: complete - 1238439.2 render0215 (172.16.9.15) 00:1D:09:F2:8C:46 (21.7 m)
        [28] 128: complete - 1238439.8 render0615 (172.16.11.115) 00:26:B9:F9:2F:C8 (6.6 m)
        [29] 129: complete - 1238439.2 render0262 (172.16.9.62) 00:1D:09:F2:BF:45 (11.3 m)
        [30] 130: complete - 1238439.9 render0575 (172.16.11.75) 00:26:B9:F9:39:58 (17.4 m)
        [31] 131: complete - 1238439.1 render0217 (172.16.9.17) 00:1D:09:F2:9F:92 (11.1 m)
        [32] 132: complete - 1238439.3 render0142 (172.16.8.142) 00:1D:09:F2:AE:06 (11.0 m)
        [33] 133: complete - 1238439.8 render0615 (172.16.11.115) 00:26:B9:F9:2F:C8 (6.8 m)
        [34] 134: complete - 1238439.0 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (7.2 m)
        [35] 135: complete - 1238439.1 render0104 (172.16.8.104) 00:1D:09:F2:54:DD (11.7 m)
        [36] 136: complete - 1238439.2 render0262 (172.16.9.62) 00:1D:09:F2:BF:45 (11.1 m)
        [37] 137: complete - 1238439.3 render0193 (172.16.8.193) 00:1D:09:F2:9C:22 (11.0 m)
        [38] 138: complete - 1238439.4 render0210 (172.16.9.10) 00:1D:09:F2:B5:90 (12.6 m)
        [39] 139: complete - 1238439.5 render0327 (172.16.9.127) 00:19:B9:F7:2D:26 (11.3 m)
        [40] 140: complete - 1238439.6 render0502 (172.16.11.2) 00:26:B9:F8:FB:10 (8.3 m)
        [41] 141: complete - 1238439.7 render0030 (172.16.8.30) 00:1D:09:F2:46:3C (24.8 m)
        [42] 142: complete - 1238439.8 render0050 (172.16.8.50) 00:1D:09:F2:32:CD (14.1 m)
        [43] 143: complete - 1238439.0 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (7.2 m)
        [44] 144: complete - 1238439.9 render0322 (172.16.9.122) 00:19:B9:F7:23:B2 (11.7 m)
        [45] 145: complete - 1238439.1 render0104 (172.16.8.104) 00:1D:09:F2:54:DD (11.9 m)
        [46] 146: complete - 1238439.0 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (7.8 m)
        [47] 147: complete - 1238439.3 render0193 (172.16.8.193) 00:1D:09:F2:9C:22 (11.4 m)
        [48] 148: complete - 1238439.5 render0535 (172.16.11.35) 00:26:B9:F9:42:E8 (6.9 m)
        [49] 149: complete - 1238439.4 render0210 (172.16.9.10) 00:1D:09:F2:B5:90 (12.2 m)
        [50] 150: complete - 1238439.6 render0087 (172.16.8.87) 00:1D:09:F2:3C:E1 (11.5 m)
        [51] 151: complete - 1238439.0 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (7.3 m)
        [52] 152: complete - 1238439.5 render0535 (172.16.11.35) 00:26:B9:F9:42:E8 (6.6 m)
        [53] 153: complete - 1238439.2 render0262 (172.16.9.62) 00:1D:09:F2:BF:45 (10.9 m)
        [54] 154: complete - 1238439.3 render0193 (172.16.8.193) 00:1D:09:F2:9C:22 (11.0 m)
        [55] 155: complete - 1238439.1 render0568 (172.16.11.68) 00:26:B9:F9:3D:08 (6.8 m)
        [56] 156: complete - 1238439.0 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (7.3 m)
        [57] 157: complete - 1238439.8 render0322 (172.16.9.122) 00:19:B9:F7:23:B2 (11.0 m)
        [58] 158: complete - 1238439.9 render0561 (172.16.11.61) 00:26:B9:F9:42:C8 (10.4 m)
        [59] 159: complete - 1238439.4 render0210 (172.16.9.10) 00:1D:09:F2:B5:90 (12.4 m)
        [60] 160: complete - 1238439.5 render0535 (172.16.11.35) 00:26:B9:F9:42:E8 (7.3 m)
        [61] 161: complete - 1238439.7 render0030 (172.16.8.30) 00:1D:09:F2:46:3C (13.7 m)
        [62] 162: complete - 1238439.6 render0087 (172.16.8.87) 00:1D:09:F2:3C:E1 (12.5 m)
        [63] 163: complete - 1238439.1 render0568 (172.16.11.68) 00:26:B9:F9:3D:08 (7.3 m)
        [64] 164: complete - 1238439.0 render0534 (172.16.11.34) 00:23:AE:FE:84:48 (7.7 m)
        [65] 165: complete - 1238439.2 render0262 (172.16.9.62) 00:1D:09:F2:BF:45 (12.8 m)
        [66] 166: complete - 1238439.5 render0535 (172.16.11.35) 00:26:B9:F9:42:E8 (7.7 m)
        [67] 167: complete - 1238439.3 render0193 (172.16.8.193) 00:1D:09:F2:9C:22 (12.3 m)
        [68] 168: complete - 1238439.9 render0561 (172.16.11.61) 00:26:B9:F9:42:C8 (7.8 m)
        [69] 169: complete - 1238439.8 render0322 (172.16.9.122) 00:19:B9:F7:23:B2 (12.2 m)
        [70] 170: complete - 1238439.4 render0210 (172.16.9.10) 00:1D:09:F2:B5:90 (12.7 m)
        [71] 171: complete - 1238439.1 render0528 (172.16.11.28) 00:26:B9:F8:FB:14 (7.9 m)
        [72] 172: complete - 1238439.6 render0087 (172.16.8.87) 00:1D:09:F2:3C:E1 (12.7 m)
        [73] 173: complete - 1238439.7 render0030 (172.16.8.30) 00:1D:09:F2:46:3C (13.0 m)

[jburk]

We have a feature in our roadmap to track "accumulated time", which will be the sum total of all "elapsed time".  So as jobs restart or retry, the ET will be reset to zero, but the accumulated time will accurately reflect the sum total of all time running on the farm.

We were hoping to get it in for our 6.0 release, but dropped in due to time constraints.  It's still on the roadmap; we're hoping to get it into the release branch within the next few months.

[mich007]

Awesome! thank you