Author Topic: qbsummary is off? (Read 8439 times)

mpursley · « **on:** April 13, 2012, 09:46:23 PM »

Hey All,

Looks like qbsummary is showing that we have 19291 pending frames with priority of 1000...
But, the qube gui only shows about 6000.

Also, qbsummary is showing 27833 frames in the /comp/vfx cluster...
But the Pending Load chart in the Qube GUI shows 4523.

Is there anything I can do to find out why qbsummary is off?
Or, how to fix it?

Thanks,
Matt

___________________
$ qbsummary --pvw | head -n 2 ; qbsummary --pvw | grep -i 1000
Title: priorityVwork
bad blo comple fail kil pendi run sus unk wai total
1000 0 0 38816 22 0 19185 0 0 0 0 58023

$ qbsummary --pvw
Title: priorityVwork
bad blo comple fail kil pendi run sus unk wai total
0 0 0 26 0 0 0 0 0 0 0 26
1 0 0 4809 0 0 459 3 0 0 0 5271
2 0 0 211 0 0 0 0 0 0 0 211
3 0 0 903 0 0 0 0 0 0 0 903
4 0 0 88 0 0 0 0 0 0 0 88
5 0 0 1383 0 0 0 0 0 0 0 1383
6 0 0 45 0 0 0 0 0 0 0 45
10 0 0 6461 1 0 44 0 0 0 0 6506
11 0 0 121 0 0 0 0 0 0 0 121
15 0 0 2798 4 0 108 0 0 0 0 2910
16 0 0 317 0 0 0 0 0 0 0 317
19 0 0 53 0 0 0 0 0 0 0 53
20 0 0 7044 2 1 615 0 0 0 0 7662
21 0 0 633 0 0 0 0 0 0 0 633
22 0 0 22 0 0 0 0 0 0 0 22
25 0 0 506 0 1 14 0 0 0 0 521
26 0 0 15 0 0 0 0 0 0 0 15
30 0 0 35113 0 0 185 20 0 0 0 35318
31 0 0 154 0 0 0 0 0 0 0 154
35 0 0 108 0 0 0 0 0 0 0 108
40 0 0 502 0 0 0 0 0 0 0 502
50 0 0 9753 0 0 1 20 0 0 0 9774
60 0 0 220 0 0 0 0 0 0 0 220
75 0 0 1857 0 0 0 0 0 0 0 1857
90 0 0 42 0 0 0 0 0 0 0 42
99 0 0 302 0 0 0 0 0 0 0 302
100 0 62 54872 1077 479 7479 0 0 0 0 63969
101 0 0 212 0 0 0 0 0 0 0 212
102 0 0 186 0 0 0 0 0 0 0 186
103 0 0 348 0 0 0 0 0 0 0 348
105 0 0 400 0 0 0 0 0 0 0 400
106 0 0 52 0 0 0 0 0 0 0 52
110 0 0 349 0 0 0 0 0 0 0 349
115 0 0 214 0 0 0 0 0 0 0 214
120 0 0 614 0 0 0 0 0 0 0 614
125 0 0 380 0 0 0 0 0 0 0 380
130 0 0 509 0 0 0 0 0 0 0 509
140 0 0 22 0 0 99 20 0 0 0 141
150 0 0 1918 0 0 0 0 0 0 0 1918
160 0 0 8 0 0 0 0 0 0 0 8
190 0 0 472 0 0 0 0 0 0 0 472
199 0 0 336 0 0 0 0 0 0 0 336
200 0 8 25436 1057 64 1836 0 0 0 0 28401
201 0 0 89 0 0 0 0 0 0 0 89
205 0 0 248 0 0 0 0 0 0 0 248
210 0 0 602 0 0 18 0 0 0 0 620
220 0 0 646 0 0 0 0 0 0 0 646
230 0 0 429 0 0 0 0 0 0 0 429
240 0 0 269 0 0 0 0 0 0 0 269
250 0 0 455 0 0 0 0 0 0 0 455
275 0 0 42 0 0 0 0 0 0 0 42
300 0 0 5339 261 0 739 0 0 0 0 6339
310 0 0 539 0 0 0 0 0 0 0 539
320 0 0 26 0 0 75 0 0 0 0 101
330 0 0 0 0 0 110 0 0 0 0 110
340 0 0 0 0 0 141 0 0 0 0 141
350 0 0 262 0 0 245 0 0 0 0 507
360 0 0 0 0 0 439 0 0 0 0 439
390 0 0 53 0 0 0 0 0 0 0 53
400 0 0 4533 0 0 0 0 0 0 0 4533
410 0 0 490 0 0 0 0 0 0 0 490
420 0 0 650 0 0 0 0 0 0 0 650
430 0 0 309 0 104 0 0 0 0 0 413
440 0 0 245 0 0 0 0 0 0 0 245
450 0 0 810 0 0 0 0 0 0 0 810
490 0 0 350 0 0 0 0 0 0 0 350
500 0 0 1287 0 0 0 0 0 0 0 1287
510 0 0 306 0 0 0 0 0 0 0 306
520 0 0 330 0 0 0 0 0 0 0 330
530 0 0 558 0 0 0 0 0 0 0 558
540 0 0 245 0 0 0 0 0 0 0 245
570 0 0 100 0 0 0 0 0 0 0 100
580 0 0 120 0 0 0 0 0 0 0 120
590 0 0 349 0 0 0 0 0 0 0 349
600 0 0 966 0 0 0 0 0 0 0 966
601 0 0 68 0 0 0 0 0 0 0 68
610 0 0 101 0 0 0 0 0 0 0 101
620 0 0 178 0 0 0 0 0 0 0 178
700 0 0 483 0 0 63 0 0 0 0 546
800 0 71 264 0 0 0 0 0 0 0 335
810 0 133 180 0 0 0 0 0 0 0 313
910 0 0 150 0 0 159 0 0 0 0 309
1000 0 0 38816 22 0 19291 0 0 0 0 58129
2000 0 0 1042 0 0 1 0 0 0 0 1043
3000 0 0 17290 601 0 4734 9 0 0 0 22634
3999 0 0 59 0 1 0 0 0 0 0 60
4000 0 0 3434 1 31 1810 24 0 0 0 5300
9999 0 0 0 0 0 0 0 0 0 0 0
total 0 274 242526 3026 681 38665 96 0 0 0 285268

TCSH 17:07 mpursley@megamind: ~/
$ qbsummary --cvw
Title: clusterVwork
bad blo comple fail kil pendi run sus unk wai total
/TEMBOTEST 0 0 5 0 0 0 0 0 0 0 5
/cg 0 0 216 0 0 0 0 0 0 0 216
/cg/feature 0 0 7658 1 31 2270 25 0 0 0 9985
/cg/feature, /cg/vfx 0 0 49 0 0 0 0 0 0 0 49
/cg/vfx 0 204 44095 0 105 7708 34 0 0 0 52146
/cg/vfx, /cg/feature 0 0 1272 0 0 0 0 0 0 0 1272
/cg/vfx,/cg/feature 0 0 460 0 0 0 0 0 0 0 460
/cg/vfx,/cg/feature,/comp/vfx 0 0 80 0 0 0 0 0 0 0 80
/cg/vfx,/comp 0 0 38 0 0 0 0 0 0 0 38
/comp/feature 0 0 3600 931 0 845 0 0 0 0 5376
/comp/vfx 0 70 176395 2063 545 27741 38 0 0 0 206852
/comp/vfx,/cg/feature,/cg/vfx 0 0 196 0 0 0 0 0 0 0 196
/comp/vfx,/cg/vfx 0 0 6 0 0 0 0 0 0 0 6
/comp/vfx,/cg/vfx,/cg/feature 0 0 321 0 0 0 0 0 0 0 321
/comp/vfx,/comp 0 0 98 0 0 0 0 0 0 0 98
/publish/feature 0 0 5954 9 0 4 0 0 0 0 5967
/shooter 0 0 0 0 0 0 0 0 0 0 0
/shooter/feature 0 0 1900 22 0 18 0 0 0 0 1940
/shooter/vfx 0 0 0 0 0 0 0 0 0 0 0
total 0 274 242343 3026 681 38586 97 0 0 0 285007

$ qbjobs --pending --user all
total: 0/332 cpu(s) 0/1702 work
% id pid pgrp label status user type name cpus priority cluster groups
0 57522 1 57522 qube1 pending ssmith cmdrange HUD__sh0640_stereo_check_wip_0004_stadium_l__MAYA 0/13 3000 /cg/vfx cg,comp-cg
0 57524 1 57524 qube1 pending ssmith cmdrange HUD__sh0640_stereo_check_wip_0004_chars_r__MAYA 0/13 3000 /cg/vfx cg,comp-cg
0 57526 1 57526 qube1 pending ssmith cmdrange HUD__sh0640_stereo_check_wip_0004_chars_l__MAYA 0/13 3000 /cg/vfx cg,comp-cg
0 57528 1 57528 qube1 pending ssmith cmdrange HUD__sh0640_stereo_check_wip_0004_stadium_r__MAYA 0/13 3000 /cg/vfx cg,comp-cg
0 57534 1 57534 qube1 pending ssmith cmdrange HUD__sh0650_stereo_check_wip_0006_stadium_l__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57536 1 57536 qube1 pending ssmith cmdrange HUD__sh0650_stereo_check_wip_0006_chars_r__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57538 1 57538 qube1 pending ssmith cmdrange HUD__sh0650_stereo_check_wip_0006_chars_l__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57540 1 57540 qube1 pending ssmith cmdrange HUD__sh0650_stereo_check_wip_0006_stadium_r__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57543 1 57543 qube1 pending ssmith cmdrange HUD__sh0660_stereo_check_wip_0006_stadium_l__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57545 1 57545 qube1 pending ssmith cmdrange HUD__sh0660_stereo_check_wip_0006_chars_r__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57547 1 57547 qube1 pending ssmith cmdrange HUD__sh0660_stereo_check_wip_0006_chars_l__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57549 1 57549 qube1 pending ssmith cmdrange HUD__sh0660_stereo_check_wip_0006_stadium_r__MAYA 0/20 3000 /cg/vfx cg,comp-cg
0 57629 1 57629 qube1 pending abradshaw cmdrange BALL__sh1210_lit_ad_reflection_wip_0003__MAYA 0/20 360 /cg/vfx cg
0 57638 1 57638 qube1 pending abradshaw cmdrange BALL__sh1240_lit_ad_reflection_wip_0003__MAYA 0/20 330 /cg/vfx cg
25 57642 1 57642 qube1 pending abradshaw cmdrange BALL__sh1260_lit_ad_reflection_wip_0002__MAYA 0/20 320 /cg/vfx cg
0 57660 1 57660 qube1 pending abradshaw cmdrange BALL__sh1275_lit_ad_reflection_wip_0003__MAYA 0/20 350 /cg/vfx cg
0 57664 1 57664 qube1 pending abradshaw cmdrange BALL__sh1280_lit_ad_reflection_wip_0003__MAYA 0/20 340 /cg/vfx cg
0 58058 1 58058 qube1 pending esanford cmdrange HUD__sh0610_lit_spotlights_wip_0001__MAYA 0/20 3000 /cg/vfx cg

[attachment deleted by admin]

jburk · « **Reply #1 on:** April 14, 2012, 01:00:30 AM »

The discrepancy is most likely coming from pending frames in jobs that are in a blocked terminal state, most likely failed or killed.

The charts filter out pending work from terminal and blocked jobs, which is why they are usually showing less pending work than qbsummary.

When you kill jobs via the QubeGUI, it does not (currently) kill the work associated with the job. This differs from the cmdline behavior, which kills both the job itself and all work for the job. This discrepancy led me to classify this GUI behavior as a bug, and GUI versions later than 6.3.4 will kill work with the job.

If you want to try and (temporarily) shrink this discrepancy and don't have 10's of 1000's of jobs, you can "kill all killed" jobs with the cmdline tool qbkill, which will kill the associated work:

qbkill -killed -u all 0

This can be an expensive operation if you have 10's of 1000's of jobs in Qube, so run it during a period when your supervisor is not heavily loaded, and your users are not present.

mpursley · « **Reply #2 on:** April 17, 2012, 10:34:12 PM »

Huh, ok. Is there an easy way to get qbsummary or qb.query.jobinfo to filter out pending work from terminal and blocked jobs, like the GUI does?

mpursley · « **Reply #3 on:** April 26, 2012, 04:41:26 PM »

Yes? No?

Thanks,
Matt

Quote from: mpursley on April 17, 2012, 10:34:12 PM

Huh, ok. Is there an easy way to get qbsummary or qb.query.jobinfo to filter out pending work from terminal and blocked jobs, like the GUI does?

mpursley · « **Reply #4 on:** May 08, 2012, 12:50:24 PM »

Or, how about this...

Is there an easy way to get qbsummary or qb.query.jobinfo to show only pending work from terminal and blocked jobs? If so, I can subtract that from the total and the same values that the GUI is showing...

Thanks,
Matt

Quote from: mpursley on April 17, 2012, 10:34:12 PM

Huh, ok. Is there an easy way to get qbsummary or qb.query.jobinfo to filter out pending work from terminal and blocked jobs, like the GUI does?

Author Topic: qbsummary is off? (Read 8439 times)

mpursley

qbsummary is off?

jburk

Re: qbsummary is off?

mpursley

Re: qbsummary is off?

mpursley

Re: qbsummary is off?

mpursley

Re: qbsummary is off?