How to regex query job names, and email on completion?

[seven6ty]

Hi there!  So I'm writing a standalone PyQT application, that allows a user to batch modify priorities for jobs, based upon usernames, job statuses, and certain strings being present in the job names that are running on the farm.  (I'm kind of surprised something like this isn't already built in?)

For the username, I know I can retrieve that user's jobs on the farm with the following:
    qb.jobinfo( filters={'user': userName})
And for all jobs with a given status, I know I can do something like this:
    qb.jobinfo( filters={status='running'})

But what about if I want to search for certain strings being present in the job names?  (As a bonus, it would be awesome if there was a way to match regex's, but that's not totally necessary.)  I know that I can get a listing of all jobs, and then iterate over each one of the job names trying to find a match, but that's terribly slow and inefficient.  Surely there has to be a method built into the API that will find all jobs containing a given string in the name, yes?

Also, I would like to be able to have the option of using python to send an email, upon the completion of a job.  Would anyone happen to know how I might go about doing that?  I don't need to know the code for generating the email, just how to go about executing a command (for a job that is already on the farm), upon job completion.

Thanks!

[BrianK]

Howdy.

You are using the qb module.  Assuming you have mysql-python (the MySQLdb module), there is also a qb.query module you can use.  The qb.query module has a jobinfo function that extends qb.jobinfo.

Code: [Select]

Definition: qb.query.jobinfo(fields=[], filters={}, id=None, status=None, agenda=False, subjobs=False, callbacks=False, sqlserver='', sqluser='qube_readonly', updatedAfter=None, updatedBefore=None, submittedAfter=None, submittedBefore=None, limit=0, offset=0, basicFields=['*'], sortOrder='ASC', sortKey='id', where=None, dbConn=None)
Docstring:
sortOrder: ascending/descending order for sorted values. options=['ASC', 'DESC'] (default='ASC')
where: an optional "where" statement for mysql, e.g. "name like 'foo' OR user like 'foo'"

Notice there are several new kwargs, most importantly there is the "where" that can be used to to do wildcard searches on any column in the database.

As for emails, you have 2 options:  You can set the email settings on the supervisor and adjust our mail templates to send email upon certain triggers, or you can simply create your own python callback.  You will need to enable to python callback language on the supervisor using supervisor_language_flags (python is off by default).
mail template example: http://docs.pipelinefx.com/x/34F
supervisor_language_flags: http://docs.pipelinefx.com/x/84F
callbacks: http://docs.pipelinefx.com/x/SYJ
Knowledge base on callbacks: http://docs.pipelinefx.com/x/dIN

[seven6ty]

Thanks Brian!  Great info to have.  I'm not entirely sure I follow you though?  If I run the following:

import qb
qb.query.jobinfo( where='somestring' )

It says the module has no query attribute.  I can't seem to import mysql, MySQLdb, or mysql-python.  Could you possibly explain how I would go about utilizing those? Would I have to download that from somewhere?

Also, would you happen to have an example for the second item, for sending email based upon triggers?  Thanks!

[BrianK]

For qb.query, you have to import it explicitly:

Code: [Select]

In [1]: import qb.query

In [2]: len(qb.query.jobinfo(where='user like "brian%" and name like "%test%"'))
Out[2]: 310

In [3]: len(qb.query.jobinfo(where='user = "briank" and name like "%test%"'))
Out[3]: 310


As far as callbacks, if you're looking for a mail callback, you can find an example here: http://www.pipelinefx.com/forum/index.php?topic=670.0
If you're looking for a python callback, see John's response and example here: http://www.pipelinefx.com/forum/index.php?topic=1498.msg3619#msg3619

[seven6ty]

Okay, a couple of questions.

1.  Could you please give me the Python commands to check if python and email are enabled on the supervisor, plus the commands I'd need to have my admin run to enable them?

2.  If I'm modifying a jobID that is already on the render farm, is there much difference, besides using the qb.modify() command, instead of qb.submit?

[BrianK]

Okay, a couple of questions.

1.  Could you please give me the Python commands to check if python and email are enabled on the supervisor, plus the commands I'd need to have my admin run to enable them?

You could do a qb.supervisorconfig() and check supervisor_language_flags (you'll need to decode the binary - http://docs.pipelinefx.com/display/QUB065/supervisor_language_flags) to see if mail and python are turned on.

You could also run the system command (os.system, subprocess.Popen, etc) "qbadmin s --config | grep language" which will give you the language flags as strings:

BKMac:modules briank$ qbadmin s --config | grep language
29: supervisor_language_flags=113 (qube,mail,dependency,auto_wrangling)

(of course, on Windows, you can't use grep, so you'd have to parse the results, or try your hand at FINDSTR)

2.  If I'm modifying a jobID that is already on the render farm, is there much difference, besides using the qb.modify() command, instead of qb.submit?

I assume by "jobID", you mean "job", as you can't modify the actual id of a job.

In that case, it's pretty similar, other than you can only use one set of modification parameters at a time and you must provide the job id of the job you're modifying.  Also, you cannot modify every job parameter.  As of 6.5, these are the modifiable job attributes:

"priority"
"timeout"
"name"
"account"
"notes"
"cluster"
"restrictions"
"requirements"
"reservations"
"groups"
"hosts"
"omithosts"
"omitgroups"
"hostorder"
"pgrp"
"data"
"cpus"
"prod_show"
"prod_shot"
"prod_seq"
"prod_client"
"prod_dept"
"prod_custom1"
"prod_custom2"
"prod_custom3"
"prod_custom4"
"prod_custom5"

Example:

Code: [Select]

In [24]: qb.jobinfo(id=6671)[0].get("name")
Out[24]: 'cmdrange Job'

In [25]: qb.modify({"name":"new name"}, 6671)
Out[25]: ['6671']

In [26]: qb.jobinfo(id=6671)[0].get("name")
Out[26]: 'new name'

Note that the prod* attributes are new to 6.5 & won't exist in earlier versions.

In writing this response, I realized that the above list is not documented anywhere.  I'll be sure to add it to qb.modify's docstring for the next release.

[BrianK]

There may be better ways of decoding the binary, but here's a quick one:

Code: [Select]

In [50]: python_enabled = bool(int(bin(int(qb.supervisorconfig().get('supervisor_language_flags')))[-3]))

In [51]: mail_enabled = bool(int(bin(int(qb.supervisorconfig().get('supervisor_language_flags')))[-5]))

In [52]: python_enabled
Out[52]: False

In [53]: mail_enabled
Out[53]: True

there are probably better ways, but that certainly works. 

[seven6ty]

1.)  Okay, not totally following the second part, but when I run "qbadmin s --config | grep language" in a shell, I get the following:
12: supervisor_language_flags=63

I didn't see 63 listed in the directory you gave?  Mine didn't print out a list of langauges within parenthesis, like your example?

I'm thinking it isn't enabled.  In that case, would you happen to know what commands I could instruct our engineering guys to run to enable these, or a page to point them to?

2.)  Correct, I'm not trying to alter the JobID's, but parameters within that jobID.  I'm trying something like this, to get it to send an email upon completion:
qb.modify( jobid=<jobIDNum>, modifyParamDict={ 'callbacks':[{'language':'python', 'triggers':'done-job-self', 'code':str('print "\nhi!\n"')}]} )

Or failing that, something like this:
qb.modify jobid=<jobIDNum>, modifyParamDict={'mailaddress':<mailAddress>, 'callbacks':[{'triggers':'complete-job-self', 'language':'mail'}]} )

But it sounds like you're saying both of these won't work, since I can only modify one aspect of the modifyParamDict then?  Gotcha.  Seems like I'll need to have one of our Engineers get the languages straightened out before I can try that though.

Thanks!

[seven6ty]

Oh, and I noticed that I don't see "callbacks" as a modifiable job parameter...  So I take it this can't be modified after the job has been launched??? 

[BrianK]

1.)  Okay, not totally following the second part, but when I run "qbadmin s --config | grep language" in a shell, I get the following:
12: supervisor_language_flags=63

I didn't see 63 listed in the directory you gave?  Mine didn't print out a list of langauges within parenthesis, like your example?

I'm thinking it isn't enabled.  In that case, would you happen to know what commands I could instruct our engineering guys to run to enable these, or a page to point them to?

I'm guessing that you're on an older version of Qube if you're not seeing the strings.

Have a look at the page I linked & also the code examples of how to decode the language flags number.

Your number is 63.  63 in binary is 111111.  That means every language flag is turned on, so there's no need for anyone to change anything.

Just so it's here, however, for it to be changed, an admin would need to edit qb.conf on the supervisor - either by editing the file directly or by using the WranglerView's Administration > "Configure Local Host" option.  They need to change supervisor-language_flags - just changing the text.  http://docs.pipelinefx.com/x/84F

2.)  Correct, I'm not trying to alter the JobID's, but parameters within that jobID.  I'm trying something like this, to get it to send an email upon completion:
qb.modify( jobid=<jobIDNum>, modifyParamDict={ 'callbacks':[{'language':'python', 'triggers':'done-job-self', 'code':str('print "\nhi!\n"')}]} )

Or failing that, something like this:
qb.modify jobid=<jobIDNum>, modifyParamDict={'mailaddress':<mailAddress>, 'callbacks':[{'triggers':'complete-job-self', 'language':'mail'}]} )

But it sounds like you're saying both of these won't work, since I can only modify one aspect of the modifyParamDict then?  Gotcha.  Seems like I'll need to have one of our Engineers get the languages straightened out before I can try that though.

Oh, and I noticed that I don't see "callbacks" as a modifiable job parameter...  So I take it this can't be modified after the job has been launched???  Sad

I chose my words poorly.  You can modify multiple parameters at a time and you can modify multiple jobs at a time, but you can't modify multiple different parameters of multiple different jobs at the same time - it would take multiple calls to qb.modify.  In other words, qb.modify({"name":"new name","priority":5000}, 6671,6672) is valid.

That being said, no, you cannot modify callbacks.