qb(un)block can't find jobs when called through a python script

[mrdebout]

Hello
i searched this forum and qube's documentation but i couldn't find anything remotedly addressing this problem

So what happens is quite simple:
I have a blocked job waiting for several events to occur before it gets unblocked. Let's call this job "job id 1234"
I have another job which consists of a python script that does several actions before calling qbunblock in order to release "job id 1234".

What happens is that, on the same machine, when i launch the python script in a shell "job id 1234" gets unblocked, but when it is executed as a qube job the unblocking fails.


Of course i have tried launching the qube job with export_environment as flagstring and passing the environment from the python script to the qbunblock subprocess, as a matter of fact i fiddled with the environment and all the different methods of launching a subprocess in python i could think of (and then some more)

I have attached a python script that you can use to try to replicate this quirk.
This script does a qbjobs --xml (to test if we see "job id 1234"), a qbunblock (what i need) and a qbblock (so we don't have to go through the setup everytime we want to test this)
The script calls these qbtools with as job id the first parameter you pass to the script

ie: test.py 1234
calls qbjobs --xml 1234 && qbunblock 1234 && qbblock 1234

these are the outputs for both shell and qube execution of the script (qbjobs --xml truncated for obvious reasons)

SHELL:

<object type = 'job_list'>
        <item>
                <object type = 'job'>
                        <id>12301</id>
...
STD_OUT: unblocked work: nothing
unblocked job: 12301.0

STD_OUT: blocked work: nothing
blocked job: 12301.0

QUBE:
<object type = 'job_list'>
        <item>
                <object type = 'job'>
                        <id>12301</id>
STD_OUT: unblocked work: nothing
unblocked job: nothing

STD_OUT: blocked work: nothing
blocked job: nothing


So we see right here that even if qbjobs could get the info for job 1234 it couldn't block/unblock it. If it hadn't found the job, at least we would have something consistent, but it isn't even the case.
Oh and the two modes of execution output the same environment though qbjobs --xml

Anybody got a clue ?
Thanks in advance

Deboute Benjamin
Attitude-Studio

[attachment deleted by admin]

[njushchyshyn]

Could this be a permissions issue?
If you are the user that submitted the blocked job ... you have permission to unblock your job, so doing so with python should work.

... but when a JOB runs that python code, it may be running as the qubeproxy user ... which maybe doesn't have the authority to unblock someone else's (your) job.

I can't be sure that this is the issue ... but it's certainly one of the first things I'd probably look into.

[mrdebout]

You know what ? This idea looks quite promising to me, i'm gonna take a look

[Scot Brew]

If you are just looking to unblock a job when another one completes, then you should look at the "dependency" field.  This is exposed from the qubegui job submission dialog and also from python.

For example, one can set it to unblock when jobs 1234 and 5678 complete
    dependency = link-complete-job-1234,link-complete-job-5678

If you need it to do more than that, then you should continue down the callback route you are on.  Yes, checking on the qube user permissions would be advisable as the callback is executed on the supervisor machine.

[mrdebout]

Yes, I managed to make do by assigning the qubeproxy user as owner of the job i wanted to unblock.

Thanks for your input Scot, but alas it was not a situation that could be handled by the dependency field as i have to implement parallel and serial runs inside a given group. The python script i talked about checks a separate database to see if all the parallel jobs are finished before unblocking an eventual serie of serial jobs

Thanks to all in poiting me toward what i had neglected