supervisor and mysqld

[jesse]

Just today we had our supervisor become unavailable.

I noticed that mysqld is running at about 120% on our dualcore supervisor box.
the only thing this box does is run supervisor.

When the supervisor became unavailable the qube GUI and any submission tools we had either froze or stopped updating.

I had everyone close the gui and performed a server reboot.

It came up at, and myslqd immediately started running at 120% again.  Normally I only seem mysql burst to that level.  This has been running hard for a great number of minutes.

After the reboot, I eventually got a single instance of the gui to run and found that qube was still processing the queue, and picked up right where it left off.

We had about 10 jobs queue with an average frame length of 200 frames.  The supervisor server did not touch it's swap memory.  There are about 9900 jobs stored in mysql.
The logs showed some connection issues:

ERROR: unable to handle connection overflow - spawned max threads: 32 rejected host:

What might have caused this problem?  I set the max threads on the supervisor to 64.  I Waited for qube supervisor to "calm down".  This seemed to get us past this bump, but I suspect this will happen again if we can not identify the problem.

Any assistance would be greatly appreciated.
Thanks,
jesse

[jason.fowler]

Hi Jesse,

Sorry for the delay, but here are some suggestions to try - first, like you already did, you can increase the maximum threads on the supervisor even more, perhaps to 128.  Just note that the more threads, the more communication can be handled, but the more resource-intensive the supervisor service becomes.

Additionally, if you are able to remove any old jobs, that should increase performance.  Finally, we have made a number of performance-enhancing changes to our Qube 5.3 version supervisor, so if you have not yet upgraded from 5.2, you might consider that.

Hope this helps!

-Jason
PipelineFX

[chantk]

Jesse, our supervisor suffered with the same problem as yours. Tons of "spawned max threads" messages.

The problem may due to MySQL not being able to respond fast enough to the threads. You may want to try optimizing and/or repair the tables:

# mysqlcheck -orv qube

or simply all tables:

# mysqlcheck -Aovr