Workers are assigned to groups in only one place... either their local qb.conf or the supervisor's central qbwrk.conf (one or the other, not both). If you want to ensure that a worker is only in one group, only assign it to one group. It's as simple as that.
If you're trying to restrict access to the worker such that it *only* accepts jobs sent to X, then you would want to set a worker_restriction & use clusters instead of groups.
example:
worker_restrictions = /test
worker_cluster = /test
then when you submit a job, select the '/test' Priority Cluster in the submission dialog.
See section 10.3.26 in the Administration Guide for more details on restrictions.
http://pipelinefx.com/docs/