Hey dyamash1,
Not every renderer maps well into the standard maya renderer option list. Generally output format is better controlled using the render globals from within maya. Especially if you are using the non-native Renderer such as Mental Ray, or Gelato. If an output such as layered psd files are what you use, which I suspect you use specifically to pipe into After Effects since Adobe likes Adobe file formats, then you're best off setting the file format in your scene file before dispatching the render.
Also the Maya update is now available as is the new version of qube!. As a quick note, this is one of our rare updates where you must upgrade qube! in order to use the maya update. So please make sure that you are running qube 4.0-5 before attempting to use the maya job type.
Thanks,
Anthony