Lessons learnt while writing shaders

I wrote more shader code over the past week than I ever did before that, and there are several things I learnt while doing so about that. I thought that might be useful to share that even if some are straightforward or just common sense.

First of all, PIX is your friend. I repeat: PIX is your friend! Make sure you can run this tool with your app and get the vertex and pixel shader debugging features working, this is invaluable... That being said, PIX is a little bit whimsical at times so you need to make sure that your app is complian with PIX, ie:

• Shaders are compiled in debug mode so you don't have to debug the assembly output but your actual code. This is done by passing D3DXSHADER_DEBUG | D3DXSHADER_SKIPOPTIMIZATION as flags to D3DXCompileShader(...). 
• Having 0 warnings while compiling your shaders improves stability
• If you experience troubles, you can try switching to softvare vertex processing or using the reference device (whis is going to be super slow but helps catching some bugs)

Next thing is about 3D maths and how it is done on computers - both CPU and GPU... There are many reasons why maths can go wrong down the chain if you're not careful. The kind of questions I tend to ask myself to try and prove me wrong and then eventually right when I write 3D maths includes:

• Which coordinate system this Matrix/Vector is into?
• Should I do Matrix * Vector or Vector * Matrix ??
• Should I use this Matrix or it's inverse? Transpose?
• Is it Z-up or Y-up?
• Is it a left handed or right handed coordinate system?
• In that float4, what should the W component be?

Regarding the last bullet point on homogenous coordinates, I tend to use float4 over float as much as I can. I find it clearer especially when vectors and positions are stored in textures. By doing so, you minimize the number of .xyz that suffixes your variable when doing float4->float3 or the float4( vec, 0 or 1) when you do float3->float4 ... So basically, I pull the needed data from the shader inpu and textures info float4s and then try to stick on working with those and not mixing types too much. I still need to investigate the performance implications but my early tests tend to show that the shader compiler is really good at optimizing all that.

I'll probably post separately about lots of 3D maths quirks so I'll have that reference material somewhere in a reliable place.

-m