The GLSL Tutorial is now available in the new layout. Check it out here.
Core GLSL Tutorial (GLSL version >= 3.3) is also available here.
The gl_NormalMatrix is present in many vertex shaders. In here some light is shed on what is this matrix and what is it for. This section was inspired by the excellent book by Eric Lengyel "Mathematics for 3D Game Programming and Computer Graphics".
Many computations are done in eye space. This has to do with the fact that lighting needs to be performed in this space, otherwise eye position dependent effects, such as specular lights would be harder to implement.
Hence we need a way to transform the normal into eye space. To transform a vertex to eye space we can write:
vertexEyeSpace = gl_ModelViewMatrix * gl_Vertex;
So why can't we just do the same with a normal vector? First a normal is a vector of 3 floats and the modelview matrix is 4x4. This could be easily overcome with the following code:
normalEyeSpace = vec3(gl_ModelViewMatrix * vec4(gl_Normal,0.0));
So, gl_NormalMatrix is just a shortcut to simplify code writing? No, not really. The above line of code will work in some circunstances but not all.
Lets have a look at a potential problem:
In the above figure we see a triangle, with a normal and a tangent vectors. The following figure shows what happens when the modelview matrix contains a non-uniform scale.
Note: if the scale was uniform, then the direction of the normal would have been preserved, The length would have been affected but this can be easily fixed with a normalization.
In the above figure the modelview matrix was applied to all the vertices as well as to the normal and the result is clearly wrong: the normal is no longer perpendicular to the surface.
So now we know that we can't apply the modelview in all cases to transform the normal vector. The question is then, what matrix should we apply?
We know that, prior to the matrix transformation T.N = 0, since the vectors are by definition perpendicular. We also know that after the transformation N'.T' must remain equal to zero, since they must remain perpendicular to each other. Let's assume that the matrix G is the correct matrix to transform the normal vector. T can be multiplied safely by the upper left 3x3 submatrix of the modelview (T is a vector, hence the w component is zero). This is because T can be computed as the difference between two vertices, therefore the same matrix that is used to transform the vertices can be used to transform T. Hence the following equation:
The dot product can be transformed into a product of vectors, therefore:
Note that the transpose of the first vector must be considered since this is required to multiply the vectors. We also know that the transpose of a multiplication is the multiplication of the transposes, hence:
We started by stating that the dot product between N and T was zero, so if the following equation is true then we are on the right track.
Applying a little algebra yieds
Therefore the correct matrix to transform the normal is the transpose of the inverse of the M matrix. OpenGL computes this for us in the gl_NormalMatrix.
In the beginning of this section it was stated that using the modelview matrix would work in some cases. Whenever the 3x3 upper left submatrix of the modelview is orthogonal we have:
This is because with an orthogonal matrix, the transpose is the same as the inverse. So what is an orthogonal matrix? An orthogonal matrix is a matrix where all columns/rows are unit length, and are mutually perpendicular. This implies that when two vectors are multiplied by such a matrix, the angle between them after transformation by an orthogonal matrix is the same as prior to that transformation. Simply put the transformation preserves the angle relation between vectors, hence normals remain perpendicular to tangents! Furthermore it preserves the length of the vectors as well.
So when can we be sure that M is orthogonal? When we limit our geometric operations to rotations and translations, i.e. when in the OpenGL application we only use glRotate and gl_Translate and not glScale. These operations guarantee that M is orthogonal. Note: gluLookAt also creates an orthogonal matrix!