opengl es - Transforming Normals in webgl -

i have 2 point clouds normals. before rendering points on webgl canvas, need transform pointcloud1 same coordinate space pointcloud2 , have transformation matrix, t, (rot,scale,trans) works points. want same normals. after reading tutorials, tried this:

n' = transpose(inverse(t))*n

i understand math behind tutorials mention should take upper 3*3 matrix in t. not sure if should

1. multiply upper 3*3 t n = [a b c] or
2. multiply 4*4 t n = [a b c 0].

if use second option, n' = [a' b' c' d']. should divide vector d' make homogeneous?

since use rotations, translations, , scaling, don't need worry whole inverse-transpose stuff. @ least long "scaling", mean uniform scaling. if use non-uniform scaling, normal transformation need special consideration (see answer here background on non-uniform scaling: normal matrix non uniform scaling).

it true inverse-transpose of top-left 3x3 sub-matrix used transforming normals. in case, top-left 3x3 sub-matrix contains rotation , scaling. inverse-transpose of rotation , scaling matrices same original matrix. whole inverse-transpose calculation no-op.

well, strictly speaking, inverse-transpose of uniform scaling matrix not same original matrix. it's still uniform scaling matrix, means difference length of resulting normal vector. since have re-normalize normals anyway after transforming them scaling involved, still doesn't make practical difference.

whether explicitly create separate 3x3 matrix normals, or use regular 4x4 matrix on normal vector extended 0 component, largely matter of preference. mathematically, result same. advantage of using 4x4 matrix it's simpler not pass 3x3 matrix shader. downside it's potentially less efficient, since you're performing multiplication larger matrix necessary.

actually, nicest approach imho pass separate 3x3 normal matrix shader, , make contain rotation part of transformation, not scaling. way, can skip step have re-normalize normals after applying transformation, since pure rotation leaves length of vector unchanged.

on last part of question: if use 4x4 matrix, not divide 4th component of resulting vector. 4th component zero, , don't want divide zero...