Scene appearance is a mixture of light transport phenomena ranging from direct reflection to complicated effect such as inter-reflection and subsurface scattering. To decompose scene appearance into meaningful photometric components is very helpful in scene understanding and image editing. However, it has proven to be a difficult task. In this paper, we explore the difference of direct components obtained by multi-frequency illumination for light transport component decomposition. We apply independent vector analysis (IVA) to this task with no fixed constraints. Experiment results have verified the effectiveness of our method and its applicability to generic scenes.