Photoinduced growth of the antiferromagnetic state in the ferrimagnetic phase were demonstrated in double layered Fe oxide LuFe2O4 by 12 fs infrared pulse under inter-layer excitation. Inter-layer charge imbalance successively induces the changes of charge/magnetic structures interacting with local Fe-O stretching and inter-layer sliding phonons through the exchange interaction.