We study bi-rational Weyl group actions on certain matrix Lax operators given in factorized form. These actions generalize the W(A[(1)][m-1] x A[(1)][n-1]) symmetry considered before by Kajiwara et.al. Our study is motivated by two recent developments: one is on discrete isomonodromic systems and the other is on the bi-rational Weyl group actions arising from the quiver mutations. We also discuss further generalizations using the results by G. Frieden on the geometric crystals.