By studying the maximal ordered abelian quotient of bi-ordered groups, we show that Clay-Rolfsen's obstruction of bi-orderability, which uses the classical Alexander polynomial, is not strengthened using the twisted Alexander polynomials for a Q-linear representation having a finite image. This contrasts with many known applications of the Alexander polynomial.