This study was aimed to investigate the osseointegration ability of poly(ether ether ketone) (PEEK) implants with modified surface roughness and/or surface chemistry. The roughened surface was prepared by a sandblast method, and the phosphate groups on the substrates were modified by a two-step chemical reaction. The in vitro osteogenic activity of rat mesenchymal stem cells (MSCs) on the developed substrates was assessed by measuring cell proliferation, alkaline phosphatase activity, osteocalcin expression, and bone-like nodule formation. Surface roughening alone did not improve MSC responses. However, phosphorylation of smooth substrates increased cell responses, which were further elevated in combination with surface roughening. Moreover, in a rabbit tibia implantation model, this combined surface modification significantly enhanced the bone-to-implant contact ratio and corresponding bone-to-implant bonding strength at 4 and 8 weeks post-implantation, whereas modification of surface roughness or surface chemistry alone did not. This study demonstrates that combination of surface roughness and chemical modification on PEEK significantly promotes cell responses and osseointegration ability in a synergistic manner both in vitro and in vivo. Therefore, this is a simple and promising technique for improving the poor osseointegration ability of PEEK-based orthopedic/dental implants.