© 2014 SPE. All rights reserved. Wellbore instability is a major challenge for the engineer evaluating the borehole and formation conditions during drilling. Instability issue is especially important to understand in areas with high stress variations, significant structure anisotropy, or pre-existing fractures system. Borehole (in)stability is influenced by rock strength, structural properties, and the interaction with far and near field principal stresses. During drilling, the borehole conditions also impact borehole integrity. Factors that we can measure in the borehole during with logging while drilling (LWD) to understand these conditions include the mud weight, mud loss, ROP (Rate of Penetration), RPM (Rotation Per Minute), WOB (Weight on Bit), and TORQ (Power swivel torque value). We conducted borehole instability analysis for Site C0002A and C0002F of the Nankai Trough transect based on riser and riserless drilling during IODP Expedition 338. The borehole shape, determined from LWD resistivity images, indicated the most of drilling occurred in low stress level environments, however in a few instances the bottom hole assembly became stuck. We used our geomechanical model to evaluate the required drilling parameters. Which related to estimate rock strength and measure physical properties. Based on our analysis, we interpret that borehole instability during IODP Expedition 338 may have been caused by weak bedding plane and fluid overpressure state. Furthermore we suggested the optimized drilling processes for the future deep drilling based on the simulation results from this model.