The visual impression of fabric is important in determining and communicating its quality. Many researchers have studied the visual texture and aesthetics of fabrics, naturally by using fabric materials. However, textile designers often draw inspiration from other materials, then design fabrics by unveiling the nature of the materials’ rich texture from the view of general design elements and principles. This study aimed to provide empirical support for the validity of such a cross-material design strategy. The visual textural impression was compared between 17 woven fabrics and 10 tree barks in two psychological experiments. Participants were asked to sort the fabrics based on their visual textural resemblance and match them to images of barks. The participants were then asked to rank the fabrics and barks separately according to seven bipolar adjective pairs related to design principles. Results from visual matching showed that the 17 fabrics and 10 bark images formed three distinct groups. The second experiment showed that the groupings predicted from the ranking data were consistent with those found by the visual matching. This study demonstrates that the design elements and principles used underlie the perception of visual texture commonly across different materials, strengthening the effectiveness of cross-material design.