Kernel hardness influences the milling and baking quality of wheat. Stress environments such as heat and combined heat-drought can produce harder kernels, thereby affecting the overall wheat quality. Beside puroindoline genes that are known to determine hardness, other QTLs contribute to the hardness. These QTLs, especially under stress conditions, need extensive research. Moreover, understanding the modification or stabilization of hardness under stress condition and the relationship with stress tolerance will facilitate the selection of superior lines that maintain both high yield and quality even under the stress environment. Therefore, in the current work, we aimed to identify the genetic loci and marker trait associations (MTAs) that contributes for hardness under optimum conditions in Japan, and heat and combined heat-drought (HD) conditions in Sudan. We used a panel of multiple synthetic derivatives (MSD) having diverse Aegilops tauschii genome segments and investigated the association between hardness stabilization and stress tolerance. Under stress conditions, we observed that less reduction of kernel weight is associated with either low change or stable kernel hardness. We identified 47 markers associated with hardness under all conditions; the D genome was the main contributor. For the first time, we found a significant association with hardness under stress conditions on chromosome 4D. We dissected several candidate genes associated with the change of hardness under stress conditions. Our results will improve the understanding of the genetic factors that affect wheat hardness stability.