As hadron accelerators achieve increasing beam power, damage to targets is becoming increasingly severe. To mitigate this damage, nonlinear beam optics based on octupole magnets is attractive. Nonlinear optics can decrease the beam-focusing hazard due to failure of the rastering magnet. As a side effect of nonlinear optics, the beam size at the tail is known to expand drastically compared with linear optics, owing to defocusing in the nonlinear case; this may cause severe beam loss downstream of the octupole magnets. Beam shape and beam loss as a side effect of nonlinear optics have thus far been studied via a simplified filament model that ignores beam-divergence spread at the octupole magnet, so that the problem may be treated by a simplified transfer matrix. Several beam-tracking studies have been performed for the specified case given by the specified emittance and Twiss parameters, whereas a simplified model is required for application to the pragmatic beam tuning. In this study, a new generalized model is proposed for application to an octupole magnet, regardless of the filament-model approximation. It is found that the transverse distribution obtained by beam tracking can be specified by the introduction of only two parameters, namely the normalized octupole strength of K-8* and the cot phi of the phase advance from the octupole to the arbitrary position. With the new generalized model, the distribution with allowable beam loss is studied in detail. The best uniform shape is found for specific parameters of K-8* and cot phi that cannot be deduced from the filament model. However, to achieve the flat distribution, a considerable K-8* is required, incurring a large beam loss at the position having cot phi similar to -0.5 downstream of the octupole. To achieve the two antagonistic requirements of reduction of the beam-peak density and minimization of the beam loss, the transverse distribution is surveyed for a large range of beam position. It is found that a bell-shaped distribution with K-8* similar to 1 and cot phi similar to 3 can satisfy requirements. This result is applied to beam transport in the spallation neutron source at J-PARC. The beam profile calculated using nonlinear optics is compared with the experimental result. It is proven that the transverse beam distribution with nonlinear optics and an octupole magnet can be simplified by the present generalized model, which can be applied to other facilities using octupole magnets. The calculation result shows good agreement with the experimental data, and the peak current density is reduced by 50% compared with the linear-optics case with allowable beamwidth and beam loss.