Inelastic neutron scattering has been employed to determine the complete crystal field (CF) level diagrams at the Pr sites (8c) and (4a) in the clathrate compound Pr3Pd20Si6 with cubic Cr23C6-type crystal structure. The splitting of the H-3(4) multiplet of Pr3+ was found to be Gamma(3) -> Gamma(5) (1.44 meV) -> Gamma(4) (4.46 meV) -> Gamma(1) (10.7 meV) at the (8c) site and Gamma(5) -> Gamma(3) (0.70 meV) -> Gamma(4) (10.8 meV) -> Gamma(1) (24.8 meV) at the (4a) site. We present additional measurements of the specific heat down to 2K and single-crystal high-field magnetization up to 28 T, and show that a calculation with no adjustable parameter based on our CF parameters can quantitatively reproduce the magnetic part of the specific heat and the magnetization curves along three cubic high-symmetry directions. We extrapolate the CF parameters obtained for Pr3Pd20Si6 to those of the isostructural compounds Ce3Pd20Si6, Nd3Pd20Si6 and Er3Pd20Si6, and compare the results with inelastic neutron scattering experiments reported in literature. For the (8c) site, the experimentally determined CF level diagrams are consistent and in good agreement with the extrapolation. In contrast, for the (4a) site, the few available experimental results are inconsistent and in poor agreement with the extrapolation. The extrapolation says that in Ce3Pd20Si6 the CF ground-state at both Ce sites (8c) and (4a) is the Gamma(8) quartet.