This paper presents a unified framework for various multi-scale structures. With this framework, two factors of multi-scale convolution, i.e. feature propagation and cross-scale communication are explored. A generic and efficient multi-scale convolution unit named Multi-Scale cross-Scale Share-weights Convolution (MS3-Conv) is proposed. Experimental results showed that, the proposed MS3-Conv can achieve better super resolution performance than conventional convolution methods with less parameters and computational cost. By observation of the visual quality, results also showed that the MS3-Conv outperform in the reconstruction of high-frequency image details.