Galaxy sizes and their evolution over cosmic time have been studied for decades and serve as key tests of galaxy formation models. However, at z ≳ 1 these studies have been limited by a lack of deep, high-resolution rest-frame infrared imaging that accurately traces stellar mass distributions. Here, we leverage the new capabilities of the James Webb Space Telescope (JWST) to measure the 4.4 μm sizes of ∼1000 galaxies with and 1.0 ≤ z ≤ 2.5 from public CEERS imaging in the Extended Groth Strip deep field. We compare the sizes of galaxies measured from NIRCam imaging at 4.4 μm (λ_rest ∼ 1.6 μm) with sizes measured at 1.5 μm (λ_rest ∼ 5500 Å). We find that, on average, galaxy half-light radii are ∼9% smaller at 4.4 μm than 1.5 μm in this sample. This size difference is markedly stronger at higher stellar masses and redder rest-frame V − J colors: galaxies with M_* ∼ 10¹¹ M_⊙ have 4.4 μm sizes that are ∼30% smaller than their 1.5 μm sizes. Our results indicate that galaxy mass profiles are significantly more compact than their rest-frame optical light profiles at cosmic noon, and demonstrate that spatial variations in age and attenuation are important, particularly for massive galaxies. The trend we find here impacts our understanding of the size growth and evolution of galaxies, and suggests that previous studies based on rest-frame optical light may not have captured the mass-weighted structural evolution of galaxies. This paper represents a first step toward a new understanding of the morphologies of early massive galaxies enabled by JWST's infrared window into the distant universe.