Symmetry is a powerful concept in physics, but its applicability to far-from-equilibrium states is still being understood. Recent attention has focused on how far-from-equilibrium states lead to spontaneous symmetry breaking. Conversely, ultrafast optical pumping can be used to drastically change the energy landscape and quench the magnetic order parameter in magnetic systems. Here, we find a distinct symmetry-dependent ultrafast behaviour by use of ultrafast x-ray scattering from magnetic patterns with varying degrees of isotropic and anisotropic symmetry. After pumping with an optical laser, the scattered intensity reveals a radial shift exclusive to the isotropic component and exhibits a faster recovery time from quenching for the anisotropic component. These features arise even when both symmetry components are concurrently measured, suggesting a correspondence between the excitation and the magnetic order symmetry. Our results underline the importance of symmetry as a critical variable to manipulate the magnetic order in the ultrafast regime.