Short-wavelength free-electron lasers are now well established as essential and unrivalled sources of ultrabright coherent X-ray radiation. One of the key characteristics of these intense X-ray pulses is their expected few-femtosecond duration. No measurement has succeeded so far in directly determining the temporal structure or even the duration of these ultrashort pulses in the few-femtosecond range. Here, by deploying the so-called streaking spectroscopy technique at the Linac Coherent Light Source, we demonstrate a non-invasive scheme for temporal characterization of X-ray pulses with sub-femtosecond resolution. This method is independent of photon energy, decoupled from machine parameters, and provides an upper bound on the X-ray pulse duration. We measured the duration of the shortest X-ray pulses currently available to be on average no longer than 4.4 fs. Analysing the pulse substructure indicates a small percentage of the free-electron laser pulses consisting of individual high-intensity spikes to be on the order of hundreds of attoseconds.