Image generator splits detail by frequency to keep attention costs down

Transformers that generate images directly as pixels face an awkward trade-off. Capturing finer detail usually means more tokens, and more tokens make the model's global attention step steadily more expensive. A new preprint, posted on arXiv and not yet peer reviewed, argues that the two need not be linked. Jiongfan Yang and colleagues describe SpecSlot, an image generator that sends only the coarse structure of an image through global communication and handles fine detail locally. The authors report that it matches or beats a comparable baseline, in some settings at substantially lower computational cost.

A Transformer processes data as a sequence of tokens. Pixel-space generators produce pixel values directly rather than working in a compressed intermediate representation, and in these models each token typically stands for one small square patch of the image. Smaller patches capture finer detail but make the sequence longer. Global attention, the operation in which every token exchanges information with every other token, becomes increasingly costly as that sequence grows.

The authors start from the observation that detail and global communication need not go together. Image content can be described in terms of frequency. Low-frequency components carry broad shapes and smooth variations, while high-frequency components carry sharp edges and fine texture. According to the study, low-frequency structure benefits from exchange across the whole image, whereas high-frequency detail can be modelled locally. To act on this, SpecSlot uses the discrete cosine transform (DCT). This standard mathematical technique re-expresses a block of pixels as a weighted sum of patterns of increasing frequency, and the weights are called coefficients.

Each spatial anchor, a position on the image grid, is expanded into several 'slots', and each slot is assigned its own non-overlapping band of DCT coefficients. This separates two choices that are normally tied together: how densely the image is sampled in space and how finely its frequency content is divided. Only the lowest-frequency slot from each anchor takes part in global attention, while all slots interact with their neighbours locally. The model can therefore add spectral detail without making the global sequence longer. A component the authors call a spatial head adds a local correction to the frequency-based prediction, while the generative state, training target and loss all remain in pixel space. SpecSlot is a rectified-flow model, meaning it learns to turn random noise into images along simple, near-straight paths.

Image quality is measured with FID (Fréchet Inception Distance), which scores how statistically similar generated images are to real ones; lower is better. The baseline is a model called JiT, matched to SpecSlot for FLOPs, a count of arithmetic operations used as a proxy for computing cost. On the ImageNet dataset at 256-pixel resolution, the authors report that SpecSlot reaches a final FID comparable to this baseline while converging faster in training. At 512 pixels, they report an FID improvement of 2.84 over the same control with 3.9% fewer FLOPs. SpecSlot also comes within 0.69 FID of a costlier configuration, JiT-B/16, at 38.2% lower cost. For text-to-image generation, SpecSlot is reported to beat JiT-B/8 on a benchmark called DPG-Bench by 1.06 at 40.6% lower forward cost, the cost of one pass through the network.

Every comparison in the abstract is against a JiT variant, so the abstract does not say how the approach fares against other image generators. The size of the gain also varies. At 256 pixels the benefit is faster convergence rather than better final quality, and the clearer quality improvement appears only at 512 pixels. Whether that pattern holds at higher resolutions or larger model sizes remains to be shown.