Synthesis Network

Unlike traditional GANs that start from random noise, the StyleGAN synthesis network begins with a learned constant tensor. This 4x4x512 tensor is a fixed, trainable parameter that provides a stable, rich base for the generator to build upon. This design choice removes the need to map the first layer of the network from a latent vector, allowing the mapping network to focus exclusively on generating style controls.

The synthesis network builds the image progressively, starting from a low resolution (e.g., 4x4 pixels) and adding new layers to increase resolution up to 1024x1024. Each block consists of:

• An upsampling operation (typically nearest-neighbor).
• Two convolutional layers with modulated weights.
• Noise injection for stochastic details. This hierarchical approach allows the network to first establish coarse structures (pose, face shape) and later refine fine details (hair texture, skin pores), leading to higher quality and more stable training.

The defining feature of the synthesis network is the use of Adaptive Instance Normalization (AdaIN). At each convolutional layer, the style vector w (from the mapping network) is transformed into a pair of parameters (y_s, y_b) that modulate the layer's activations. The operation is: AdaIN(x, y) = y_s * (x - μ(x)) / σ(x) + y_b

• x: The feature map from the previous layer.
• μ(x), σ(x): The mean and standard deviation of x. This replaces traditional batch normalization, allowing per-sample style control where y_s scales the feature variance and y_b shifts the feature mean.

To model random, high-frequency details (e.g., freckles, hair placement, skin texture), the synthesis network injects per-pixel noise before each AdaIN operation. A noise map—single-channel and filled with uncorrelated Gaussian values—is broadcast to all feature maps, scaled by a learned per-channel weight. This introduces controlled stochasticity, ensuring the generator can produce unique, non-deterministic variations even when using the same style vector w.

A single style vector w is fed to every layer of the synthesis network, but its effect is specialized per layer. The mapping network's output w is transformed by a separate, learned affine transformation for each synthesis block, producing layer-specific style parameters. This creates a hierarchical control mechanism:

• Coarse styles (4x4 - 8x8 layers): Control pose, general face shape, and hairstyle.
• Middle styles (16x16 - 32x32 layers): Control facial features, eyes, and mouth.
• Fine styles (64x64 - 1024x1024 layers): Control color scheme, micro-structure, and background.

Each resolution block in the synthesis network culminates in a dedicated ToRGB layer. This is a 1x1 convolutional layer that projects the high-dimensional feature maps at that resolution directly into a 3-channel RGB image. During progressive growing, the final output is a weighted sum of the outputs from the current and previous resolution's ToRGB layers, enabling smooth transitions. This design allows the network to separate style information (handled by the modulated convolutions) from color information (handled by the final ToRGB projections).

In StyleGAN, the mapping network is an 8-layer multilayer perceptron (MLP) that transforms an input latent vector z from the initial latent space (Z-space) into an intermediate latent vector w in the W-space. This non-linear transformation is crucial for feature disentanglement, allowing different aspects of the generated image (e.g., pose, hairstyle, lighting) to be controlled by separate, more linear dimensions in w. The output w vectors are then fed into the synthesis network via AdaIN layers to modulate style.

Adaptive Instance Normalization (AdaIN) is the key mechanism that injects style information from the mapping network into the synthesis network. For each convolutional layer in the synthesis network, AdaIN aligns the channel-wise mean and variance of the activation maps (the "content") with the corresponding values provided by a style vector (derived from w). The operation is defined as: AdaIN(x, y) = σ(y) * (x - μ(x)) / σ(x) + μ(y) where x is the content activation and y is the style vector. This allows for precise, per-layer stylistic control without traditional conditional concatenation.

Unlike typical generators that start from a learned or random tensor, the StyleGAN synthesis network begins with a constant 4x4x512 tensor that is learned during training. This constant serves as the foundational "canvas" upon which all styles are applied. The use of a constant input, combined with style modulation via AdaIN, forces the network to learn stable, hierarchical features—coarse styles (e.g., pose, face shape) affect early layers, while fine details (e.g., eye color, micro-textures) are controlled by later layers.

To generate stochastic details like hair strands, skin pores, or background texture, StyleGAN uses per-pixel noise injection. Before each AdaIN operation in the synthesis network, a single-channel noise image (scaled by a learned per-feature scaling factor) is added to the corresponding feature map. This noise is unique for each generated image and is critical for producing high-fidelity, non-deterministic fine details that make outputs appear more realistic and varied, even when the style vectors (w) are identical.

Progressive growing is a training technique used in the original StyleGAN and its predecessor, ProGAN. The synthesis network is trained starting from a low resolution (e.g., 4x4 pixels). New layers are gradually added to increase the resolution (e.g., to 8x8, 16x16, up to 1024x1024). This allows the model to first learn large-scale structural features stably before focusing on finer details. While later StyleGAN versions (StyleGAN2, StyleGAN3) moved to a residual, fixed-resolution architecture, progressive growing was foundational for achieving high-resolution synthesis.

Style mixing is a demonstration of the disentanglement achieved by the mapping and synthesis networks. During generation, different style vectors (w1, w2) can be applied to different blocks of the synthesis network. For example, w1 (controlling coarse features) can be used for layers 1-4, and w2 (controlling fine features) for layers 5+. This creates a hybrid image that combines high-level attributes from one source with low-level details from another, proving that the network has learned to separate semantic features across the hierarchical layers.