Is a dense layer automatically nonlinear?

No. Nonlinearity depends on the activation or surrounding operations; an affine transformation alone is not nonlinear in the usual neural-network sense.

Do more examples add more weights?

No. The same layer parameters are reused across examples.

Can a trained weight equal zero?

Yes. Fully connected means that the parameterized connection exists, even if its current value is zero.

AI Glossary / Deep learning / Intermediate

Fully connected layer

Definition

A fully connected layer is a neural-network layer in which each output unit has a trainable weight for every input feature in the vector it receives. It combines those weighted inputs, usually adds a bias, and may be followed by an activation function. The layer is also commonly called a dense layer. Fully connected describes the pattern of possible connections, not a guarantee that every learned weight is nonzero.

What it means

The layer can learn different combinations of the same input information. One output might emphasize one set of features while another uses a different combination. A layer with m input features and n outputs normally has m times n weights, plus n biases when bias is enabled. This makes the size easy to calculate but can make a large dense layer expensive. The inputs may be original measurements or representations created by earlier network layers.

How it works

For one input vector, an affine dense transformation computes z = W x + b under a column-vector convention. Each row of W supplies the weights for one output. An activation may then transform z. Libraries use different matrix orientation conventions: PyTorch's Linear exposes an affine transformation, while Keras Dense can include a chosen activation. With higher-rank tensors, common implementations apply the transformation to the final feature axis and preserve leading dimensions. They do not automatically mix every token or spatial position into every other position.

Three practical examples

Classification head: a dense layer turns an existing image representation into one score for each target category.
Tabular prediction: dense layers combine prepared customer or transaction features to estimate a numerical target or class scores.
Token processing: the same dense transformation can map each token's feature vector to a different width while retaining the sequence dimension.

Think of it this way

Think of a mixing desk with several output channels, where every output can assign its own level to every incoming signal. This resembles a full matrix of weights. The analogy is limited because neural layers may use negative weights, learned biases, and nonlinear activations; the outputs are numerical representations rather than necessarily audible mixtures.

Worked example

Take an input x = [2, -1, 3]. A two-output layer has first-output weights [1, 2, 0] and bias 0.5, and second-output weights [-1, 0, 2] and bias -1. The first affine output is 1 times 2 plus 2 times -1 plus 0 times 3 plus 0.5 = 0.5. The second is -1 times 2 plus 0 times -1 plus 2 times 3 minus 1 = 3. There are six weights and two biases, totaling eight parameters. A ReLU activation would leave these two positive values unchanged.
If a batch contains four such three-feature vectors, the output shape is four by two; the parameter count stays eight because the layer shares its weights across examples. If an input instead has shape four by five by three, applying a standard last-axis dense transformation gives four by five by two. Flattening each five-by-three example first would produce fifteen inputs per example and a different layer with thirty weights plus two biases. Those operations have different meanings and should not be treated as interchangeable.

Illustrative case study

A fictional text-analysis team expects a dense layer to combine information across twenty token positions. Its input has separate batch, token, and feature dimensions, so the implementation actually transforms each token's features independently using shared weights. The team adds an explicitly chosen aggregation stage before its document-level output. The case illustrates why checking tensor dimensions can prevent an architecture from silently performing a different operation than intended.

Common misconceptions and mistakes

Fully connected does not mean connected to every layer in the entire model. Dense weights are not the same as a dense graph connecting all examples to one another. Adding two affine layers without an intervening nonlinearity still produces an affine mapping, so extra layers alone do not guarantee greater nonlinear expressiveness. A large parameter count is not evidence of greater accuracy. Flattening an image can also discard the explicit arrangement that convolutional or other structured operations exploit.

Business use

Dense layers are useful building blocks for combining features and producing task-specific outputs. For deployment, calculate both parameter storage and the computation performed for the actual input shapes. A representation of width 1,000 mapped to 100 outputs requires 100,100 parameters with biases, before considering training states or other layers. Compare model quality and operational cost with smaller alternatives, and inspect whether the model's input organization matches the intended relationships among fields, positions, or time steps.

Check your understanding

A dense layer maps ten input features to four outputs with one bias per output. How many parameters does it have?

Answer: Ten times four weights plus four biases, totaling forty-four parameters. Batch size does not change this count.

Related terms

Bias
Activation function
Hidden layer
Multilayer perceptron
Dense layer
weight matrix
linear layer
tensor shape

Learn next

calculate a small dense layer by hand, then trace its input and output shapes for a batch and a sequence. Compare a last-axis transformation with flattening and pooling to understand which operation actually combines positions.

Sources

Keras, Dense layer reference: https://keras.io/api/layers/core_layers/dense/ PyTorch, Linear reference: https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html

Previous: Fully-connected Next: Fully independent training conditional approximation