Is a multilayer perceptron feed-forward?

Yes, in its usual form: successive layers transform inputs into outputs without recurrent connections.

Can skip connections still be feed-forward?

Yes. A connection can bypass intermediate layers while preserving an acyclic computation graph.

Does backpropagation contradict the name?

No. Backpropagation computes training derivatives backward through the graph; feed-forward describes the architecture's prediction flow.

AI Glossary / Graphs and symbolic AI / Intermediate

Feed-forward network

Definition

A feed-forward network is a neural network whose prediction computation moves through a directed structure without recurrent cycles. Information passes from inputs through intermediate transformations to outputs. A multilayer perceptron is a familiar example, but feed-forward describes the direction and structure of computation rather than requiring every neuron to connect to every neuron in the next layer.

What it means

The network represents a function: give it an input, apply its learned transformations, and obtain an output. Hidden layers can build useful combinations of the original features. The output might be a numerical estimate, class scores or another representation, depending on the task and training objective.

The absence of recurrent connections does not mean a feed-forward network cannot use historical data. Past measurements can be explicitly included in its input. It means that the basic prediction graph does not carry a recurrent hidden state around a cycle. Its architecture alone also does not establish that its predictions are accurate or well calibrated.

How it works

In a typical multilayer perceptron, a layer forms weighted combinations of its inputs, adds biases and applies an activation function. The next layer receives those outputs. A final transformation produces values appropriate to the task, such as an unrestricted number for regression or normalized class scores for classification.

Nonlinear hidden activations matter. Composing only affine transformations can be collapsed into a single affine transformation, regardless of how many such layers are stacked. Nonlinearity lets the model express more complicated input-output relationships.

During training, a loss measures disagreement with the chosen objective. Backpropagation computes derivatives through the computation graph, and an optimizer updates parameters. This backward derivative calculation does not turn the prediction architecture into a recurrent network. Training and prediction are different processes, and their directions of information flow should not be confused.

Three practical examples

1. A demand-estimation prototype maps calendar features, recent aggregate demand and planned promotions to a numerical prediction. Historical information is available because it is explicitly encoded in the inputs.

2. A document classifier maps a fixed-length feature vector to category scores. The classifier's feed-forward layers operate on that representation; they do not automatically explain how the original text was converted into features.

3. A quality-inspection model combines measured dimensions and sensor readings to estimate a category. Its usefulness depends on representative training and evaluation data, not simply the presence of hidden layers.

Think of it this way

Think of a sequence of processing stations: each station transforms what it receives and sends the result onward. Some routes can skip a station while still moving forward, but no station sends its output back into an earlier step of the same prediction cycle. The analogy describes the computation graph, not independent workers or a guarantee that every intermediate value has a human-readable meaning.

Worked example

Take two inputs, x1 = 2 and x2 = −1. Define two hidden units using ReLU, which returns the larger of zero and its input. Let h1 = ReLU(x1 + x2) and h2 = ReLU(−x1 + 2x2 + 1).

The first pre-activation is 2 + (−1) = 1, so h1 = 1. The second is −2 + 2(−1) + 1 = −3, so h2 = 0. Let the output be y = 3h1 − 2h2 + 0.5. Substituting the hidden values gives y = 3.5.

Now change only x2 to 1. Then h1 = ReLU(3) = 3 and h2 = ReLU(1) = 1, giving y = 9 − 2 + 0.5 = 7.5. The fixed transformations are evaluated in forward order for each input. These hand-selected weights demonstrate computation, not training, predictive accuracy or a calibrated probability. In particular, the unrestricted output 3.5 should not be described as a probability.

Illustrative case study

Fictional case study: Hazel Forecast builds a feed-forward prototype for daily workload. The team first includes a field calculated after the day ends, producing misleadingly strong retrospective results. A review identifies that the field would be unavailable when a real forecast is made.

The team removes that information, defines an input cutoff time and evaluates on later periods. It compares the network with a simple baseline before deciding whether the extra complexity is justified. The case illustrates evaluation discipline; it reports no actual forecast improvement or successful production deployment.

Common misconceptions and mistakes

Feed-forward does not mean untrained, shallow or fully connected. It also does not mean that backward derivative calculations are forbidden during learning. Those calculations support parameter updates without adding recurrent prediction connections.

Adding layers without appropriate nonlinearities does not automatically create a richer function class than an affine model. More parameters also do not guarantee better generalization. Finally, outputs need interpretation: a score, a logit and a probability are different quantities. Even a probability-shaped output requires evaluation before it can be treated as a dependable confidence estimate.

Business use

For a business application, start with the prediction target, available-at-decision-time inputs and the cost of different errors. Establish a baseline and a suitable evaluation split before tuning network size. Include preprocessing in the documented model pipeline so training and deployment use features consistently.

Inspect performance on relevant subgroups and changed operating conditions. Consider latency, maintenance and explanation needs alongside average accuracy. A feed-forward model can be a useful component in a larger system, but decisions about thresholds, human review and downstream actions remain application choices that the network architecture does not make for you.

Check your understanding

Why is the second hidden unit zero for the first worked input?

Answer: Its pre-activation is −3, and ReLU maps negative inputs to zero. The output calculation therefore receives no contribution from h2 for that input.

Related terms

Learn next

Trace a small network by hand, distinguishing weights, biases and activations. Then study loss functions and backpropagation, followed by train-validation-test separation, before interpreting a trained network's performance claims.

Backpropagation

Sources

Primary source: Zhang, Lipton, Li and Smola, Dive into Deep Learning, “Multilayer Perceptrons”: https://d2l.ai/chapter_multilayer-perceptrons/mlp.html . The source supports the layered computation and nonlinear activation discussion. The numerical network and fictional case are original teaching examples; no model training or benchmark is claimed.

Previous: Feed forward (control) Next: Feed-forward neural network density model