Creating a Multilayer Perceptron from Scratch

Last Updated on August 19, 2026 by Editorial Team Author(s): Caden Lippie Originally published on Towards AI. Creating a Multilayer Perceptron from Scratch A perceptron is a fundamental component of artificial neural networks. Inspired by the neurons in our brains*, these perceptrons make decisions and “learn” by iterating to minimize errors. When these single perceptrons are combined in layers, they form networks that can “learn” more complex patterns and make more nuanced decisions. *Organic neurons can have different purposes, and more neurons do not necessarily mean more intelligence. I think this is really interesting because this is very similar to how artificial neurons work.Fun Fact: The African elephant brain has 257 billion neurons, which is 3x the human brain. — https://pmc.ncbi.nlm.nih.gov/articles/PMC4053853/ In an attempt to grasp these mechanisms more thoroughly, I decided to create my own simple multilayer perceptron (MLP) and compute everything by hand (using a calculator). This process gave me a more confident and comprehensive understanding of this concept, which I can lean on when dealing with more complex systems. This process also helped to demystify AI and has given me a greater appreciation for the computing power of modern computers. Architecture When building an MLP, one of the first decisions that has to be made is the architecture. In other words, how many layers of perceptrons, how many perceptrons in each layer, and the connections between the perceptrons. Since I am doing all these calculations manually, I wanted to keep the architecture simple. Here I have what looks like a fully connected 3-layer network, but is functionally just 2 computational layers, given that the input layer doesn’t contain any parameters, just passes the inputs to the next layer. The hidden layer learns the patterns in the data and passes its own version of input information to the output layer, which has the responsibility of making the prediction. In a deeper network, the first hidden layers learn more surface-level patterns while the deeper layers learn the complex and specific patterns in the data. The learning process that was mentioned earlier involves updating parameters (weights and biases) to minimize loss. These parameters were randomly chosen values between 0 and 1, again to make calculations easier. 1st Forward Pass Step 1: Compute weighted sum The first forward pass will yield the first prediction, which will start the learning process. Here in step 1, you calculate the hidden neurons by multiplying the inputs by the weights that connect those inputs to the hidden neuron and then adding the bias associated with the hidden neuron. The weights are used to distribute the importance of certain inputs to the neuron. This is useful, for example, if you are trying to find the difference between a tulip and a rose, one hidden neuron could roughly be responsible for detecting color, and another neuron could focus on the petal shape. If the inputs are the RGB values and petal length and width, then RGB should account for more decision-making power in hidden neuron 1, and length and width should account for more in hidden neuron 2. The weights will make those distributions accordingly. The biases are also useful for setting the baseline for each neuron. If you have data that has 0’s as inputs without biases, that would break the model. A bias ensures the model will function regardless of the training data and allows the model to fit data that doesn’t intersect the origin. Step 2: Pass weighted sum through activation function to introduce non-linearity A key part of an MLP is the activation function. Without an activation function, the model could be reduced to a single equation and would behave more like linear regression. For example, this whole network could be collapsed into this one equation: (0.5(0.4) + 0.8(0.3) + 0.1)0.5 + (0.5(0.2) + 0.8(0.6) + 0.1)0.7 + 0.1. Instead, adding an activation function introduces non-linearity and makes the network impossible to reduce to a single linear equation. A simple and common activation function that is used in hidden layers is ReLU, which sets any negative number to 0 and retains the value of positive numbers. I chose sigmoid here because it works for both hidden and output layers and is straightforward to differentiate during backpropagation, which will be helpful in the next section. https://medium.com/@krishnakalyan3/introduction-to-exponential-linear-unit-d3e2904b366c Step 3: Compute weighted sum for next layer This step mirrors step 1, except the outputs of the hidden layer now serve as the inputs. This layered structure helps to learn complex patterns because the layers feed into each other and use the previous layers’ “knowledge” to inform decisions. Step 4: Pass weighted sum into activation function Here, the same process detailed in step 2 is used to compress the output into a probability between 0 and 1. Since this layer is an output layer, the activation function choice is a little more constrained than for a hidden layer. For this example, our model is a binary classifier (0 or 1). Therefore, the activation function needs to compress the output between 0 and 1. If this were a multi-class classification problem, then a softmax function would be a better choice. Step 5: Calculate Loss Finally, the loss is calculated to determine how well the model is performing. There are different ways to calculate loss, but for this example, I used mean squared error (MSE), which is calculated by squaring the difference between the prediction and the label. With only a single example, there is nothing to average, so this simplifies to plain squared error. However, with multiple examples, the mean would be taken across all of them. It is worth noting that the scalar loss value itself doesn’t appear in the weight update equations, but instead, the gradient of the loss is what drives learning. The loss value is better used as a benchmark to track progress across training iterations. While it may seem like a loss of 0 is the gold standard, that is not necessarily true. A loss of 0 would imply that the model […]