0 - Intro

Single layer Single output Perceptron

A (single layer single output) perceptron is a function $f : R^{n} ⟶ R$ , where $f (x) = g (w \cdot x + b)$ , ( $w \in R^{n}, b \in R$ ).Where $g$ is an activation function.

decision boundary

if $f$ is a SLSO perceptron used for $0, 1$ classification, then at a threshold $t$ the decision boundary $d b (f, t) \equiv f (x) - t = 0$ . That is, its the set of all points in $R^{n}$ defined by all $x$ for which $f (x) - t = 0$ .
The idea, is $f (x) - t \geq 0$ is the class "1" and $f (x) - t < 0$ is the class "0".

injective activation produces linear decision boundary

let $f (x) = g (w \cdot x + b)$ be a perceptron such that $g$ is injective then for any $t \in R A N G E (g)$ , $d b (f, t)$ is a hyperplane in $R^{n}$ .

$* p r o o f * :$ If $g$ is injective, then the restriction $g : R ⟶ RANGE (g)$ is bijective. Hence,
$g (w \cdot x + b) = t ⟺ w \cdot x + (b - g^{- 1} (t)) = 0$ Clearly the set of all $x$ satisfying that is a hyperplane.

$◻$

The above theorem is one way!

We have shown that injective activations product linear decision boundaries, but that doesn't mean that non injective activations produce non linear boundaries.

Activations for SLSO Perceptrons:

Support/Figures/Pasted image 20250108204219.png

we will eventually put pictures of descicion boundraries of SLSO perceptrons on $R^{2}$ for different activations from https://www.desmos.com/3d/j9vlqfpmxo here.

We can then see that the $X O R$ gate cannot be separated by a linear boundary.