Tutorial 13: Dual Numbers - Simplex Programming Language

What You'll Learn

Creating and using dual numbers
Computing derivatives automatically
Multi-dimensional gradients
Integrating with Neural Gates

Prerequisites

Complete Tutorial 11: Neural Gates first.

Step 1: Your First Dual Number

A dual number carries both a value and its derivative. Create one with dual::variable():

fn main() {
    // Create a variable at x = 3.0
    let x = dual::variable(3.0);

    // x.val = 3.0 (the value)
    // x.der = 1.0 (dx/dx = 1)

    // Compute y = x^2
    let y = x * x;

    println("y = {}", y.val);     // 9.0
    println("dy/dx = {}", y.der); // 6.0 (= 2x at x=3)
}

How It Works

When you multiply two dual numbers, the product rule is applied automatically: (a, a') * (b, b') = (a*b, a'*b + a*b')

Step 2: Complex Functions

Derivatives propagate through all operations including transcendental functions:

fn activation(x: dual) -> dual {
    // Sigmoid activation: 1 / (1 + e^(-x))
    dual::constant(1.0) / (dual::constant(1.0) + (-x).exp())
}

fn main() {
    let x = dual::variable(0.0);
    let y = activation(x);

    println("sigmoid(0) = {}", y.val);    // 0.5
    println("sigmoid'(0) = {}", y.der);  // 0.25
}

Step 3: Computing Gradients

For functions with multiple inputs, use the gradient helper:

use simplex::diff::{gradient};

// f(x, y) = x^2 + 2xy + y^2
fn f(x: dual, y: dual) -> dual {
    x.pow(2.0) + dual::constant(2.0) * x * y + y.pow(2.0)
}

fn main() {
    // Compute gradient at (1, 2)
    let grad = gradient(f, [1.0, 2.0]);

    println("df/dx = {}", grad[0]);  // 6.0 (= 2x + 2y)
    println("df/dy = {}", grad[1]);  // 6.0 (= 2x + 2y)
}

Step 4: Training a Simple Model

Use dual numbers to compute gradients for training:

// Simple linear regression: y = w*x + b
fn predict(x: f64, w: dual, b: dual) -> dual {
    w * dual::constant(x) + b
}

fn loss(pred: dual, target: f64) -> dual {
    (pred - dual::constant(target)).pow(2.0)
}

fn main() {
    var w = 0.0;
    var b = 0.0;
    let lr = 0.01;

    // Training data: y = 2x + 1
    let data = [(1.0, 3.0), (2.0, 5.0), (3.0, 7.0)];

    for _ in 0..100 {
        for (x, y) in data {
            // Forward pass with gradients
            let w_dual = dual::variable(w);
            let b_dual = dual::constant(b);  // Only tracking w gradient

            let pred = predict(x, w_dual, b_dual);
            let l = loss(pred, y);

            // Update w using gradient
            w = w - lr * l.der;
        }
    }

    println("Learned w: {}", w);  // ~2.0
}

Exercise

Extend the training example to also learn b by using multidual<2> to track both gradients simultaneously.

What's Next?

In Tutorial 14: Edge Hive, you'll learn to build local-first AI applications that run on user devices.