Optimizer Comparisons (SGD, Momentum, Adam)

You are given a small CNN model and a tiny image dataset (CIFAR-10 subset). Implement three optimizers from scratch—plain SGD, SGD with momentum, and Adam—and train the network for 10 epochs with each optimizer. After training, report the final training loss, final validation accuracy, and the total wall-clock time for each optimizer. Use the same initial weights, batch size (128), and starting learning rate (0.01) for all three runs. Do not use any high-level optimizer classes from PyTorch/TensorFlow; write the update rules yourself inside the training loop. Briefly explain which optimizer converges fastest in terms of epochs and which gives the best final accuracy.