I'll keep updating this list as I find more resources. This is a short selection of the ones I remember clearly enough to explain when and how they helped me. If you know another good source, don't hesitate to reach out.
For Optimal Control
Applied Optimal Control, a lecture series by Christian Hubicki. One of the best parts is that he writes the math on the board first, then writes the code as a direct translation from that math into a programming language.
An Introduction to Trajectory Optimization, a tutorial by Matthew Kelly. This was the first paper and video I used to learn optimization, which is why I later reproduced its cart-pole example as part of my trajectory optimization work.
For Reinforcement Learning
For PPO, I'd first read Schulman et al.'s original paper, then watch Simply Explaining Proximal Policy Optimization (PPO) by Johnny Code. That channel has other good videos too.
For gradient methods, Policy Gradient in 30 min by Zachary Huang. I like how he presents things intuitively, always showing the problem first (why something is needed) and then how a new approach solves it, so you can follow the progression from REINFORCE to actor-critic to PPO.