One of the biggest evolutions in neural network models in the past few years is the advent of closed-form continuous-time or “CFC” networks.
These networks essentially use a closed form solution for a network model, and gating mechanisms that are pretty fascinating from a design perspective.