Use this article as a notebook, a reference guide to the nanodegree video content.
RNN.2
RNN one way to incorporate memory. Sequential. RNN tempo dependencies, generate captions, having memories, as opposed to feed forward NN (only current input). Best analogy from a single image hard to tell if a picture of a cat is moving. Single image is current input, a series of images that form a gif is temporal data. Temporal dependencies. Non linear functions.
Simple RNN Elmam Net
RNN.3
Feed forward nets are limited — unable to capture temporal dependencies. Inputs with dependency across time : video, music, … overview of historic RNN, significant RNN architecture of the past.
Simple RNN Elman Networks aka Jordan networks 1990.
Past RNN suffers the vanishing gradient problem.
RNNs have a key flaw, as capturing relationships that span more than 8 or 10 steps back is practically impossible. This flaw stems from the “vanishing gradient” problem in which the contribution of information decays geometrically over time.
LSTM mid 1990s invented to address vanishing gradient problem. “some signals known as states can be stored / introduced at a time / or not introduced at a specific time using gates.
RNN.4 applying RNN everywhere in real life, lots of use cases. Speech recognition, dragon Alexa…
