LSTM (1997): Teaching Networks to Remember
The gated memory cell that carried speech recognition and translation for two decades before Transformers

Contents
Read the sentence “The keys that I left on the table next to the window, where the light was best, were missing.” To know that “were” agrees with “keys”, you carried a fact across fifteen words. Humans do that without noticing. For a neural network in the 1990s it was close to impossible, and the reason was arithmetic.
The fix arrived in December 1997, in a paper by Sepp Hochreiter and Jürgen Schmidhuber in the journal Neural Computation. They called it Long Short-Term Memory, or LSTM. For roughly two decades it was the default way to make a network handle sequences, and it ran inside the speech recognition and translation tools that people used daily before the Transformer arrived. This post explains the problem it solved, the mechanism in plain terms, and why it eventually gave way.
The problem: networks that forget
A recurrent neural network (RNN) reads a sequence one step at a time and keeps a small summary, its hidden state, which it updates at every step. The state is the network’s memory. At each word, the new state is computed from the old state and the new input.
Training such a network uses the algorithm covered in backpropagation (1986), applied to the network unrolled through time: one copy of the network per step, all sharing weights. The error at the end of a sentence has to flow backwards through every copy to reach the early words.
Here is where the arithmetic bites. Each step multiplies the backward signal by a factor that depends on the weights and the activation function. If that factor is typically below one, the signal shrinks exponentially with the number of steps, and after fifty steps it has all but vanished. If it is above one, the signal explodes instead. Hochreiter analysed this in his 1991 diploma thesis, and the effect is called the vanishing gradient problem. The practical result: an early word’s influence on a late prediction cannot be learned, because the learning signal never reaches it.
Plain RNNs could therefore only learn short-range patterns. Linking “keys” to “were” across fifteen words was already hard. Linking a name in the first paragraph to a pronoun in the fifth was out of reach.
The idea: a lane that does not shrink
Hochreiter and Schmidhuber’s paper describes the issue as “insufficient, decaying error backflow” and proposes a design that enforces constant error flow. The core component is the constant error carousel: a memory cell whose value is carried forward from step to step by a plain addition, with a multiplier of exactly one on the self-connection. A signal travelling back through that path does not shrink, because multiplying by one preserves it.
A memory that can only ever hold on would be useless, so the cell needs controls. The paper’s answer is multiplicative gate units that learn to open and close access to the constant error flow. A gate is a number between 0 and 1 for each element of the cell, produced by a small neural layer from the current input and the previous output. Multiply something by a gate and a value near 1 lets it through, while a value near 0 blocks it.
The original 1997 design had two gates, input and output:
- The input gate decides how much of the new candidate information gets written into the cell.
- The output gate decides how much of the cell’s content is exposed to the rest of the network at this step.
Between them the cell holds a value across many steps, takes writes when the input gate opens, and reveals itself when the output gate opens. The paper reported that the method could bridge time lags in excess of 1,000 discrete steps, which is the figure to remember: against plain RNNs struggling at tens of steps, that was a different class of memory.
The forget gate, added in 2000
The 1997 cell could write and read, but it had no way to wipe itself. In a long continuous stream of input, the cell’s value could grow without limit, because every write added to it. In 2000 Felix Gers, Jürgen Schmidhuber and Fred Cummins published “Learning to Forget: Continual Prediction with LSTM” in Neural Computation, introducing a third gate that scales the old cell value before the new write is added. A forget gate near 1 keeps the memory; near 0 clears it.

That version is the one most people mean today. When a library says “LSTM”, it implements the three-gate cell. Its operation each step is:
- The forget gate scales the old memory.
- The input gate scales a candidate update, which is added to the memory.
- The output gate scales a squashed copy of the memory to produce the new hidden state.
The key property is that the memory path involves addition and an element-wise multiply by a gate. When the forget gate sits near 1, the memory and its gradient pass through almost unchanged, and that is the lane the 1997 paper wanted.
In code, using one framework’s built-in layer, the machinery fits in a few lines:
| |
Two return values matter: the hidden state h (what the cell shows to the next layer) and the cell state c (the private memory). Every framework gives you both, which is a direct reflection of the 1997 design.
From lab curiosity to the thing in your phone
LSTM was slow to take off. Through the 2000s it picked up wins on benchmark sequence tasks, but the broader field was elsewhere. What changed things was the same ingredient that revived the rest of the field, covered in AlexNet and ImageNet: more data and much more compute, thanks in part to the GPU story in how gaming GPUs became AI hardware.
The deployments are well documented:
- 2014. Kyunghyun Cho and colleagues published the Gated Recurrent Unit (GRU), a simplified relative with fewer gates.
- 2015. Google deployed LSTM for speech recognition in Google Voice. Reporting from the time said it cut transcription errors by 49 percent.
- 2016. Google released its neural machine translation system, built on LSTM networks, and reported translation errors falling by around 60 percent against its previous phrase-based system on the language pairs it tested. Apple announced LSTM use for QuickType on the iPhone and for Siri.
- 2017. Facebook was reported to be performing about 4.5 billion automatic translations a day with LSTM networks.
Those numbers come from the company announcements and press coverage summarised in the standard references, and like any vendor-reported figure they describe the vendor’s own systems on the vendor’s own tests. The direction is clear all the same. Between 2015 and 2017, LSTM was the thing that made computers noticeably better at listening and translating.
Beyond speech and translation, LSTM networks handled handwriting recognition, music generation, text prediction and many time-series tasks. Their strength was streaming: process one step, update the state, move on. That makes them a good fit for a phone keyboard predicting your next word as you type.
The sequence-to-sequence era and the road to attention
Translation systems of that era used two LSTMs: an encoder that read the source sentence into a fixed-size vector, and a decoder that generated the translation from it. The fixed vector was a bottleneck. A long sentence had to be squeezed into the same number of values as a short one, and long sentences suffered.
The fix was to let the decoder look back at all of the encoder’s intermediate states and pick the relevant ones at each output step, a mechanism called attention. It was first bolted onto LSTMs. The shift is explained intuitively in attention explained, and the 2017 paper that took the final step, dropping the recurrence entirely, is Attention Is All You Need. The wider sweep of those years is in from the Transformer to ChatGPT.
Word representations were the other piece. LSTMs read sequences of vectors, and the vectors came from embeddings such as those in word2vec; the geometry behind them is in vectors and embeddings. Early contextual models such as ELMo used LSTMs to produce word meanings that shifted with the sentence, which led straight into BERT.
Why Transformers replaced it
The reasons are practical and worth stating plainly.
Parallelism. An LSTM must process step 5 before step 6, because step 6 needs the state from step 5. Training cannot spread across the sequence, so GPUs sit partly idle. A Transformer processes all positions at once during training. When you can feed a model a trillion words, speed of training decides which architecture wins.
Reach. Even with gates, an LSTM compresses everything it has seen into a fixed-size state. Attention lets a position look directly at any earlier one. Long-range dependencies that LSTMs handled with effort come almost free.
Scaling. The Transformer’s gains held up as size grew, a pattern examined in scaling laws. LSTMs scaled less cleanly.
None of this makes LSTM wrong. The cost of a Transformer grows quickly with sequence length, while an LSTM does constant work per step and holds a constant amount of state. That trade-off explains why recurrent ideas keep returning. In 2024 Hochreiter’s group published xLSTM, which they describe as an upgrade of the original with changes that allow parallel training, a sign that the fixed-state design still attracts research effort.
What to take from it
LSTM is a clean example of an idea that fixes one mechanical problem very precisely. The vanishing gradient is a statement about repeated multiplication; the constant error carousel removes the repeated multiplication from the memory path; the gates stop that path from being a dumb pipe. Each piece answers a specific failure.
It is also a reminder about timing. The 1997 paper sat for years before the compute and data arrived to prove it, much like the pattern in the AI winters. Good ideas sometimes wait. If you want to see what runs the sequence models inside your phone’s older features, or why a small gated cell was a landmark, this is the paper to read first, and the follow-up to read second is the Transformer.
Sources: Hochreiter and Schmidhuber, “Long Short-Term Memory”, Neural Computation 9(8), 1997 · Paper PDF · Long short-term memory, Wikipedia · Backpropagation, Wikipedia




