# Deep learning

How neural networks learn, the frameworks that train them and the architectures behind modern AI, in one map.

## Overview

Deep learning is the branch of machine learning that uses neural networks with many layers. Each layer transforms its input a little, and stacked together they learn rich representations directly from raw data: pixels, audio, text and tables. Deep learning powers image recognition, speech, language models and generative AI.

A child learns to recognise cats not from a rulebook but from many examples, gradually building up from edges and shapes to whole animals. Deep networks learn in layers in a similar way.

## How neural networks learn

A [neural network](/tools/neural-networks) is layers of weighted sums passed through [activation functions](/tools/activation-functions). Training repeats one loop: make predictions, measure the error, compute how each weight contributed using [Backpropagation](/tools/backpropagation), and update weights with an optimizer such as those in [Optimizers](/tools/optimizers). Careful [weight initialization](/tools/weight-initialization) and [learning rate schedules](/tools/learning-rate-schedules) decide whether training starts well and settles well. Techniques such as [Normalization layers](/tools/normalization-layers), [Dropout](/tools/dropout), [Mixed precision](/tools/mixed-precision) and [Gradient checkpointing](/tools/gradient-checkpointing) make training stable and efficient.

## Architectures

- Vision: [CNN](/tools/cnn), [ResNet](/tools/resnet), [ViT](/tools/vit), [U-Net](/tools/u-net)
- Sequences: [Recurrent neural networks](/tools/recurrent-neural-networks), [LSTM](/tools/lstm), [GRU](/tools/gru), [Mamba](/tools/mamba)
- Transformers: [Attention mechanism](/tools/attention-mechanism), [Transformers](/tools/transformers), [Mixture of experts](/tools/mixture-of-experts)
- Generative: [Diffusion models](/tools/diffusion-models), [GAN](/tools/gan), [VAE](/tools/vae)
- Tables: [Tabular foundation models](/tools/tabular-foundation-models), [In-context learning](/tools/in-context-learning), [TabPFN](/tools/tabpfn), [TabICL](/tools/tabicl), [FT-Transformer](/tools/ft-transformer)
- Representations: [Word embeddings](/tools/word-embeddings), [Contrastive learning](/tools/contrastive-learning), [Self-supervised learning](/tools/self-supervised-learning)

## Making models smaller and faster

[Knowledge distillation](/tools/knowledge-distillation), [Pruning](/tools/pruning) and quantization methods such as [GPTQ](/tools/gptq) and [AWQ](/tools/awq) shrink models for deployment.

## The full map

### Frameworks

- [PyTorch](https://www.ozercemkelahmet.com/tools/pytorch.md)
- [PyTorch Lightning](https://www.ozercemkelahmet.com/tools/pytorch-lightning.md)
- [TensorFlow](https://www.ozercemkelahmet.com/tools/tensorflow.md)
- [Keras](https://www.ozercemkelahmet.com/tools/keras.md)
- [JAX](https://www.ozercemkelahmet.com/tools/jax.md)
- [Hugging Face](https://www.ozercemkelahmet.com/tools/hugging-face.md)

### Building blocks

- [Attention mechanism](https://www.ozercemkelahmet.com/tools/attention-mechanism.md)
- [Backpropagation](https://www.ozercemkelahmet.com/tools/backpropagation.md)
- [Normalization layers](https://www.ozercemkelahmet.com/tools/normalization-layers.md)
- [Dropout](https://www.ozercemkelahmet.com/tools/dropout.md)
- [Optimizers](https://www.ozercemkelahmet.com/tools/optimizers.md)
- [Word embeddings](https://www.ozercemkelahmet.com/tools/word-embeddings.md)
- [U-Net](https://www.ozercemkelahmet.com/tools/u-net.md)
- [Knowledge distillation](https://www.ozercemkelahmet.com/tools/knowledge-distillation.md)
- [Pruning](https://www.ozercemkelahmet.com/tools/pruning.md)

### Neural network basics

- [Neural networks](https://www.ozercemkelahmet.com/tools/neural-networks.md)
- [Activation functions](https://www.ozercemkelahmet.com/tools/activation-functions.md)
- [Weight initialization](https://www.ozercemkelahmet.com/tools/weight-initialization.md)
- [Learning rate schedules](https://www.ozercemkelahmet.com/tools/learning-rate-schedules.md)

### Architectures

- CNN
- [Recurrent neural networks](https://www.ozercemkelahmet.com/tools/recurrent-neural-networks.md)
- LSTM
- GRU
- Transformers
- ResNet
- ViT
- Autoencoders
- VAE
- GAN
- Diffusion models

### Frontier topics

- [World models](https://www.ozercemkelahmet.com/tools/world-models.md)
- [Vision-language-action models](https://www.ozercemkelahmet.com/tools/vision-language-action-models.md)

### Large tabular foundation models

- [Tabular foundation models](https://www.ozercemkelahmet.com/tools/tabular-foundation-models.md)
- [In-context learning](https://www.ozercemkelahmet.com/tools/in-context-learning.md)
- TabPFN
- TabICL
- TabSTAR
- XTab
- SAINT
- FT-Transformer
- TabNet
- Masked cell modeling
- Imputation networks

## Related topics

- [Generative AI and LLMs](/topics/generative-ai-llms)
- [Machine learning](/topics/machine-learning)
- [MLOps and AI infrastructure](/topics/mlops-ai-infrastructure)

## Frequently asked questions

### What is the difference between machine learning and deep learning?

Deep learning is a subset of machine learning that uses multi-layer neural networks to learn representations directly from raw data.

### Which framework should I learn?

PyTorch is the most widely used today; JAX is strong for large-scale research on TPUs and GPUs.

## Summary

Deep learning stacks neural network layers trained by backpropagation. Frameworks such as PyTorch and JAX, building blocks such as attention and normalisation, and architectures from CNNs to transformers form its toolkit.