HackerTube 中文?

Let's build the GPT Tokenizer

Andrej KarpathyYouTube ↗

Further reading 1

How far is next-token prediction from answering a person? →Separate three questions: how an input becomes a representation, how training changes a network and how a capable text predictor becomes an assistant. An explanation at one level does not settle the others.This video helps answer · What is the next unit, and how far back can prediction look?

Caption notes

Karpathy builds a vocabulary from bytes. Counts of adjacent pairs choose a merge, and the new units take part in subsequent counts, changing the sequence the model receives.

Public machine transcript (may contain transcription errors) ↗
Original passages · 2
find the byte pairs that occur the most
Let's build the GPT Tokenizer · ▶ 26:38–26:42
iteratively start minting new tokens
Let's build the GPT Tokenizer · ▶ 26:42–26:44

Hacker News comments

View original on HN ↗