Tokenizers now and in the future
MP3•Episode home
Manage episode 500166975 series 3676690
Content provided by pretrained.fm, Pierce Freeman, and Richard Diehl Martinez. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by pretrained.fm, Pierce Freeman, and Richard Diehl Martinez or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
History of tokenization going back to 70s language modeling, modern tokenization approaches with BPE, and the future of token-free bitestreams
10 episodes