Conversations about science, technology, history, philosophy and the nature of intelligence, consciousness, love, and power. Lex is an AI researcher at MIT and beyond.
…
continue reading
Content provided by TWIML and Sam Charrington. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by TWIML and Sam Charrington or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Similar to The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
The award-winning WIRED UK Podcast with James Temperton and the rest of the team. Listen every week for the an informed and entertaining rundown of latest technology, science, business and culture news. New episodes every Friday.
…
continue reading
Android Backstage, a podcast by and for Android developers. Hosted by developers from the Android engineering team, this show covers topics of interest to Android programmers, with in-depth discussions and interviews with engineers on the Android team at Google. Subscribe to Android Developers YouTube → https://goo.gle/AndroidDevs
…
continue reading
The Data Skeptic Podcast features interviews and discussion of topics related to data science, statistics, machine learning, artificial intelligence and the like, all from the perspective of applying critical thinking and the scientific method to evaluate the veracity of claims and efficacy of approaches.
…
continue reading
Every Friday and Sunday, Slate’s popular daily news podcast What Next brings you TBD, a clear-eyed look into the future. From fake news to fake meat, algorithms to augmented reality, Lizzie O’Leary is your guide to the tech industry and the world it’s creating for us to live in.
…
continue reading
Hanselminutes is Fresh Air for Developers. A weekly commute-time podcast that promotes fresh technology and fresh voices. Talk and Tech for Developers, Life-long Learners, and Technologists.
…
continue reading
The 21st Century began with the rise of the Internet and social media. The next decade will mark the rise of the Intelligent Machines. AI will inhabit all our devices from cars and appliances to smart phones and robots. The Intelligent Machines podcast explores the most exciting revolution humanity has ever seen, filled with promise and peril. More than ever we need to understand what these new devices will bring to our lives and how to make best use of them as the 21st century unfolds. On t ...
…
continue reading
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
TechStuff is getting a system update. Everything you love about Tech Stuff now twice the bandwidth with new hosts, Oz Woloshyn (Sleepwalkers) and Karah Preiss (Sleepwalkers). Oz and Karah bring humour and wit to the table as they break down what's happening in tech...and what it says about us. TechStuff is the podcast where technology meets culture. We speak to the folks building the future to understand what tomorrow will look like and how our technology is changing us: how we live, how we ...
…
continue reading
Centered on Android, unafraid to go far and wide in tech. Each week, hosts Daniel Bader and Will Sattelberg bring you a conversation about the decisions being made from how Google designs its Pixel phones to questions about the efficacy of mass consumption artificial intelligence products. Join us as we reflect on the facts, the feelings, and the connections we're all making today.
…
continue reading
Player FM - Podcast App
Go offline with the Player FM app!
Go offline with the Player FM app!
))
Recurrence and Attention for Long-Context Transformers with Jacob Buckman - #750
Manage episode 511974479 series 2355587
Content provided by TWIML and Sam Charrington. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by TWIML and Sam Charrington or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Today, we're joined by Jacob Buckman, co-founder and CEO of Manifest AI to discuss achieving long context in transformers. We discuss the bottlenecks of scaling context length and recent techniques to overcome them, including windowed attention, grouped query attention, and latent space attention. We explore the idea of weight-state balance and the weight-state FLOP ratio as a way of reasoning about the optimality of compute architectures, and we dig into the Power Retention architecture, which blends the parallelization of attention with the linear scaling of recurrence and promises speedups of >10x during training and >100x during inference. We review Manifest AI’s recent open source projects as well: Vidrial—a custom CUDA framework for building highly optimized GPU kernels in Python, and PowerCoder—a 3B-parameter coding model fine-tuned from StarCoder to use power retention. Our chat also covers the use of metrics like in-context learning curves and negative log likelihood to measure context utility, the implications of scaling laws, and the future of long context lengths in AI applications.
The complete show notes for this episode can be found at https://twimlai.com/go/750.
774 episodes
Recurrence and Attention for Long-Context Transformers with Jacob Buckman - #750
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Share
Manage episode 511974479 series 2355587
Content provided by TWIML and Sam Charrington. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by TWIML and Sam Charrington or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://podcastplayer.com/legal.
Today, we're joined by Jacob Buckman, co-founder and CEO of Manifest AI to discuss achieving long context in transformers. We discuss the bottlenecks of scaling context length and recent techniques to overcome them, including windowed attention, grouped query attention, and latent space attention. We explore the idea of weight-state balance and the weight-state FLOP ratio as a way of reasoning about the optimality of compute architectures, and we dig into the Power Retention architecture, which blends the parallelization of attention with the linear scaling of recurrence and promises speedups of >10x during training and >100x during inference. We review Manifest AI’s recent open source projects as well: Vidrial—a custom CUDA framework for building highly optimized GPU kernels in Python, and PowerCoder—a 3B-parameter coding model fine-tuned from StarCoder to use power retention. Our chat also covers the use of metrics like in-context learning curves and negative log likelihood to measure context utility, the implications of scaling laws, and the future of long context lengths in AI applications.
The complete show notes for this episode can be found at https://twimlai.com/go/750.
774 episodes
सभी एपिसोड
×
Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.
Similar to The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
Conversations about science, technology, history, philosophy and the nature of intelligence, consciousness, love, and power. Lex is an AI researcher at MIT and beyond.
…
continue reading
The award-winning WIRED UK Podcast with James Temperton and the rest of the team. Listen every week for the an informed and entertaining rundown of latest technology, science, business and culture news. New episodes every Friday.
…
continue reading
Android Backstage, a podcast by and for Android developers. Hosted by developers from the Android engineering team, this show covers topics of interest to Android programmers, with in-depth discussions and interviews with engineers on the Android team at Google. Subscribe to Android Developers YouTube → https://goo.gle/AndroidDevs
…
continue reading
The Data Skeptic Podcast features interviews and discussion of topics related to data science, statistics, machine learning, artificial intelligence and the like, all from the perspective of applying critical thinking and the scientific method to evaluate the veracity of claims and efficacy of approaches.
…
continue reading
Every Friday and Sunday, Slate’s popular daily news podcast What Next brings you TBD, a clear-eyed look into the future. From fake news to fake meat, algorithms to augmented reality, Lizzie O’Leary is your guide to the tech industry and the world it’s creating for us to live in.
…
continue reading
Hanselminutes is Fresh Air for Developers. A weekly commute-time podcast that promotes fresh technology and fresh voices. Talk and Tech for Developers, Life-long Learners, and Technologists.
…
continue reading
The 21st Century began with the rise of the Internet and social media. The next decade will mark the rise of the Intelligent Machines. AI will inhabit all our devices from cars and appliances to smart phones and robots. The Intelligent Machines podcast explores the most exciting revolution humanity has ever seen, filled with promise and peril. More than ever we need to understand what these new devices will bring to our lives and how to make best use of them as the 21st century unfolds. On t ...
…
continue reading
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
TechStuff is getting a system update. Everything you love about Tech Stuff now twice the bandwidth with new hosts, Oz Woloshyn (Sleepwalkers) and Karah Preiss (Sleepwalkers). Oz and Karah bring humour and wit to the table as they break down what's happening in tech...and what it says about us. TechStuff is the podcast where technology meets culture. We speak to the folks building the future to understand what tomorrow will look like and how our technology is changing us: how we live, how we ...
…
continue reading
Centered on Android, unafraid to go far and wide in tech. Each week, hosts Daniel Bader and Will Sattelberg bring you a conversation about the decisions being made from how Google designs its Pixel phones to questions about the efficacy of mass consumption artificial intelligence products. Join us as we reflect on the facts, the feelings, and the connections we're all making today.
…
continue reading
Player FM - Podcast App
Go offline with the Player FM app!
Go offline with the Player FM app!
