fbpx
Podcast: DBRX and Open Source Mixture of Experts LLMs with Hagay Lupesko Podcast: DBRX and Open Source Mixture of Experts LLMs with Hagay Lupesko
Learn about cutting-edge developments in AI and Data Science from the experts who know them best on ODSC’s Ai X Podcast.... Podcast: DBRX and Open Source Mixture of Experts LLMs with Hagay Lupesko

Learn about cutting-edge developments in AI and Data Science from the experts who know them best on ODSC’s Ai X Podcast. Each week we release an interview with a leading expert, core contributor, experienced practitioner, or acclaimed instructor who is helping to shape the future of the AI industry through their work or research.

In today’s episode of ODSC’s Ai X Podcast, we speak with Hagay Lupesko, Senior Director of Engineering in the Mosaic AI team at Databricks and one of the key architects behind Databricks’ groundbreaking large language model, DBRX.

Previously Haguy was the VP of Engineering at Mosaic ML, which was acquired by Databricks in 2023. Hagay has also held AI engineering leadership roles at Meta, AWS, and GE Healthcare. 

Our topic today is the the open-source DBRX large language model, which stands out in the LLM AI landscape due to its innovative use of the Mixture of Experts (MoE) architecture. This architecture allows DBRX to efficiently scale by distributing tasks across 64 combinable experts, allowing the model to select the most suitable set of internal configurations from a pool of experts for each specific task. This results in faster processing and potentially better performance compared to traditional LLM architectures. 

Start listening now to get the full impact of Hagay’s knowledge and insights into DBRX and Mixture of Experts, and don’t forget to subscribe to ODSC’s Ai X Podcast to ensure you never miss an episode. Finally, like what you hear? Leave a review or share it with a friend! You can listen on SpotifyAppleSoundCloud, and Castbox.

 

Show Notes:

Learn more about and connect with Hagay Lupesko: https://www.linkedin.com/in/hagaylupesko/

Learn more about DBRX, its use of Mixture of Experts, and scaling laws:

Introducing DBRX: A New State-of-the-Art Open LLM

https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Mixture of Experts: https://opendatascience.com/what-is-mixture-of-experts-and-how-can-they-boost-llms

Lost in the Middle: How Language Models Use Long Contexts: https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering: https://hotpotqa.github.io

Scaling Laws: https://en.wikipedia.org/wiki/Neural_scaling_law

Training Compute-Optimal Large Language Models: https://arxiv.org/pdf/2203.15556

Advance Architecture Features & Techniques in DBRX:

Rotary Position Encodings (RoPE)

Gated Linear Units (GLU)

Grouped Query Attention (GQA)

GPT-4 Tokenizer (tiktoken)

ODSC Team

ODSC Team

ODSC gathers the attendees, presenters, and companies that are shaping the present and future of data science and AI. ODSC hosts one of the largest gatherings of professional data scientists with major conferences in USA, Europe, and Asia.

1