fbpx
Watch: State of the Art Natural Language Understanding at Scale Watch: State of the Art Natural Language Understanding at Scale
Natural language understanding is a key component in many data science systems that must understand or reason about text. Common use cases include question... Watch: State of the Art Natural Language Understanding at Scale

Natural language understanding is a key component in many data science systems that must understand or reason about text. Common use cases include question answering, paraphrasing or summarization, sentiment analysis, natural language BI, language modeling, and disambiguation. Building such systems usually requires combining three types of software libraries: NLP annotation frameworks, machine learning frameworks, and deep learning frameworks.

[Related Article: The State of Automatic Text Summarization with NLP]

This talk by David Talby at ODSC West 2018 introduces the NLP library for Apache Spark. It natively extends the Spark ML pipeline API’s which enabling zero-copy, distributed, combined NLP & ML pipelines, which leverage all of Spark’s built-in optimizations. Benchmarks and design best practices for building NLP, ML and DL pipelines on Spark will be shared. The library implements core NLP algorithms including lemmatization, part of speech tagging, dependency parsing, named entity recognition, spell checking and sentiment detection. The talk demonstrates using these algorithms to build commonly used pipelines, using PySpark on notebooks that will be made publicly available after the talk.

ODSC Team

ODSC Team

ODSC connects you to the world’s largest community of practicing data scientists, AI experts, and industry executives. Our data science conferences in Boston, San Francisco, London, and other locations attract thousands of data science attendees. Our co-located CxO Summit and AI Expo is a magnet for top executives from some of the world’s largest companies seeking solutions.

1