data engineering Archives - Open Data Science - Your News Source for AI, Machine Learning & more

How to Start Your Next Data Engineering Project

Modeling data engineeringposted by ODSC Community June 29, 2022

Many programmers who are just starting out struggle with starting new data engineering projects. In our recent poll on YouTube, most viewers admitted that they have the most difficulty with even starting a data engineering project. The most common reasons noted in the poll were: Finding... Read more

What to Consider When Building Data Pipelines

Tools & Languages Workflow data engineering Pipelinesposted by ODSC Community June 14, 2022

In 2021 we watched Fivetran raise 150 Million, Matillion 16 million and Informatica went public. All of these companies have some piece of their business connected to data pipelines. Also sometimes referenced as ETL, ELT, E(t)LT, and CDC. For today when I say data pipeline I... Read more

A Day in the Life of a Data Engineer

Career Insights Guest contributor data engineeringposted by ODSC Community May 9, 2022

The first time I considered becoming a data professional was in my last year of college when I took a class in machine learning and big data. I saw the potential impact and use cases of machine learning that have simply not been seen before. Even... Read more

40 Data Engineering Platforms & Skills Needed in 2022

Career Insights Featured Post Azure data engineeringposted by ODSC Team January 25, 2022

Data engineering is overtaking “data science” as the hot skillset of the 2020s. Companies are actively seeking people to collect data and load it into pipelines for the rest of the data science team to clean and organize. Without this data, there would be no data... Read more

Top 10 Skills for Data Engineers in 2021

Career Insights Featured Post 2021 Azure data engineeringposted by ODSC Team January 11, 2021

Data engineering is an increasingly sought after job role, and despite a tumultuous 2020, the chart above showed it’s more in demand than ever. Due to the pandemic, jobs were scarce in April but quickly rebounded before the traditional summer lull hit. Demand increased significantly in... Read more

Data Engineer Jobs on the Rise – Skills Needed and Who’s Hiring

Career Insights Featured Post Azure data engineeringposted by Sheamus McGovern October 12, 2020

If you review the schedule or contents of any data science learning platform, such as with the Ai+ Training Platform, you’ll notice that data engineering is a primary focus. That’s no surprise, as data engineer is currently one of the fastest-growing roles. Job site, DICE’s, recent... Read more

Building a Production-Level Data Pipeline Using Kedro

Blogs from ODSC Speakers Conferences Machine Learning Modeling best practice data engineering Europe 2020 Kedro pipeline production codeposted by ODSC Community August 24, 2020

How can Kedro help you? Suppose you are a self-taught data scientist who does not have much experience in software development. One morning, your senior executive asks you to provide an ad-hoc analysis – perks of the job, and when you do, she thanks you for... Read more

The Most Secure Safe in the World: Data Science and Data Engineering

Blogs from ODSC Speakers Business + Management Conferences data engineering Data Science data storageposted by Stephanie Kirmer August 19, 2019

Editor’s Note: If you’re interested in data storage, and the intersection of data science and data engineering, make sure to see Stephanie at ODSC West 2019, and watch her talk, “Integrating Elasticsearch with Analytics Workflows“ Bringing data science and data engineering together in service of business... Read more