data engineering Archives

8 Data Lake Vendors to Make Your Data Life Easier in 2023

Modeling data engineering Data Lake MLOpsposted by ODSC Team May 31, 2023

Data has to be stored somewhere. Data warehouses are repositories for your cleaned, processed data, but what about all that unstructured data your organization is starting to notice? Where does it go? To make your data management processes easier, here’s a primer on data lakes, and... Read more

The 30 Most Useful Python Libraries for Data Engineering

Modeling data engineeringposted by Sheamus McGovern January 5, 2023

For the upcoming Data Engineering Summit on January 18th, we’ve reached out to some of the top experts in the field to speak on the topic. We observed from our discussions and research that the most popular data engineering programming languages include Python, Java, Scala, R,... Read more

Ten Areas of Data Engineering Every Team Should Excel At

Modeling data engineeringposted by Sheamus McGovern December 22, 2022

Data engineering is undoubtedly one of the fastest-growing fields of technology and many companies are expanding the responsibility of their data teams to better manage the data stack. Gone are the days when it was simply batch-ingested ETL to analytics dashboards. Companies are increasingly seeking to... Read more

Announcing the Free Virtual Data Engineering Summit January 18th

Conferences Featured Post data engineeringposted by ODSC Team December 9, 2022

ODSC and Ai+ couldn’t be more excited to announce our first-ever Data Engineering Summit. This free virtual event will take place on January 18th, 2023, and will be a full day of expert-led instruction in data engineering and related topics. The Data Engineering Summit is designed... Read more

How to Start Your Next Data Engineering Project

Modeling data engineeringposted by ODSC Community June 29, 2022

Many programmers who are just starting out struggle with starting new data engineering projects. In our recent poll on YouTube, most viewers admitted that they have the most difficulty with even starting a data engineering project. The most common reasons noted in the poll were: Finding... Read more

What to Consider When Building Data Pipelines

Tools & Languages Workflow data engineering Pipelinesposted by ODSC Community June 14, 2022

In 2021 we watched Fivetran raise $565 million, Airbyte $150 Million, Matillion $100 million, Rivery raised $16 million and Informatica went public. All of these companies have some piece of their business connected to data pipelines. Also sometimes referenced as ETL, ELT, E(t)LT, and CDC. For... Read more

A Day in the Life of a Data Engineer

Career Insights Guest contributor data engineeringposted by ODSC Community May 9, 2022

The first time I considered becoming a data professional was in my last year of college when I took a class in machine learning and big data. I saw the potential impact and use cases of machine learning that have simply not been seen before. Even... Read more

40 Data Engineering Platforms & Skills Needed in 2022

Career Insights Featured Post Azure data engineeringposted by ODSC Team January 25, 2022

Data engineering is overtaking “data science” as the hot skillset of the 2020s. Companies are actively seeking people to collect data and load it into pipelines for the rest of the data science team to clean and organize. Without this data, there would be no data... Read more

Top 10 Skills for Data Engineers in 2021

Career Insights Featured Post 2021 Azure data engineeringposted by ODSC Team January 11, 2021

Data engineering is an increasingly sought after job role, and despite a tumultuous 2020, the chart above showed it’s more in demand than ever. Due to the pandemic, jobs were scarce in April but quickly rebounded before the traditional summer lull hit. Demand increased significantly in... Read more

Data Engineer Jobs on the Rise – Skills Needed and Who’s Hiring

Career Insights Featured Post Azure data engineeringposted by Sheamus McGovern October 12, 2020

If you review the schedule or contents of any data science learning platform, such as with the Ai+ Training Platform, you’ll notice that data engineering is a primary focus. That’s no surprise, as data engineer is currently one of the fastest-growing roles. Job site, DICE’s, recent... Read more