Data Engineer Jobs on the Rise – Skills Needed and Who’s Hiring Data Engineer Jobs on the Rise – Skills Needed and Who’s Hiring
If you review the schedule or contents of any data science learning platform, such as with the Ai+ Training Platform,  you’ll... Data Engineer Jobs on the Rise – Skills Needed and Who’s Hiring

If you review the schedule or contents of any data science learning platform, such as with the Ai+ Training Platform,  you’ll notice that data engineering is a primary focus. That’s no surprise, as data engineer is currently one of the fastest-growing roles. Job site, DICE’s, recent 2020 Tech Job Report listed it as the fastest-growing job with 50% growth in 2019. While the “rise of the data engineer,” is a popular phrase, the demand for these engineers has long outstripped the supply. If DICE’s data is correct, the demand for them will continue to do so. 

Data Engineer Hiring

Fig 1. source dice.com

What is a data engineer?

Data engineers are responsible for the overall data infrastructure that supports business products and services. Companies are gathering ever-increasing amounts of data that drive everything from business intelligence dashboards to platform workflows. They design the infrastructure that stores, moves, and integrates data from many different sources. They create environments for data scientists to analyze data (lakes and warehouses), enable ETL at scale, and optimize the ecosystem to ensure continuous insights.

Experts estimate that it takes two to three data engineer jobs per data science job to help maintain that pipeline, driving the high demand for these engineers.


What are the Core Competency Skills?

To better understand who employers are seeking for data engineering jobs, we examined our own jobs portal as well as other sources. In total, we reviewed over 1,350 job postings for Q2 and Q3, 2020 for this role. 

Data Engineer Skills

Fig 2 source: odsc.com

Definitions of a data engineer role vary, but generally it is someone who:

  • Understands data platforms, infrastructure frameworks, and data pipeline workflows.
  • Knows how to source, transform, and analyze data at scale.
  • Employs programming skills and advanced techniques like machine learning to create the “glue” code that binds the data workflow life-cycle.

Fig 2. Is a representative set of skills sourced from our job portal for this role. Unsurprisingly, data skills, such as database, SQL, NoSQL, and ETL (extract, transform, load) feature prominently.  From the postings, it is evident that this role’s domain is in the cloud and that experience on AWS or Azure is a top requirement.  

Additionally, the jobs posted indicated that the ability to engineer data at scale with a mixture of pipelines and utilize other infrastructure tools is a defining feature of this role. Knowledge of areas, such as distributed systems, APIs, big data, and data lakes, as well as of platforms, such as Hadoop, Apache Spark Kafka, and Airflow is also important.

Programming skills also feature prominently in job postings with python leading the pack. Java made the cut also, which is unsurprising since it has been the main infrastructure language for more than a decade. Scala, the language of Apache Spark, is also in high demand. Languages that many data sciences favor, such as R, did not feature prominently. However, machine learning, an expansive topic in itself, was one of the top requirements for a data engineering position.  

According to this report, there is an overlap in skills between a data engineer and a machine learning engineer. This is not surprising as these data engineers need machine learning and analysis skills to build environments that facilitate large scale AI-driven projects.  

Who’s Hiring Now?

Hows hiring data engineers

Fig 3. Who’s Hiring Now

To determine who’s hiring data engineers in the USA, we looked at a sample set from various job sites in August and September.  Big tech including Amazon, Facebook, and Apple are actively recruiting. The finance industry is weathering the pandemic better than many and that’s reflected in strong hiring by Capital One, JPMorgan Chase, and USAA among others. Healthcare companies like United Healthcare and CVS Health are also hiring teams of data engineers.  Interestingly, consulting companies Capgemini and Accenture are also hiring as many industries seek guidance on navigating the crisis.     

Paths to Data Engineering & Upskilling

The Paths to a data engineering role vary. With the Ai+ Training Platform, you gain access to our massive library of data science training courses, workshops, keynotes, and talks. All skills are ideal for those looking to break into the field or to acquire the latest skills needed to get ahead. Some highlighted courses include:

SQL for Data Science: Mona Khalil | Senior Data Scientist | Greenhouse

Data Science in the Industry: Continuous Delivery for Machine Learning with Open-Source Tools: Team from ThoughtWorks, Inc.

How to do Data Science with Missing Data: Matt Brems | Managing Partner, Distinguished Faculty | BetaVector, General Assembly

Continuously Deployed Machine Learning: Max Humber | Lead Instructor | General Assembly

Sheamus McGovern

Founder of ODSC and Software Architect specializing in, complex multi-platform systems across multiple industries including finance, healthcare, and education.