Data Engineer : Understanding this cutting-edge, high-demand job around data

At school I followed a rather generalist training, my objective was to orient myself towards data-related jobs that deal with all aspects : from data collection to statistical exploitation and machine learning, including storage and management of massive and distributed databases such as big data.

The objective of this article is to focus on a few key concepts/businesses involved in the data domain and to show how they fit together to reach the current state of the art. My ambition here is simply to define, contextualise and show the intuition and position of these concepts in relation to each other.

The Data Engineer is one of the main jobs in Big Data. Discover through this article his/her skills, his/her responsibilities as well as his/her differences with the role of Data Scientist.

The Data Engineer’s role is to transform data into a format suitable for analysis. His/Her mission is to design and build pipelines to transform data into a format that can be used by Data Scientists or Data Analysts. These data often come from multiple sources, and are aggregated in a database well adapted to the expected need.

The Data Engineer’s skills