🚀 A Data Engineer's Journey

📌 Preface

Event: AWS Community Day 2023
Speaker: Kaung SiThu – Lead Data Engineer
Audience: Tech enthusiasts, cloud professionals and aspiring young engineers
Purpose: To share insights into data engineering, career transition, and AWS experiences persuading new generation of engineers interested in data field

Data engineering is a field that powers the backbone of modern business intelligence and decision-making. At AWS Community Day 2023, I had the opportunity to share my journey, experiences, and insights into this fascinating field. This article is a summary of my talk, using real-world analogies and personal reflections to make the topic both engaging and insightful. Whether you're an aspiring data engineer or a seasoned professional, I hope you find value in this narrative. 🚀

Note: I've embedded the slides as they are to get as close as to the talk rather than restructuring the images or illustrations in the text flow.

🪧 Introduction

When people ask, “What is data engineering?” they often expect a complicated, jargon-heavy explanation. But let me simplify it. Imagine a kitchen 🍳. If data analysts are the chefs creating delicious dishes (insights), data engineers are the prep cooks ensuring ingredients (raw data) are cleaned, chopped, and ready to use. Without them, the kitchen (business analytics) would be in chaos!

In this talk, I shared my journey transitioning into data engineering, explored key responsibilities, and gave a peek into AWS data solutions that make our jobs easier.

🔍 What is Data Engineering?

At its core, data engineering is all about collecting, transforming, and preparing data so it can be easily used by businesses to make better decisions. Rather than just throwing out definitions, I like to explain it through something most of us are already familiar with — cooking! 🍽️

🍳 The Kitchen Vs The Business Analytics

Data Engineers = Prep Cooks 👨‍🍳👩‍🍳
- They gather, clean, and organize the ingredients (raw data).
Data Analysts = Chefs 🔪
- They take these ingredients and turn them into a final dish (insights and reports).
Data Platform = The Kitchen 🏠
- The space where all of this magic happens.

A well-functioning kitchen means faster, better, and more scalable decision-making! 📊

🔄 The Career Shift to Data Engineering

Transitioning from manufacturing system engineering to data engineering was not an overnight decision. It took curiosity, effort, and a willingness to learn. Many professionals hesitate to make such a switch, hesitating or doubting their capabilities of the necessary skills. But here’s what I learned:

🎯 Key Lessons:

Your Past Skills Are Transferable ➝ My system engineering background gave me an analytical mindset that proved invaluable in data engineering.
Continuous Learning is Essential 📚 ➝ I embraced reading and watching online articles, documentations, courses, and mentors to bridge knowledge gaps.
Overcoming Impostor Syndrome 😨 ➝ Doubts creep in, but every challenge is a stepping stone.

🏆 Essential Traits of a Data Engineer

To thrive in this data field -- especially data engineering, here are some key skills and mindsets:

✅ Attention to Detail – Double-checking work, breaking down complex problems.
✅ Curiosity – Asking ‘why?’ and ‘how?’ about data.
✅ Continuous Learning – Staying up to date with trends and tools.

☁️ AWS Experiences – Why Cloud?

The cloud is not just a buzzword—it’s a game-changer. Instead of maintaining expensive on-prem servers, companies can scale up or down as needed. AWS dominates the market, and for a good reason:

🔹 Cost efficiency – Pay for what you use.
🔹 Scalability – Grow as your data grows.
🔹 Community Support – Large user base = more resources & solutions.

🚀 Getting Started with AWS Analytics Solutions

Our team took on AWS, learning as we built our data platform. It was a mix of excitement and challenges, but AWS’s strong support helped us get up to speed quickly.

💡 One surprising experience? AWS support responded to my late-night query within minutes and solved an issue that had blocked me for the whole day! 🤯

🔥 The Data Kitchen

Some of the key AWS services we use include:

AWS Glue: Serverless ETL solution 🛠️
Amazon Redshift: Data warehousing solution 📊
AWS Step Functions: Orchestration tool 🔄

Each of these plays a vital role in streamlining data workflows.

🛠️ AWS Glue

AWS Glue helps us automate data integration and transformation. Key features:
✅ Serverless execution
✅ Supports Python & Spark
✅ Job bookmarking for incremental data loads

But it has some limitations—Excel support isn’t native, and some features still require coding.

🏛️ Amazon Redshift

Amazon Redshift is a powerful data warehouse. Features include:
✅ Columnar storage for faster queries
✅ Auto-scaling & managed storage
✅ Query optimization with Redshift Advisor 🤓

🔄 AWS Step Functions

Step Functions simplify workflow orchestration. Benefits:
✅ Visual workflows – Easier debugging 🧐
✅ Fault tolerance – Handles failures gracefully 🔄
✅ Serverless – No infrastructure maintenance 🚀

🎯 The Data Journey Continues!

Data engineering is an ever-evolving field with endless opportunities. Whether you’re just starting out or looking to switch careers, stay curious, keep learning, and embrace challenges. 🌟

Let’s keep cooking up great data insights! 🍽️📊