EdX

Advanced Data Engineering (edX)

Offered by AI (Pragmatic AI Labs),
Advanced Data Engineering (edX)

Become an expert in scaling data systems. Master Celery, Airflow, graph databases. Build real-world solutions for massive datasets and complex workflows. Optimize performance at enterprise scale.

Class Deals by MOOC List - Click here and see EdX's Active Discounts, Deals, and Promo Codes.

Master Scalable Data Engineering with Cutting-Edge Tools

  • Learn to handle massive datasets efficiently with this advanced course
  • Gain practical expertise in scaling data systems using modern technologies
  • Ideal for data scientists, engineers & professionals with data handling experience

Course Highlights:

  • Leverage Celery & RabbitMQ for scalable data consumption
  • Optimize workflows with Apache Airflow for efficient management
  • Utilize Vector & Graph databases for robust data management at scale
  • Hands-on projects for real-world experience in solving data challenges
  • Create scalable systems & analyze performance for optimum results

Upskill to design, build & optimize data engineering pipelines that can handle complex, large-scale datasets. Prepare for demanding data roles by mastering advanced techniques with this comprehensive training.
This course is part of the following programs:

What you'll learn

  • Create and manage data pipelines and their lifecycle
  • Connect and work with message queues to manage data processing
  • Use vector, graph, and key/value databases for data storage at scale

Syllabus

Module 1: Queues and Databases-RabbitMQ and MySQL (6 hours)
\\- Video: Meet your instructor: Alfredo Deza (1 minute, Preview module)
\\- Video: About this course (2 minutes)
\\- Reading: Connect with your instructor (10 minutes)
\\- Reading: Meet your instructor: Noah Gift (10 minutes)
\\- Reading: Course structure and discussion etiquette (10 minutes)
\\- Video: Introduction (1 minute)
\\- Video: Overview of Queues (5 minutes)
\\- Video: What is Celery? (3 minutes)
\\- Reading: Key Terms (10 minutes)
\\- Reading: Introduction to Celery (10 minutes)
\\- Video: Use cases for RabbitMQ (3 minutes)
\\- Reading: Using RabbitMQ with Docker (10 minutes)
\\- Reading: External lab: Start RabbitMQ in a development environment (10 minutes)
\\- Video: Overview of a Flask and Celery application (3 minutes)
\\- Video: Summary (1 minute)
\\- Quiz: Introduction to RabbitMQ and Flask (30 minutes)
\\- Video: Introduction (0 minutes)
\\- Video: Configuring Celery with Flask (4 minutes)
\\- Video: Connecting Celery with RabbitMQ (5 minutes)
\\- Reading: Key Terms (10 minutes)
\\- Reading: Build a web app by using Python and Flask (10 minutes)
\\- Reading: Background tasks with Celery (10 minutes)
\\- Video: Defining a Celery task in Flask (3 minutes)
\\- Video: Fire and forget task in Flask (2 minutes)
\\- Video: Retrieve values from asynchronous tasks (3 minutes)
\\- Reading: External lab: Add a new Celery task for RabbitMQ (10 minutes)
\\- Video: Summary (1 minute)
\\- Quiz: RabbitMQ with Celery and Flask (30 minutes)
\\- Video: MySQL Overview (2 minutes)
\\- Reading: Key Terms (10 minutes)
\\- Reading: Getting Started with MySQL (10 minutes)
\\- Video: MySQL from Terminal (3 minutes)
\\- Video: Archive and Drop Database (5 minutes)
\\- Video: Import external database Sakila (7 minutes)
\\- Video: Modify database Sakila (4 minutes)
\\- Video: Bash pipelines with MySQL (5 minutes)
\\- Video: MySQL to Python Standard Library Web Server (4 minutes)
\\- Ungraded Lab: Linux Hacking with MySQL (60 minutes)
\\- Quiz: Quiz-MySQL for Data Engineering (30 minutes)
\\- Reading: Lesson Reflection (10 minutes)
\\- Discussion Prompt: Meet and greet (optional) (10 minutes)
\\- Quiz: Queues and Databases - Final week quiz (30 minutes)

Module 2: Optimizing Workflow Management at Scale with Apache Airflow (5 hours)

  • Video: Introduction (1 minute, Preview module)
  • Video: What is Apache Airflow? (6 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: What is Apache Airflow (10 minutes)
  • Video: Installing Apache Airflow from PyPI (5 minutes)
  • Video: Using Apache Airflow with Docker (6 minutes)
  • Reading: Exploring the Airflow User Interface (10 minutes)
  • Reading: External lab: Install Apache Airflow (10 minutes)
  • Video: Exploring the Airflow UI (6 minutes)
  • Quiz: Quiz-Installing Apache Airflow (30 minutes)
  • Reading: Lesson Reflection (10 minutes)
  • Video: Introduction (0 minutes)
  • Video: Exploring directed acyclic graphs (DAG) (10 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: External lab: Create a DAG (10 minutes)
  • Video: Creating a DAG (7 minutes)
  • Video: Running a backfill (4 minutes)
  • Reading: Architecture overview (10 minutes)
  • Video: Testing and validation (7 minutes)
  • Video: Summary (0 minutes)
  • Quiz: Quiz-Apache Airflow Fundamentals (30 minutes)
  • Reading: Lesson Reflection (10 minutes)
  • Video: Introduction (1 minute)
  • Video: Identifying a task to build a DAG (4 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: External Lab: Build a data pipeline for census data (10 minutes)
  • Video: Retrieving remote data (4 minutes)
  • Video: Cleaning and normalizing data (4 minutes)
  • Video: Inspecting the UI for results (4 minutes)
  • Reading: Build Data Pipelines with Apache Airflow (10 minutes)
  • Video: Summary (1 minute)
  • Reading: Lesson Reflection (10 minutes)
  • Quiz: Quiz-Creating a pipeline (30 minutes)
  • Quiz: Final Week Quiz-Optimizing Workflow Management at Scale with Apache Airflow (30 minutes)

Module 3: Achieving Scalability with Vector, Graph, and Key/Value Databases (5 hours)

  • Video: Picking the proper database (3 minutes, Preview module)
  • Video: What are vector databases and how they work (2 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: What is a Vector Database? (10 minutes)
  • Video: Implementing Semantic search (4 minutes)
  • Video: Quickstart Qdrant (3 minutes)
  • Reading: External Lab: Run Quickstart of Qdrant (10 minutes)
  • Video: Qdrant Rust Client (3 minutes)
  • Reading: External Lab: Extend Semantic Search (10 minutes)
  • Video: Vector Database Architectures (2 minutes)
  • Video: Hands-on lab: Enhance Semantic Search (3 minutes)
  • Reading: Jaccard index (10 minutes)
  • Quiz: Quiz-Introduction to Vector Databases (30 minutes)
  • Reading: Lesson Reflection (10 minutes)
  • Video: Graph data models and database concepts (2 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: Rust CLI with Clap (10 minutes)
  • Video: Introduction to Amazon Neptune (2 minutes)
  • Reading: External Lab: Rust Graph CLI Tool (10 minutes)
  • Video: Graph algorithms: UFC graph centrality in Rust (4 minutes)
  • Video: Kosaraju Community Detection in Graphs (4 minutes)
  • Video: Shortest Path with Graphs (3 minutes)
  • Reading: Amazon Neptune (10 minutes)
  • Video: Key Components of Rust CLI Tool (1 minute)
  • Video: Lab Walkthrough: Building a Rust Graph CLI Tool (2 minutes)
  • Quiz: Quiz-Introduction to Graph Databases (30 minutes)
  • Reading: Lesson Reflection (10 minutes)
  • Quiz: Final Quiz-Achieving Scalability with Vector, Graph, and Key/Value Databases (30 minutes)
  • Ungraded Lab: Social Media Recommender (60 minutes)

Module 4: Real-world Advanced Data Engineering Projects (5 hours)

  • Video: Learn AWS CloudShell for Dynamo Development (4 minutes, Preview module)
  • Video: Learn AWS CodeCatalyst for Dynamo Development (5 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: Amazon CodeCatalyst (10 minutes)
  • Video: Leveraging AWS CodeWhisperer for Dynamo Development (4 minutes)
  • Video: Create a Table with CLI (1 minute)
  • Video: Populate a Table With Batching Records (1 minute)
  • Video: Query a Table with Records (2 minutes)
  • Reading: External Lab: Extended DynamoDB (10 minutes)
  • Video: Project Walkthrough (2 minutes)
  • Quiz: Quiz-Building a solution with DynamoDB with the AWS CLI (30 minutes)
  • Reading: Lesson Reflection (10 minutes)
  • Video: Introduction (1 minute)
  • Video: Overview of a pipeline requirements (3 minutes)
  • Reading: Key Terms (10 minutes)
  • Reading: Quick start for SQLAlchemy (10 minutes)
  • Video: Using SqlAlchemy with Pandas (6 minutes)
  • Reading: Explore and analyze data with Python (10 minutes)
  • Video: Persisting data in a task (6 minutes)
  • Video: Reviewing the results (4 minutes)
  • Video: Summary (1 minute)
  • Quiz: Quiz-Persisting data through a multi-task DAG with Pandas (30 minutes)
  • Reading: Lesson Reflection (10 minutes)
  • Reading: Recommended Next Steps (10 minutes)
  • Quiz: Final Quiz-Advanced Data Engineering (30 minutes)
  • Ungraded Lab: Jupyter Sandbox (60 minutes)
  • Ungraded Lab: VS Code Sandbox (60 minutes)
Go to Class
MOOC List is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

Related Courses

Analytics in Python (edX) EdX
Columbia University,ColumbiaX

Analytics in Python (edX)

CS: Programming Business

Learn the fundamental of programming in Python and develop the ability to analyze data and make data-driven decisions. Data is the lifeblood of an organization. Competency in programming is an essential skill for successfully extracting information and knowledge from data. The goal of this course is to introduce learners to the basics of programming in Python and to give a working knowledge of how to use programs to deal with data.

This course is archived
5-12 Weeks
Python Big Data Analytics
Details
MySQL for Data Analytics and Business Intelligence (Coursera) Coursera
Edureka

MySQL for Data Analytics and Business Intelligence (Coursera)

CS: Software Engineering

Embark on a thorough educational expedition through the "Fundamentals of Database Management with MySQL" module. This foundational program aims to impart a robust comprehension of MySQL, a prominent Relational Database Management System (RDBMS). Participants will delve into crucial concepts, ranging from the fundamentals of SQL to more advanced topics like data retrieval, relationships, business intelligence, and data visualization.

Oct 5th 2026
1 Week
SQL Spreadsheets MySQL
Details
ETL and Data Pipelines with Shell, Airflow and Kafka (Coursera) Coursera
IBM

ETL and Data Pipelines with Shell, Airflow and Kafka (Coursera)

CS: Information & Technology

After taking this course, you will be able to describe two different approaches to converting raw data into analytics-ready data. One approach is the Extract, Transform, Load (ETL) process. The other contrasting approach is the Extract, Load, and Transform (ELT) process. ETL processes apply to data warehouses and data marts. ELT processes apply to data lakes, where the data is transformed on demand by the requesting/calling application.

Sep 21st 2026
5-12 Weeks
ETL Kafka Data Pipelines
Details
Mastering Data Analysis in Excel (Coursera) Coursera
Duke University

Mastering Data Analysis in Excel (Coursera)

Statistics & Data Analysis Data Science

Important: The focus of this course is on math - specifically, data-analysis concepts and methods - not on Excel for its own sake. We use Excel to do our calculations, and all math formulas are given as Excel Spreadsheets, but we do not attempt to cover Excel Macros, Visual Basic, Pivot Tables, or other intermediate-to-advanced Excel functionality. This course will prepare you to design and implement realistic predictive models based on data. In the Final Project (module 6) you will assume the role of a business data analyst for a bank, and develop two different predictive models to determine which applicants for credit cards should be accepted and which rejected. Your first model will focus on minimizing default risk, and your second on maximizing bank profits.

Sep 21st 2026
5-12 Weeks
Business Excel Data Analysis
Details
Generative AI and LLMs on AWS (edX) EdX
AI (Pragmatic AI Labs)

Generative AI and LLMs on AWS (edX)

Computer Science

Unlock scalable generative AI with expert training on deploying and optimizing large language models on AWS for peak performance and compliance. Master deploying generative AI models like GPT on AWS through hands-on labs. Learn architecture selection, cost optimization, monitoring, CI/CD pipelines, and compliance best practices. Gain skills in operationalizing LLMs using Amazon Bedrock, auto-scaling, spot instances, and differential privacy techniques. Ideal for ML engineers, data scientists, and technical leaders.

Self Paced
Self-Paced
AWS CI/CD Large Language Models
Details