Career path

Data Engineer

You become the person whose pipelines everyone else runs on.

Every dashboard a CEO reads, every model an ML engineer trains and every AI chatbot answering from company documents depends on data that someone made reliable, fast and correct. That someone is the data engineer: the architect of a company's data infrastructure. It's less hyped, it's in steady demand, and when things work, you're why.

SQLPythonPostgresDuckDBdbtAirflow / DagsterSparkKafkaBigQuery / SnowflakeDocker
A day in your future life
08:00

Your alerting caught it before anyone else: last night's payments load had 0 rows. A partner API changed its date format.

08:40

Fixed the parser, backfilled the night's data, added a test. The CEO's 9 am dashboard is correct. Nobody even noticed.

11:00

Design review: the AI team wants every support ticket embedded within 5 minutes of arriving. You sketch a streaming pipeline.

15:00

You cut the warehouse bill by 38% by partitioning one huge table that every query was scanning in full.

17:30

Data contract signed with the app team: they can't break your pipeline again without a failing check.

What you'll build

  • Pipelines that move and clean data on schedule
  • Data warehouses and models that analysts love
  • Streaming systems for real-time data
  • Data foundations for AI (document and embedding pipelines)

Your maths edge

Less formula-heavy, but deeply logical: set theory (that's what SQL joins are), relational algebra, complexity at scale, distributed systems and data modelling. The maths mindset of precision and edge cases is exactly what makes pipelines correct.

Demand

Among the largest tech posting categories in India (about 8% of live tech postings in late 2026), and its share has nearly doubled. AI makes it more needed, not less.

Entry pay (India, rough)

Roughly ₹5–12 LPA in India at entry; grows quickly with cloud and Spark experience.

The honest truth

Fewer flashy job titles for freshers. Many start as analysts or backend engineers and move across. Strong SQL plus one end-to-end pipeline project plus cloud basics is what makes a fresher stand out.

2-minute taste test

Try a few minutes of the job

Three moments from a data engineer's week. Pick an answer, then see the reasoning. There's no score and no right path for you. It's just a feel for the work.

Situation 1 of 3

Your nightly pipeline failed halfway. You rerun it and now revenue is double-counted for that day. What went wrong in the design?

The roadmap

From your first line of code to people using what you built

Tick off steps as you go. Finishing a quest reveals a sealed insight, and finishing a stage unlocks a capability. Everything is saved in this browser.

Before you start: where to use AI (and where not to)

Use AI freely for

  • Explaining a concept, an error message or someone else's code, then asking follow-up 'why' questions
  • Quizzing you: 'ask me 5 questions on what I just learned'
  • Reviewing code you wrote: bugs, edge cases, security holes

Use carefully

  • Writing code for a concept you're learning: write it yourself first, then compare with the AI's version
  • Choosing libraries: AI may suggest outdated packages, so check the docs and the last release date
  • Facts and numbers: verify anything you'll put in a README, resume or memo

Don't use AI for

  • Code you can't explain line by line. If you can't explain it, you don't own it, and interviews will show it
  • Skipping the struggle. The 20 minutes you spend stuck is when learning happens
  • Deciding what to build or what your analysis concludes. That judgement is your job
Type the first version yourself. Delegate the second. Review every one.

Every quest has its own "use AI for / avoid" note. The full guide is in the AI Playbook.

Remaining work
~250h
At this pace
17 weeks
20 quests

Sprint Zero

Five days: Python, serious SQL, core data concepts and your toolbox. Everything else you'll learn by building pipelines.

5 days4 quests~24 hours
Finish this stage to unlock: You can query and move data, and you know where every tool in the modern data stack fits.

Learn by building

Eight builds that turn into one real data platform: ingest, model, transform, orchestrate, scale, stream, test, serve.

~6 weeks8 quests~62 hours
Finish this stage to unlock: You can build a reliable, tested, orchestrated data platform from raw sources to dashboards and AI.
Job signal: Start applying for data engineering and analytics engineering internships now. How to apply →

Company-style builds

Real platform briefs with business deadlines. You choose the architecture and defend it.

~4 weeks3 quests~74 hours
Finish this stage to unlock: You can design and run a data platform a business depends on every morning.
Job signal: Apply broadly now: data engineer, analytics engineer, big data and platform roles. How to apply →

Architect mode

You know how the pieces work, so stop typing every line. From here on you design, write the spec, direct AI agents and review their work like a tech lead.

~1.5 weeks2 quests~17 hours
Finish this stage to unlock: You can design a system, break it into slices, direct AI agents to build it and catch their mistakes before users do.

Your meaningful project

One project that real people use beats ten tutorial clones. This becomes the story in every interview.

4–6 weeks3 quests~73 hours
Finish this stage to unlock: You have shipped something people actually use, and you can tell the story of how you built it, measured it and improved it. That's the whole job.
Job signal: This project is your headline. Put it at the top of your resume and LinkedIn and talk about it in every interview. How to apply →
Other paths
Full-Stack Applied AI Engineer AI / ML Engineer Data ScientistCompare all →