AI engineering is not ML research. It is not traditional software engineering either. It is a new discipline that requires unique skills, team structures, and cultural norms.
Building an effective AI engineering team requires rethinking how we hire, organize, and lead. This post shares what we have learned building AI teams from scratch.
| Role | Focus | Key Skills |
|---|
| ML Engineer | Model training, optimization | PyTorch, training pipelines, MLOps |
| AI Engineer | Application integration | LLM APIs, prompting, RAG systems |
| ML Platform Engineer | Infrastructure | Kubernetes, GPU clusters, serving |
| AI Product Manager | Product strategy | User research, metrics, prioritization |
| AI Designer | User experience | AI-specific UX patterns |
| Dimension | ML Engineer | AI Engineer |
|---|
| Primary work | Train models | Use models |
| Key skills | Math, PyTorch | APIs, prompting, systems |
| Output | Models, weights | Applications, features |
| Background | ML/PhD | Software engineering |
| Scarcity | Very high | High |
AI Engineer responsibilities:
| Area | Responsibilities |
|---|
| Integration | Connect LLMs to applications |
| Prompting | Design and optimize prompts |
| RAG | Build retrieval systems |
| Evaluation | Measure and improve quality |
| Production | Deploy and monitor AI features |
| Area | Responsibilities |
|---|
| Training | Fine-tune and train models |
| Optimization | Reduce latency, improve efficiency |
| Infrastructure | Training pipelines, data processing |
| Research | Evaluate new techniques |
| Collaboration | Work with AI Engineers on deployment |
| Attribute | Why It Matters | How to Assess |
|---|
| Systems thinking | AI is complex | System design interview |
| Experimentation mindset | AI is iterative | Past project discussion |
| Communication | Explain uncertainty | Technical presentation |
| Curiosity | Field changes fast | Learning habits |
| Production focus | Ship, not just prototype | Past deployment experience |
| Round | Focus | Duration |
|---|
| 1 | Technical screen | 45 min |
| 2 | System design | 60 min |
| 3 | AI-specific deep dive | 60 min |
| 4 | Product/collaboration | 45 min |
| 5 | Culture fit | 30 min |
| Category | Example Question |
|---|
| Prompting | "How would you improve this prompt?" |
| RAG | "Design a retrieval system for X" |
| Evaluation | "How would you measure success?" |
| Production | "Walk me through debugging a quality issue" |
| Trade-offs | "When would you fine-tune vs use RAG?" |
| Red Flag | Why It Matters |
|---|
| Only research experience | May struggle with production |
| Cannot explain trade-offs | Lacks practical understanding |
| Dismisses evaluation | Quality will suffer |
| No production deployments | Untested in real conditions |
| Cannot simplify explanations | Communication issues |
| Topology | Best For | Trade-offs |
|---|
| Centralized AI team | Early stage, shared capabilities | Can become bottleneck |
| Embedded AI engineers | Product integration | Duplication, inconsistency |
| Platform + embedded | Scale, consistency | Coordination overhead |
| Full-stack AI pods | Fast iteration | Requires senior talent |
| Company Stage | Team Structure |
|---|
| Early (1-10 AI engineers) | Single team, generalists |
| Growth (10-30) | Platform team + embedded |
| Scale (30+) | Platform + domain pods |
| Team Size | Composition |
|---|
| 3 | 2 AI Engineers, 1 ML Engineer |
| 5 | 3 AI Engineers, 1 ML Engineer, 1 Platform |
| 10 | 5 AI Engineers, 2 ML Engineers, 2 Platform, 1 Lead |
| 20 | Multiple pods with above ratios |
| Starting Point | Path to AI Engineering |
|---|
| Backend engineer | Prompting → RAG → Evaluation → Production |
| ML researcher | APIs → Production systems → Reliability |
| Frontend engineer | AI UX → Integration → Full stack AI |
| Data engineer | Pipelines → Embeddings → RAG systems |
| Skill | Junior | Mid | Senior |
|---|
| Prompting | Basic | Advanced | Expert + teaching |
| RAG | Understands | Builds | Architects |
| Evaluation | Uses frameworks | Designs metrics | Defines strategy |
| Production | Deploys | Debugs | Designs systems |
| ML fundamentals | Aware | Understands | Deep knowledge |
| Investment | ROI | Timeline |
|---|
| Internal workshops | High | Immediate |
| Conference attendance | Medium | Quarterly |
| Certification programs | Low-Medium | Ongoing |
| Pair programming | Very High | Continuous |
| External training | Medium | As needed |
| Traditional | AI Engineering |
|---|
| Deterministic outputs | Probabilistic outputs |
| Fix bugs permanently | Manage error rates |
| Ship and done | Continuous improvement |
| Specs are complete | Requirements evolve |
| Test coverage = confidence | Evaluation suites = confidence |
| Old Mindset | New Mindset |
|---|
| "It works or it does not" | "It works 94% of the time" |
| "Tests pass = ready" | "Evaluation metrics acceptable = ready" |
| "Bug = something is broken" | "Quality regression = needs investigation" |
| Practice | Implementation |
|---|
| Hypothesis-driven | State expected outcome before building |
| Measure everything | Metrics for every experiment |
| Fast iteration | Small changes, quick evaluation |
| Document learnings | Shared knowledge base |
| Celebrate failures | Learning from what did not work |
| Challenge | Traditional | AI Engineering |
|---|
| Estimation | Relatively predictable | High uncertainty |
| Progress visibility | Features shipped | Metrics improved |
| Quality definition | Works/does not work | Acceptable error rate |
| Technical debt | Code quality | Prompt drift, model dependencies |
| Metric | Purpose | Target |
|---|
| Features shipped | Velocity | Team-specific |
| Quality scores | Output quality | Domain-specific |
| Incident rate | Reliability | Less than 1/week |
| Experimentation velocity | Learning speed | 2-3 per week |
| Knowledge sharing | Team growth | Weekly sessions |
| Horizon | Planning Approach |
|---|
| This week | Specific tasks |
| This month | Features and experiments |
| This quarter | Capabilities and outcomes |
| This year | Vision and bets |
| Pitfall | Problem | Solution |
|---|
| Hiring ML PhDs for AI Engineering | Wrong skill set | Hire for production skills |
| Centralized team as bottleneck | Slow product teams | Embed or enable |
| No evaluation culture | Quality unknown | Mandate measurement |
| Research-focused team | Does not ship | Focus on production |
| Ignoring AI-specific UX | Poor user experience | Include AI designers |
- 1AI Engineering is its own discipline - Do not conflate with ML research or traditional engineering.
- 2Hire for production skills - Most teams need engineers who ship, not researchers who publish.
- 3Structure evolves with scale - Start centralized, move to platform + embedded.
- 4Embrace uncertainty culturally - AI teams must be comfortable with probabilistic outcomes.
- 5Invest in evaluation - Teams without measurement culture will struggle with quality.
- 6Plan loosely, iterate quickly - AI development is inherently experimental.
Building AI teams is challenging because the discipline is new. But the teams that figure it out will define the next generation of software.
